[tracker] [tests] Python test suite reliability: striving for non-flaky, parallelizable, random-orderable tests.

[jayaddison]

This is an umbrella issue to track progress making the Sphinx test suite reliable. That means:

• No flaky tests.
• Tests can run in parallel.
• Tests are independent of each other.

Regarding the last item: test independence is important to ensure that each test is checking the behaviours it claims to, and does not inadvertently rely on the side-effects of some other test. Similarly, a test must not begin failing if any other test is run before it -- if it does, that could mean (but does not definitely mean) that the application's behaviour itself may vary based on the order in which code is evaluated.

It might seem like there are a large number of items here! However, please bear in mind that we have a total of more than 2000 individual pytest test cases that run.

Test flakiness

• test_build_linkcheck.test_connect_to_selfsigned_fails (pr: [tests] Increase timeouts for linkcheck build tests. #12166)
• test_build_linkcheck.test_defaults (pr: [tests] Increase timeouts for linkcheck build tests. #12166)
• test_build_linkcheck.test_linkcheck_exclude_documents (pr: [tests] Attempt to uncover and resolve test flakiness in 'test_linkcheck_exclude_documents'. #12189)
• ... report another?

Test parallelization

• tests.test_builders.test_build_linkcheck (pr: [tests] linkcheck: bind each test HTTP server to a unique port per-testcase. #12126)
• ... report another?

Test independence

• tests.test_builders.test_build_text.py (pr: text builder: resolve an ordering conflict between two section-number-related tests #12868)
• tests.test_domains.test_domain_cpp (pr: [test] fix side-effect due to repeated fields #12187)
• tests.test_builders.test_build_html (pr: [tests] Make the 'test_html_copy_source' case agnostic of test evaluation order. #12118)
• tests.test_builders.test_build_html_download (pr: Remove test result-sharing between 'test_html5_output' and 'test_html5_download' #12119)
• ... report another?

[AA-Turner]

We should aim to make the test roots source directories read-only. This would help reliability as we can assert that state does not change. It also would speed up the tests as we can avoid copying every test root tree.

A patch to track tests that add new files to source directories (~15 currently)

Subject: [PATCH] Track new files in test srcdirs
---
Index: sphinx/testing/fixtures.py
<+>UTF-8
===================================================================
diff --git a/sphinx/testing/fixtures.py b/sphinx/testing/fixtures.py
--- a/sphinx/testing/fixtures.py	(revision c0681ed22dd28c4009a352aaa453c204cdadfbbd)
+++ b/sphinx/testing/fixtures.py	(revision 6a733c958d017877a86b7619559a38e5d13cd19d)
@@ -153,8 +153,20 @@
     """
     args, kwargs = app_params
     app_ = make_app(*args, **kwargs)
+    old = frozenset(app_.srcdir.rglob('*'))
+
     yield app_
 
+    new = old - set(app_.srcdir.rglob('*'))
+    new = new - frozenset(app_.srcdir.joinpath('_build').rglob('*'))
+    new = frozenset(p for p in new if p.is_file() and p.parent.name != '__pycache__')
+    if new:
+        from pathlib import Path
+        path_san = app_.srcdir.as_posix().replace('/', '~').replace(':', '')
+        filename = f'added~{path_san}~{hash(new)}.txt'
+        Path(filename).write_text('\n'.join(sorted(map(str, new))), encoding='utf-8')
+
     print('# testroot:', kwargs.get('testroot', 'root'))
     print('# builder:', app_.builder.name)
     print('# srcdir:', app_.srcdir)

---

Something else to investigate is rolling back sys.modules, as well as sys.path.

A