Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce refleak timeout from 3h20 to 45min #543

Merged
merged 1 commit into from
Oct 17, 2024

Conversation

vstinner
Copy link
Member

No description provided.

@vstinner vstinner changed the title Reduce refleak timeout from 3h20 to 1h Reduce refleak timeout from 3h20 to 45min Oct 17, 2024
@vstinner
Copy link
Member Author

AMD64 Fedora Stable Refleaks 3.13 slowest test was 9 minutes, whereas a test was blocked for 3 hour 20 min.

cc @zware

@vstinner
Copy link
Member Author

  • AMD64 Windows11 Refleaks 3.x slowest test: 17 min 11 sec
  • ARM64 MacOS M1 Refleaks NoGIL 3.x slowest test: 6 min 14 sec
  • s390x RHEL9 Refleaks 3.x: 4 min 48 sec

@vstinner vstinner merged commit c49ec21 into python:main Oct 17, 2024
1 check passed
@vstinner vstinner deleted the refleak_timeout branch October 17, 2024 09:09
@encukou
Copy link
Member

encukou commented Oct 23, 2024

There are slower builders on 3.10 which has monolithic test_asyncio. AMD64 Fedora Stable Refleaks 3.10 and aarch64 RHEL8 Refleaks 3.10 used to take closer to an hour, and are failing now.
I expect 3.9 builders to start failing when they first build with this config.

Is there any extra background info for this change? Would it be better to revert it, or special-case the branches, like this for example?

diff --git a/master/custom/factories.py b/master/custom/factories.py
index d4e703e..23dd122 100644
--- a/master/custom/factories.py
+++ b/master/custom/factories.py
@@ -72,6 +72,11 @@ class UnixBuild(BaseBuild):
         # Adjust the timeout for this worker
         self.test_timeout *= kwargs.get("timeout_factor", 1)
 
+        # Before 3.11, test_asyncio wasn't split up, so refleaks tests
+        # need more time.
+        if branch in ("3.9", "3.10") and has_option("-R", self.testFlags):
+            self.test_timeout *= 2
+
         if self.build_out_of_tree:
             self.addStep(
                 ShellCommand(

@vstinner
Copy link
Member Author

Is there any extra background info for this change?

When multiprocessing or concurrent.futures or another test hangs, we wasted 3h in waiting, whereas slowest test takes 18 min. Then buildbot overall timeout of 4h killed the re-run job.

@zware
Copy link
Member

zware commented Oct 23, 2024

I'm in favor of either that kind of special case, or just dealing with it on the rare occasion that the 3.10 builders run. I definitely do not want to revert :)

@encukou
Copy link
Member

encukou commented Oct 24, 2024

I'd like to schedule some extra runs on those builders from time to time. Normal runs are meant for security fixes, and that's a bad time to discover latent issues.

I filed the patch as #545

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants