Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed flaky tests in test_context.py #555

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

yashsaboo
Copy link

Test test_context_api and test_null_context are flaky in nature.

TestCase passes consistently in the actual order with command:

pytest tests/test_context.py

TestCase fails when one randomises the order in test_context.py module by using the plugin pytest-random-order with command:

pytest tests/test_context.py --random-order-seed=<Insert-Value-Here> -v

The victims, test_context_api and test_null_context, fails in a failing test order because there is at least one test that runs before the victim and “pollutes” the state (e.g., global state: here, it is Context()) on which the victim depends. (Paper)

After investigation, we found the polluter test to be test_context_gevent and test_context_eventlet
For test_context_gevent: --random-order-seed=795276
For test_context_eventlet: --random-order-seed=570036

Potential Fix: After the polluters are run, reset the Context() to its initial state so that it does not pollute the victims.

Please let us know if you require the pass and fail logs.

Contributors:
Yash Saboo
Nirupam K N

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant