Hypothesis health checks don't pass reliably if "CI" is not set in the environment.

[mgorny]

With current version of hypothesis:

$ tox -e py36
GLOB sdist-make: /tmp/hyperlink/setup.py
py36 create: /tmp/hyperlink/.tox/py36
py36 installdeps: idna==2.9, pytest==5.2.4, coverage==4.5.4, pytest-cov==2.10.0, hypothesis==5.8.6
py36 inst: /tmp/hyperlink/.tox/.tmp/package/1/hyperlink-20.0.1.dev0.zip
py36 installed: atomicwrites==1.4.0,attrs==19.3.0,coverage==4.5.4,hyperlink @ file:///tmp/hyperlink/.tox/.tmp/package/1/hyperlink-20.0.1.dev0.zip,hypothesis==5.8.6,idna==2.9,importlib-metadata==1.7.0,more-itertools==8.4.0,packaging==20.4,pluggy==0.13.1,py==1.9.0,pyparsing==2.4.7,pytest==5.2.4,pytest-cov==2.10.0,six==1.15.0,sortedcontainers==2.2.2,wcwidth==0.2.5,zipp==3.1.0
py36 run-test-pre: PYTHONHASHSEED='2292208485'
py36 run-test: commands[0] | pytest --cov=hyperlink --cov-report=term-missing:skip-covered --doctest-modules src/hyperlink
=============================================================== test session starts ===============================================================
platform linux -- Python 3.6.11, pytest-5.2.4, py-1.9.0, pluggy-0.13.1
cachedir: .tox/py36/.pytest_cache
rootdir: /tmp/hyperlink, inifile: pytest.ini
plugins: hypothesis-5.8.6, cov-2.10.0
collected 130 items                                                                                                                               

src/hyperlink/_url.py ....................                                                                                                  [ 15%]
src/hyperlink/test/test_common.py ......                                                                                                    [ 20%]
src/hyperlink/test/test_decoded_url.py ..........                                                                                           [ 27%]
src/hyperlink/test/test_hypothesis.py ......F.......                                                                                        [ 38%]
src/hyperlink/test/test_parse.py .                                                                                                          [ 39%]
src/hyperlink/test/test_scheme_registration.py ......                                                                                       [ 43%]
src/hyperlink/test/test_url.py .........................................................................                                    [100%]

==================================================================== FAILURES =====================================================================
______________________________________________ TestHypothesisStrategies.test_hostnames_ascii_nolead _______________________________________________

self = <hyperlink.test.test_hypothesis.TestHypothesisStrategies testMethod=test_hostnames_ascii_nolead>

    @given(hostnames(allow_leading_digit=False, allow_idn=False))
>   def test_hostnames_ascii_nolead(self, hostname):
        # type: (Text) -> None
        """
        hostnames(allow_leading_digit=False, allow_idn=False) generates
        ASCII host names without leading digits.
E       hypothesis.errors.FailedHealthCheck: Data generation is extremely slow: Only produced 1 valid examples in 1.01 seconds (40 invalid ones and 0 exceeded maximum size). Try decreasing size of the data you're generating (with e.g.max_size or max_leaves parameters).
E       See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.too_slow to the suppress_health_check settings for this test.

src/hyperlink/test/test_hypothesis.py:177: FailedHealthCheck
------------------------------------------------------------------- Hypothesis --------------------------------------------------------------------
You can add @seed(2426485286323854916506411185325111964) to this test or run pytest with --hypothesis-seed=2426485286323854916506411185325111964 to reproduce this failure.

---------- coverage: platform linux, python 3.6.11-final-0 -----------
Name                                    Stmts   Miss Branch BrPart  Cover   Missing
-----------------------------------------------------------------------------------
src/hyperlink/_socket.py                   22     20      2      0     8%   3-53
src/hyperlink/_url.py                     690     21    349     12    96%   28-29, 52-53, 115, 122, 126, 253, 280, 378, 391, 571, 886-887, 1201, 1822-1823, 1845-1847, 2370-2372, 113->115, 195->197, 250->253, 277->280, 375->378, 388->391, 570->571, 884->886, 1200->1201, 1519->1524, 1821->1822, 2274->exit
src/hyperlink/hypothesis.py               111      5     37      2    95%   11-14, 101, 244, 100->101, 243->244
src/hyperlink/test/__init__.py             12      6      2      1    50%   14-28, 13->14
src/hyperlink/test/test_hypothesis.py      72      4      6      0    95%   10-11, 18-19
src/hyperlink/test/test_socket.py          29     26      4      1    12%   5-6, 9-45, 8->9
src/hyperlink/test/test_url.py            579      3     16      1    99%   940, 1472-1473, 1471->1472
-----------------------------------------------------------------------------------
TOTAL                                    1766     85    422     17    95%

6 files skipped due to complete coverage.

========================================================= 1 failed, 129 passed in 59.10s ==========================================================
ERROR: InvocationError for command /tmp/hyperlink/.tox/py36/bin/pytest --cov=hyperlink --cov-report=term-missing:skip-covered --doctest-modules src/hyperlink (exited with code 1)
_____________________________________________________________________ summary _____________________________________________________________________
ERROR:   py36: commands failed

(this is with 20.0.0 release)

[mgorny]

When I run it with lower system load, I get another error:

==================================================================== FAILURES =====================================================================
______________________________________________ TestHypothesisStrategies.test_hostnames_ascii_nolead _______________________________________________

self = <hyperlink.test.test_hypothesis.TestHypothesisStrategies testMethod=test_hostnames_ascii_nolead>

    @given(hostnames(allow_leading_digit=False, allow_idn=False))
>   def test_hostnames_ascii_nolead(self, hostname):
        # type: (Text) -> None
        """
        hostnames(allow_leading_digit=False, allow_idn=False) generates
        ASCII host names without leading digits.
E       hypothesis.errors.FailedHealthCheck: It looks like your strategy is filtering out a lot of data. Health check found 50 filtered examples but only 0 good ones. This will make your tests much slower, and also will probably distort the data generation quite a lot. You should adapt your strategy to filter less. This can also be caused by a low max_leaves parameter in recursive() calls
E       See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.filter_too_much to the suppress_health_check settings for this test.

src/hyperlink/test/test_hypothesis.py:177: FailedHealthCheck
------------------------------------------------------------------- Hypothesis --------------------------------------------------------------------
You can add @seed(250732993706412637926940422667233067565) to this test or run pytest with --hypothesis-seed=250732993706412637926940422667233067565 to reproduce this failure.

---------- coverage: platform linux, python 3.6.11-final-0 -----------
Name                                    Stmts   Miss Branch BrPart  Cover   Missing
-----------------------------------------------------------------------------------
src/hyperlink/_socket.py                   22     20      2      0     8%   3-53
src/hyperlink/_url.py                     690     21    349     12    96%   28-29, 52-53, 115, 122, 126, 253, 280, 378, 391, 571, 886-887, 1201, 1822-1823, 1845-1847, 2370-2372, 113->115, 195->197, 250->253, 277->280, 375->378, 388->391, 570->571, 884->886, 1200->1201, 1519->1524, 1821->1822, 2274->exit
src/hyperlink/hypothesis.py               111      5     37      2    95%   11-14, 101, 244, 100->101, 243->244
src/hyperlink/test/__init__.py             12      6      2      1    50%   14-28, 13->14
src/hyperlink/test/test_hypothesis.py      72      4      6      0    95%   10-11, 18-19
src/hyperlink/test/test_socket.py          29     26      4      1    12%   5-6, 9-45, 8->9
src/hyperlink/test/test_url.py            579      3     16      1    99%   940, 1472-1473, 1471->1472
-----------------------------------------------------------------------------------
TOTAL                                    1766     85    422     17    95%

6 files skipped due to complete coverage.

========================================================= 1 failed, 129 passed in 40.11s ==========================================================

[mahmoud]

Yes, this flakiness happens pretty regularly on some of my local tox envs, even though runs are pretty stable on CI. @wsanchez might have some suggestions.

[wsanchez]

If you set the "CI" environment variable, we suppress some of the Hypothesis health checks.

I'm hesitant to do that all the time, because you might need those health checks in your own situation.

Let's keep this ticket open, though, because some analysis of how we might avoid needing to suppress those checks is certainly in order; the CI thing is a work-around. I could use some help there, as I've done some things to reduce these issues, but I don't know how to diagnose them very well. Someone with more experience creating Hypothesis strategies could be a great help.

[mgorny]

Thanks for the tip about CI. It resolves my immediate issue. However, I still think it would be better to control the default via some command-line switch.

[wsanchez]

Perhaps, but I can't really manipulate the tox CLI…

[mgorny]

Yeah, sorry, i was thinking this was the hypothesis upstream bug. Nevermind that.