Configurable exit codes?

[rodya-mirov]

What would you like?

Right now, cypress exits with a positive exit code if there are any test failures (unless the number of failures is a multiple of 256, which is an open bug, but whatever). This is mostly not useful, because we have thousands of tests and chances are that at least one of them will flake, because that's just how UI testing is, and we look at our test failures in a more detailed report, and it's fine.

What we do want to know is if cypress had some kind of runner error not related to our tests. A variety of failure states are common -- difficulty communicating with chrome (which is transient but not as rare as I'd like), configuration or CLI argument errors, etc. These are the kind of things that I would consider a test run job to be "failed" -- that is, I need to go diagnose the run itself, rather than individual tests.

So I'd like a way to suppress the "if test failures, then exit code" behavior, so I can distinguish test failures (cypress did its job, and the test report is correct) from runner failures (cypress itself had an error that I need to diagnose, and the test report is NOT reliable).

Why is this needed?

I need to distinguish between "tests failed" (the runner worked, and I can trust the test report) and "runner failed" (something bad happened during the test run, and I cannot trust the test report).

Other

No response