Running pit/descartes with a custom test suite runner

[surli]

Characteristics

• Issue Type: bug
• Reproducibility: always
• Severity: major

Description

We use in https://github.com/xwiki/xwiki-rendering some custom TestSuite to process test resources defined in our own syntax. See: RenderingTestSuite.java and an example of such resource file.

It seems that pitest-descartes is not able to use those tests for computing the mutation scores. Specifically, it shows some file with a mutation score and a coverage null, even if we're absolutely sure those files are used in our tests.

[vmassol]

The issue is the following:

• We fail the build when the mutation score goes below the current threshold
• We added code and also added tests, but these new tests are executed by our JUnit Test Suite
• The added code made the mutation score go down since PIT/Descartes doesn't "see" the JUnit Test Suite.

This currently invalidates our strategy of failing the build when the mutation score does down unfortunately since it leads to false positives so we need to do something about it.

What I don't understand is that PITtest mutates the code under test and not the tests so why does it need to identify the @test tests?

Thanks

[oscarlvp]

PITest needs to discover all test cases, not just test classes. Then the identified test cases are selected and sorted for each mutant so the analysis can be faster.
At the current moment, PITest does not support custom test runners and this issue is probably related to this issue and to #34, #53, #71, and somehow related to #75 and #77.
Maybe this is also related to the issues you are having with methods wrongly reported as pseudo-tested, such as the one reported in #73. For this one I checked the report that @tmortagne added and the method is correctly classified given the test cases that are reported to be executed, but maybe not all covering test cases were considered by PITest.
The solution could be to develop a plugin for PITest that can handle all these particularities. This is the way the support for JUnit5 is being built. See https://github.com/pitest/pitest-junit5-plugin.

[vmassol]

Ok so PITest needs to know the test methods for optimization purposes.

Then one idea could be to collect the test methods and test classes by asking the JUnit Test Runner (or the TestNG one) since it knows the tests (it's its role to discover and execute tests). One way could to use a custom RunListener (e.g. http://maven.apache.org/surefire/maven-surefire-plugin/examples/junit.html#Using_custom_listeners_and_reporters). See also https://junit.org/junit4/javadoc/4.12/org/junit/runner/notification/RunListener.html

Ofc this is different with JUnit5. It seems even nicer in Junit5 since there's the concept of TestPlan (https://junit.org/junit5/docs/5.0.0-M2/api/index.html?org/junit/platform/launcher/TestPlan.html). See also https://stackoverflow.com/questions/48595037/whats-the-junit-5-way-for-runlisteners

We would need to find how to inject some Test Runner wrapper or RunListener/TestExecutionListener but I would go in this direction, i.e. try to delegate the finding of the tests/test classes to the test framework (since that one part of its job to do that).

WDYT?

[oscarlvp]

This is what PIT does by default. The thing is that it does not get into test cases with custom runners.
For that we would need to create a PIT test plugin like the one that has been made for JUnit5. The code is pretty easy to follow and I think most of it can be reused to match XWiki case. This plugin does rely on TestPlan as you say and the rest of the JUnit5 functionalities.

[vmassol]

ok cool, seems there's a direction to work on the problem :) Thx Oscar.

[oscarlvp]

I have been cheking and PIT does run some custom test runners and also has some mechanisms to discover custom test suites. However this does not change the proposed solution as it seems that it can not handle the XWiki custom test suite.
The custom test plugin for PIT should then implement both TestSuiteFinder and TestUnitFinder.
These finders should report as many test cases as test documents there are in the test suite.

[vmassol]

However this does not change the proposed solution as it seems that it can not handle the XWiki custom test suite.

Do we know why yet?

The custom test plugin for PIT should then implement both TestSuiteFinder and TestUnitFinder.
These finders should report as many test cases as test documents there are in the test suite.

Hmm if custom test suites are supposed to be supported by PITest then should we debug the problem and provide a PR against PITest then (rather than implement a custom plugin)?

[surli]

So for the record we still experiencing this issue. I have to decrease PIT threshold on a module since test have been added for a new feature that are not taken into account. (See the referenced PR) Do we have any plan for action for this one?

[oscarlvp]

@surli Can you point me to the exact test class? Also, are you using the latest version of PIT?

[surli]

The exact test class is the following:
https://github.com/xwiki/xwiki-rendering/blob/master/xwiki-rendering-macros/xwiki-rendering-macro-html/src/test/java/org/xwiki/rendering/macro/html/IntegrationTests.java

are you using the latest version of PIT?

We are using PIT 1.4.0, we haven't been able to upgrade because of #95

[oscarlvp]

@surli thanks for the pointers, I'll check this issue again.

[surli]

FTR I just upgraded to use descartes 1.2.5 and PIT 1.4.7, I triggered again pitest on the incriminated module and I obtained the same result as with 1.4.0.