Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Mutation Testing] Pitest Gradle Plugin #12208

Draft
wants to merge 6 commits into
base: trunk
Choose a base branch
from
Draft

[Mutation Testing] Pitest Gradle Plugin #12208

wants to merge 6 commits into from

Commits on Aug 2, 2024

  1. Deps: Update mockito and mockito kotlin to 5.12.0 and 5.4.0 

    Release Notes (Mockito): https://github.com/mockito/
mockito/releases/tag/v5.12.0
Release Notes (Mockito Kotlin): https://github.com/mockito/
mockito-kotlin/releases/tag/5.4.0

------------------------------------------------------------------------

Note that the 'org.mockito:mockito-inline' dependency is now switched
with 'org.mockito:mockito-core' and this is because the inline bytecode
functionality that 'mockito-inline' provided is now the default and part
of 'mockito-core'. Mockito version '5.0.0' introduced this change.

------------------------------------------------------------------------

FYI: This is done in order to make sure that Pitest is working with the
most recent version of Mockito when running tests. This is because,
sometimes, even though normally tests run green, Pitest says that the
suite isn't green. And this happens because Pitest, to make things fast,
does some tricky stuff to split your tests into small independent units.

For more info see: https://pitest.org/faq/
    @ParaskP7
    ParaskP7 committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    43ed638 View commit details
    Browse the repository at this point in the history

  2. Build: Update jvm target from java 8 to java 11 

    This change is required as part of the Mockito update, which is causing
the test to compile with the below error message:

Error Message: "Cannot inline bytecode built with JVM target 11 into
bytecode that is being built with JVM target 1.8. Please specify proper
'-jvm-target' option"
    @ParaskP7
    ParaskP7 committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    c2baf6a View commit details
    Browse the repository at this point in the history

  3. Test: Enable unit tests to return default values 

    This is done because otherwise the Pitest is failing when running some
tests with the below example failure message:

Example Failure Message: "java.lang.RuntimeException: Method i in
android.util.Log not mocked. See https://developer.android.com/r/
studio-ui/build/not-mocked for details."

For more info see: https://developer.android.com/training/testing/
local-tests#mocking-dependencies
    @ParaskP7
    ParaskP7 committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    edaca5b View commit details
    Browse the repository at this point in the history

  4. Build: Add pitest plugin 

    For the Pitest plugin to work with Gradle, the project needs to use the
'gradle-pitest-plugin'. However, this Pitest Gradle plugin is not
working with Android project. As such, a fork of that was created by
Karlo, an Android developer working with 'Droids On Roids', a mobile app
development company.

Main Gradle Plugin: https://github.com/szpak/gradle-pitest-plugin
Fork Gradle Plugin: https://github.com/koral--/gradle-pitest-plugin

------------------------------------------------------------------------

PS: Arcmutate and its accompanying Kotlin plugin can further enhance the
mutation testing experience on Kotlin projects, but this requires a
licence first. The good news is that using this license is free for
open source projects.

Arcmutate Kotlin Support: https://docs.arcmutate.com/docs/
kotlin.html#installation
    @ParaskP7
    ParaskP7 committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    1ad3096 View commit details
    Browse the repository at this point in the history

  5. Test: Ignore transaction launcher related unit tests 

    This is done because otherwise mutation testing will fail with the below
build failure message:

Build Failure Message: "(9) tests did not pass without mutation when
calculating line coverage"

From those 9 tests, 5 are related to the
'OrderDetailsTransactionLauncherTest' test suite, while the rest 4 are
related to the 'OrderListTransactionLauncherTest' test suite.

The problem is that Mockiot cannot mock these classes, as it cannot mock
final classes with the following settings:
- explicit serialization (e.g. withSettings().serializable())
- extra interfaces (e.g. withSettings().extraInterfaces(...))

------------------------------------------------------------------------

For more info see the full(ish) stacktrace below:

Mockito cannot mock this class: class com.automattic.android.tracks.
crashlogging.performance.PerformanceTransactionRepository.
Can not mock final classes with the following settings :
- explicit serialization (e.g. withSettings().serializable())
- extra interfaces (e.g. withSettings().extraInterfaces(...))
...
Caused by: org.mockito.exceptions.base.MockitoException:
Could not modify all classes [class com.automattic.android.tracks.
crashlogging.performance.PerformanceTransactionRepository]
...
Caused by: java.lang.IllegalStateException: Byte Buddy could not
instrument all classes within the mock's type hierarchy
...
This problem should never occur for javac-compiled classes. This problem
has been observed for classes that are:
- Compiled by older versions of scalac
- Classes that are part of the Android distribution
...
Caused by: java.lang.TypeNotPresentException: Type io.sentry
.ITransaction not present
...
Caused by: java.lang.ClassNotFoundException: io.sentry.ITransaction
    @ParaskP7
    ParaskP7 committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    7c2143e View commit details
    Browse the repository at this point in the history

  6. CI: Add mutation testing to unit testing job in pipeline

    @ParaskP7
    ParaskP7 committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    40fc7c4 View commit details
    Browse the repository at this point in the history