Cannot use this plugin in tandem with Kotlin Multiplatform Tests

[eriwen]

We got a report from one of our users that was having trouble running the test retry plugin with a kotlin project. After some digging, we found that org.jetbrains.kotlin.multiplatform.gradle.plugin creates their own test executer for KotlinJvmTest per https://github.com/JetBrains/kotlin/blob/master/libraries/tools/kotlin-gradle-plug[…]/org/jetbrains/kotlin/gradle/targets/jvm/tasks/KotlinJvmTest.kt
When this happens, this part of the test retry plugin is reached: https://github.com/gradle/test-retry-gradle-plugin/blob/v1.3.1/plugin/src/main/java/org/gradle/testretry/internal/config/TestTaskConfigurer.java#L142 so tests fail with the following:

> Configure project :
Kotlin Multiplatform Projects are an Alpha feature. See: https://kotlinlang.org/docs/reference/evolution/components-stability.html. To hide this message, add 'kotlin.mpp.stability.nowarn=true' to the Gradle properties.


> Task :jvmTest FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':jvmTest'.
> Unexpected test executer: org.jetbrains.kotlin.gradle.targets.jvm.tasks.KotlinJvmTest$Executor@1073de17

Bug reproducer project

[abelkov]

Mirror issue on Kotlin side: https://youtrack.jetbrains.com/issue/KT-49155

[osipxd]

Found a workaround:

tasks.withType<KotlinJvmTest>().configureEach {
    applyTestRetryCompatibilityWorkaround()
}

/**
 * Test retry plugin is incompatible with test tasks that override the `createTestExecuter` method.
 * This includes the [KotlinJvmTest] task which wraps the test executor with its own wrapper.
 *
 * This workaround heavily relies on the internal implementation details of the test-retry plugin and KGP.
 *
 * The test retry plugin adds a `doFirst` action, which:
 * - Retrieves the test executer using `createTestExecuter` (KGP returns wrapped test executer here)
 * - Wraps it with `RetryTestExecuter`
 * - Sets the executer using `setTestExecuter`
 *
 * In the `doLast` action, it expects that `createTestExecuter` returns the previously created `RetryTestExecuter` instance.
 * However, KGP wraps every result of `createTestExecutor` with its own wrapper, resulting in the following nesting:
 *   KotlinJvmTarget$Executer(RetryTestExecuter(KotlinJvmTarget$Executer(DefaultTestExecuter)))
 *
 * KGP wraps the executer only if `targetName` is present, as it is needed to add the target name suffix to the test name.
 * The workaround sets `targetName` to `null` after the first KGP wrapper is created,
 * so `createTestExecuter` returns the previously created executer:
 *   RetryTestExecuter(KotlinJvmTarget$Executer(DefaultTestExecuter))
 *
 * Issue: https://github.com/gradle/test-retry-gradle-plugin/issues/116 (KT-49155)
 */
private fun KotlinJvmTest.applyTestRetryCompatibilityWorkaround() {
    if (targetName == null) return

    val executeTestsActionIndex = taskActions.indexOfLast { it.displayName == "Execute executeTests" }
    check(executeTestsActionIndex != -1) { "Action executeTests not found" }

    // Add the workaround action and then move it to the correct position right before tests execution.
    doFirst("workaround for compatibility with testRetry") {
        targetName = null
    }
    val injectedAction = taskActions.removeFirst()
    taskActions.add(executeTestsActionIndex, injectedAction)
}

I hope it doesn't break any stuff related to task inputs as targetName is marked as @Input

[osipxd]

In my opinion, the problem is in createTestExecuter method semantics violation. Method name implies creation of a new instance on each call, while its implementation in the Test task looks like this:

protected TestExecuter<JvmTestExecutionSpec> createTestExecuter() {
    if (testExecuter == null) {
        return new DefaultTestExecuter(...);
    } else {
        return testExecuter;
    }
}

What could be done on a KGP side (or any other plugin's overriding an executer) is to check if the super.createTestExecuter returns the default implementation:

override fun createTestExecuter(): TestExecuter<JvmTestExecutionSpec> {
    val executer = super.createTestExecuter()

    return if (executer is DefaultTestExecuter) {
        // Wrap the executer or create a new one
    } else {
        // Executer was set by test-retry plugin, return it "as is"
        executer 
    }
}

Though it would be even better to resolve this problem in Gradle. Probably by introducing a new method getTestExecuter like this:

protected final TestExecuter<JvmTestExecutionSpec> getTestExecuter() {
    if (testExecuter == null) {
        return createTestExecuter();
    } else {
        return testExecuter;
    }
}

protected TestExecuter<JvmTestExecutionSpec> createTestExecuter() {
    return new DefaultTestExecuter(...);
}

Third-party plugins will be able to override createTestExecuter while getTestExecuter will return the executor set by test-retry plugin. The only problem is that this solution requires changes to the Gradle codebase, so probably this issue should be moved there.