Test cannot be retried

[BoraxTheClean]

I'm using Kotest with Junit5 and testing this plugin with a POC test. I'm happy to provide more information, I'm eager to use this plugin to help my team. My larger application framework is Micronaut.

import io.kotest.core.spec.style.FunSpec
import io.kotest.matchers.booleans.shouldBeTrue
import kotlin.random.Random

class FlakyTest :
  FunSpec(
    {
      test("flaky test that passes 3% of the time") {
        Random.nextBoolean().shouldBeTrue()
        Random.nextBoolean().shouldBeTrue()
        Random.nextBoolean().shouldBeTrue()
        Random.nextBoolean().shouldBeTrue()
        Random.nextBoolean().shouldBeTrue()
      }
    })

excerpt from build.gradle.kts

    retry {
        maxRetries.set(3)
        maxFailures.set(20)
        failOnPassedAfterRetry.set(false)
    }

> Task :
com.
  Test flaky test that passes 3% of the time FAILED
  io.kotest.assertions.AssertionFailedError: expected:<true> but was:<false>

FAILURE: Executed 1 tests in 3.3s (1 failed)
1 test completed, 1 failed
There were failing tests. See the report at: file:///x/index.html
> Task :x:test FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':x:test'.
> The following test methods could not be retried, which is unexpected. Please file a bug report at https://github.com/gradle/test-retry-gradle-plugin/issues
     com.curative.healthplan.api.utils.FlakyTest#flaky test that passes 3% of the time
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.8/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 22s

[pshevche]

@BoraxTheClean , thank you for reaching out. Unfortunately, Kotest is not supported by this plugin due to the dynamic nature of test declaration in Kotest. We have on our roadmap to support it: #93, but we don't have any timeline yet.

As a workaround, I can recommend using Kotest's built-in retry functionality. The reporting might not be as accurate as when using this plugin (for example, the plugin reports all tries, while Kotest only the result of the last run), but it can still help you mitigate the impact of flaky tests. If you need to enable Kotest retries globally for your entire project, consider using this extension.

Let me know if you have any other questions. Otherwise, I'd close the issue. Feel free to subscribe to the one I linked for updates. Hopefully, we'll be able to extend support of the plugin to Kotest in the near future.