Skip to content

Commit

Permalink
Improve testing with stats and errors per test section
Browse files Browse the repository at this point in the history
* TestRunner: separated the running of each "section" of a test into its own
  try-catch so that we can evaluate each section separately even if
  another section failed (e.g. evaluate all examples even if one test in
  "facts" throws an exception). I've added tests that exercise this "new
  feature".

* TestResults: refactored it a bit so that TestResults now has the three
  sections and the old TestResults becomes TestSectionResults. Also,
  TestSectionResults now has only *one* error, since that's what we get
  when there's an exception evaluating the Pkl code.

* SimpleReport:
  * Separated reporting of each test section so that it's easy to see
    which facts or which examples fail.

  * Added a "stats line" to the module and section levels that reports
    how many tests / assertions pass/fail. Note that, when reporting
    errors, there's no stats for the section since the error is all the
    info we get. Similarly, when examples fail, there's no stats because
    each example is converted into its own test so there are no
    failed assertions to report, it's all-or-nothing.

  * For the examples converted to tests, when multiple examples share
    the same name I've also added a counter ("# 1" and so on) so that
    it's easier to identify which one is failing.

* JUnitReport: fixed the reporting of failures as it wasn't consistent
  with "tests": the tests were the number of tests that we run but
  "failures" were the number of assertions that failed.
  • Loading branch information
jjmaestro committed Aug 7, 2024
1 parent 604b042 commit 8e0aa94
Show file tree
Hide file tree
Showing 7 changed files with 711 additions and 331 deletions.
22 changes: 12 additions & 10 deletions pkl-cli/src/test/kotlin/org/pkl/cli/CliTestRunnerTest.kt
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,9 @@ class CliTestRunnerTest {
assertThat(out.toString().stripFileAndLines(tempDir))
.isEqualTo(
"""
module test
succeed ✅
module test ✅ 100.0% pass [1 passed]
facts ✅ 100.0% pass [1 passed]
succeed ✅ 100.0% pass [2 passed]
"""
.trimIndent()
Expand All @@ -80,7 +81,7 @@ class CliTestRunnerTest {
facts {
["fail"] {
4 == 9
"foo" == "bar"
"foo" != "bar"
}
}
"""
Expand All @@ -96,10 +97,10 @@ class CliTestRunnerTest {
assertThat(out.toString().stripFileAndLines(tempDir))
.isEqualTo(
"""
module test
fail ❌
4 == 9 ❌
"foo" == "bar"
module test ❌ 0.0% pass [0 passed, 1 failed]
facts ❌ 0.0% pass [0 passed, 1 failed]
fail ❌ 50.0% pass [1 passed, 1 failed]
4 == 9
"""
.trimIndent()
Expand All @@ -118,7 +119,8 @@ class CliTestRunnerTest {
9 == trace(9)
"foo" == "foo"
}
["fail"] {
["bar"] {
"foo" == "foo"
5 == 9
}
}
Expand All @@ -136,8 +138,8 @@ class CliTestRunnerTest {
"""
<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="test" tests="2" failures="1">
<testcase classname="test" name="foo"></testcase>
<testcase classname="test" name="fail">
<testcase classname="test.facts" name="foo"></testcase>
<testcase classname="test.facts" name="bar">
<failure message="Fact Failure">5 == 9 ❌</failure>
</testcase>
<system-err><![CDATA[9 = 9
Expand Down
Loading

0 comments on commit 8e0aa94

Please sign in to comment.