Test failures in version 3.2.1.1

[tillea]

Hi,
I tried to upgrade the Debian packaged testthat from version 3.2.1 to 3.2.1.1. We are running a test script in our CI which boils down to

cd tests
LC_ALL=C.UTF-8 R --no-save < testthat.R

inside the source tree (and when doing so I get the very same result). The CI contains a full build log which results in

══ Failed tests ════════════════════════════════════════════════════════════════
── Error ('test-mock2.R:62:1'): can mock bindings in another package ───────────
Error in `call_type(call)`: corrupt language object
── Failure ('test-reporter-list.R:54:3'): exercise ListReporter ────────────────
expectation_type(res[[4]]$results[[1]]) (`actual`) not identical to "failure" (`expected`).
`actual`:   "warning"
`expected`: "failure"
── Failure ('test-reporter-list.R:57:3'): exercise ListReporter ────────────────
df$error (`actual`) not equal to c(FALSE, FALSE, FALSE, FALSE, TRUE) (`expected`).
`actual[2:5]`:   FALSE FALSE FALSE FALSE
`expected[2:5]`: FALSE FALSE FALSE TRUE
── Failure ('test-reporter-debug.R:51:3'): debug reporter is not called for successes ──
get_frame_from_debug_reporter(2, success_fun) is not NULL
`actual` is an environment
`expected` is NULL
── Failure ('test-srcrefs.R:19:3'): line numbers captured for expectations and warnings ──
`lines` (`actual`) not equal to c(2, 3) (`expected`).
  `actual`: 2 2 3
`expected`: 2   3
── Failure ('test-srcrefs.R:32:3'): line numbers captured when called indirectly ──
`lines` (`actual`) not equal to 4 (`expected`).
  `actual`: 4 4
`expected`: 4
── Failure ('test-srcrefs.R:42:3'): line numbers captured when called indirectly ──
`lines` (`actual`) not equal to 5 (`expected`).
  `actual`: 5 5
`expected`: 5
── Failure ('test-srcrefs.R:51:3'): line numbers captured inside a loop ────────
`lines` (`actual`) not equal to rep(2, 4) (`expected`).
  `actual[2:8]`: 2 2 2 2 2 2 2
`expected[2:4]`: 2 2 2
── Error ('test-test-that.R:55:5'): infinite recursion is captured ─────────────
<expressionStackOverflowError/stackOverflowError/error/condition>
Error: evaluation nested too deeply: infinite recursion / options(expressions=)?
Backtrace:
     ▆
  1. ├─testthat::with_reporter(...) at test-test-that.R:54:3
  2. │ └─base::tryCatch(...)
  3. │   └─base (local) tryCatchList(expr, classes, parentenv, handlers)
  4. │     └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
  5. │       └─base (local) doTryCatch(return(expr), name, parentenv, handler)
  6. ├─withr::with_options(...) at test-test-that.R:55:5
  7. │ └─base::force(code)
  8. ├─testthat::test_that(...)
  9. │ └─testthat::local_test_context()
 10. │   └─testthat::local_reproducible_output(.env = .env)
 11. │     └─withr::local_language(lang, .local_envir = .env)
 12. │       └─withr:::check_language_envvar("LC_ALL")
 13. │         └─base::warning(...)
 14. ├─base::.signalSimpleWarning(...)
 15. │ └─base::withRestarts(...)
 16. │   └─base (local) withOneRestart(expr, restarts[[1L]])
 17. │     └─base (local) doWithOneRestart(return(expr), restart)
 18. ├─testthat (local) `<fn>`(`<smplWrnn>`)
 19. │ └─rlang::cnd_entrace(e)
 20. │   └─rlang::trace_back(top = top, bottom = bottom)
 21. │     └─rlang:::map(calls, call_zap_inline)
 22. │       └─base::lapply(.x, .f, ...)
 23. │         └─rlang (local) FUN(X[[i]], ...)
 24. └─rlang:::call_type_sum(x)
 25.   ├─rlang::sym(sprintf("<%s>", rlang_type_sum(x)))
 26.   │ └─rlang::is_symbol(x)
 27.   ├─base::sprintf("<%s>", rlang_type_sum(x))
 28.   └─rlang:::rlang_type_sum(x)
 29.     └─rlang::is_installed("pillar")
 30.       └─rlang:::detect_installed(info)
 31.         ├─rlang:::list_c(...)
 32.         │ └─rlang::inject(c(!!!x))
 33.         │   └─rlang::enexpr(expr)
 34.         └─rlang:::pmap(...)
 35.           └─rlang:::.rlang_purrr_args_recycle(.l)
 36.             └─rlang:::map_int(args, length)
 37.               └─rlang:::.rlang_purrr_map_mold(.x, .f, integer(1), ...)
 38.                 └─base::vapply(.x, .f, .mold, ..., USE.NAMES = FALSE)
── Failure ('test-test-that.R:102:3'): no braces required in testthat 2e ───────
`test_that("", expect_true(TRUE))` generated warnings:
* Changing language has no effect when envvar LC_ALL='C.UTF-8'
[ FAIL 10 | WARN 1412 | SKIP 122 | PASS 819 ]

Please let me know if you need further information to track down the issue. In case you can't reproduce the issue the Debian Med team policy contains a hint to a docker file which could be useful to reproduce the problem on a Debian sid system.
Kind regards, Andreas.

[tillea]

Ping?

[nunotexbsd]

@tillea

Just for curiosity what are the changes between versions 3.2.1.1 and 3.2.1?

As I can't find ChangeLog or News file related I did a simple diff between versions:

diff -ruN 3.2.1/DESCRIPTION 3.2.1.1/DESCRIPTION
--- 3.2.1/DESCRIPTION   2023-12-02 11:50:05.000000000 +0000
+++ 3.2.1.1/DESCRIPTION 2024-04-14 04:25:11.000000000 +0100
@@ -1,6 +1,6 @@
 Package: testthat
 Title: Unit Testing for R
-Version: 3.2.1
+Version: 3.2.1.1
 Authors@R: c(
     person("Hadley", "Wickham", , "[email protected]", role = c("aut", "cre")),
     person("Posit Software, PBC", role = c("cph", "fnd")),
@@ -31,10 +31,10 @@
 Encoding: UTF-8
 RoxygenNote: 7.2.3
 NeedsCompilation: yes
-Packaged: 2023-11-30 22:07:21 UTC; hadleywickham
+Packaged: 2024-04-14 03:24:27 UTC; ripley
 Author: Hadley Wickham [aut, cre],
   Posit Software, PBC [cph, fnd],
   R Core team [ctb] (Implementation of utils::recover())
 Maintainer: Hadley Wickham <[email protected]>
 Repository: CRAN
-Date/Publication: 2023-12-02 11:50:05 UTC
+Date/Publication: 2024-04-14 05:24:52
diff -ruN 3.2.1/MD5 3.2.1.1/MD5
--- 3.2.1/MD5   2023-12-02 11:50:05.000000000 +0000
+++ 3.2.1.1/MD5 2024-04-14 04:25:11.000000000 +0100
@@ -1,4 +1,4 @@
-7a02630a3e38f7521fb5c44a3f697a8a *DESCRIPTION
+7ad03c426643c38668b5fe0673a9a970 *DESCRIPTION
 d0b909bd833cd58ea0430eda333b6447 *LICENSE
 98b8d9fa7a3c0f90c19158aa47ec85f1 *NAMESPACE
 b682a5016b2f64f2b8704f47aa54c3eb *NEWS.md
@@ -113,7 +113,7 @@
 1ece0759f60e98193d53545648d19146 *inst/examples/test-failure.R
 522cf134c79624117d579f2c4d18f42b *inst/examples/test-success.R
 543392950ccc39ed50ebf26c75de0910 *inst/include/testthat.h
-5058228172d50a29a234414c78b25399 *inst/include/testthat/testthat.h
+05f098fedb1fe85ee85de806f711779f *inst/include/testthat/testthat.h
 b5816f117a5460c67e3e08b1c23dacbc *inst/include/testthat/vendor/catch.h
 f481e7613929d4968a4a4c73031f7967 *inst/resources/catch-routine-registration.R
 f35a879e88834e75aee99fc9817547a0 *inst/resources/test-cpp.R
diff -ruN 3.2.1/inst/include/testthat/testthat.h 3.2.1.1/inst/include/testthat/testthat.h
--- 3.2.1/inst/include/testthat/testthat.h      2023-11-29 18:00:12.000000000 +0000
+++ 3.2.1.1/inst/include/testthat/testthat.h    2024-04-14 04:23:53.000000000 +0100
@@ -172,7 +172,7 @@
 extern "C" SEXP run_testthat_tests(SEXP use_xml_sxp) {
   bool use_xml = LOGICAL(use_xml_sxp)[0];
   bool success = testthat::run_tests(use_xml);
-  return ScalarLogical(success);
+  return Rf_ScalarLogical(success);
 }

 # endif // TESTTHAT_TEST_RUNNER
@@ -198,7 +198,7 @@
 #  include <R.h>
 #  include <Rinternals.h>
 extern "C" SEXP run_testthat_tests() {
-  return ScalarLogical(true);
+  return Rf_ScalarLogical(true);
 }

 # endif // TESTTHAT_TEST_RUNNER

And there is changes in testthat.h that maybe causing the error.

I have yet not updated FreeBSD port, but soon I will test it too and share results.

Thanks

[nunotexbsd]

@tillea

I found that commit at 93b88db
I will test it today

[hadley]

That release was performed by BDR, not by us, but I find it hard to believe that it's responsible for your test failures.