Skip to content

Latest commit

 

History

History
241 lines (198 loc) · 9.15 KB

notes.org

File metadata and controls

241 lines (198 loc) · 9.15 KB
Raw

Todos

Test types

  • [ ] |Expectations support
  • [ ] |improve Midje support
  • [ ] |Fudje

Major features

  • [ ] Keep track of intermediate state so post-run hooks can still fire upon Ctrl-C + enables parallelism (also necessary for parallellism)
  • [ ] parallelism (?)

Bug fixes

  • [ ] Should –focus / :focus from CLI/REPL override focus in tests.edn?
  • [ ] Exception in :each handler, what happens?

Improvements

  • [ ] |Allow filtering `–watch` e.g. to skip temp files
  • [ ] |Make a distinction between load errors in the test file being loaded vs. in other namespaces.
  • [ ] |Watch changes to `tests.edn`
  • [ ] Clean output when progress bar used without output capturing
  • [ ] |Support for test-ns-hook (?)
  • [ ] Capture stderr separately
  • [ ] |Use SignalHandler instead of shutdownhooks
  • [ ] |Provide validation in defplugin, don’t allow hooks which don’t exist

Friendly interface

  • [ ] Fail in a better way if source-paths don’t exist (esp. in –watch)
  • [ ] |Warn if –focus does not exist (maybe provide fat-finger hints?)
  • [ ] |Provide a warning when no test namespaces are found (please check test-paths and ns-patterns)
  • [ ] |Better error when plugin is not found. (currently: no method in multimethod -register)
  • [ ] |Warn if plugin in config file is not a keyword
  • [ ] |Fail fast if clojure-version < 1.9
  • [ ] |Similarly kaocha-cljs requires ClojureScript 1.10 (?)

Extra config options

  • [ ] make stack trace filtering configurable
  • [ ] Output reporter output to file (?)

Plugin ideas

  • [ ] Timing info of config/load/run steps
  • [ ] test.check configuration
  • [ ] Detect side effects (see circleci.test)

Cucumber

  • [ ] Better file/line numbers
  • [ ] detect load errors / step definition errors
  • [ ] –watch feature step/files
  • [ ] Cucumber steps don’t currently reload correctly in watch (due to cache) -> convert to use function definitions in ns

ClojureScript

  • [ ] file/line not (always) correct
  • [ ] |need to manually add test dir to classpath
  • [ ] |automatically pick a free port for the websocket

JUnit.xml

Internal

  • [ ] Add smoke tests for all reporters and plugins (anything that is loaded conditionally)

Documentation

  • [ ] Kaocha reporter interface
    • kaocha.report/printed-expression
    • kaocha.report/error-type
    • kaocha.report/error-message -> just use :message
    • kaocha/testable
    • kaocha/test-plan
    • kaocha.testable/meta
    • ::capture/output
    • ::caputer/buffer

Done

  • [X] Print non-leaf failures (#1)
  • [X] Provide inspect plugin with e.g. –print-test-ids
  • [X] Provide extra dynamic var bindings
  • [X] Global pre/post hooks
  • [X] min/max clojure/java version
  • [X] Run config hooks when doing –print-config
  • [X] Define hooks directly in tests.edn
  • [X] Boot: Deal with dynamic classpath (#1)
  • [X] Provide shorthand `:randomize?` instead of `kaocha.plugin.random/randomize?`
  • [X] Check CircleCI runner to see how to improve junit.xml output
  • [X] Only forward events that clojure.test/report understands
  • [X] Provide short plugin names on CLI when in the kaocha.plugin namespace
  • [X] Run CI on different levels of Java and Clojure
  • [X] ClojureScript support
  • [X] Cucumber
  • [X] main namespace for cloverage
  • [X] TAP reporter (provided by clojure.test?)
  • [X] :kaocha.filter/skip-meta should do a meta merge, retaining :kaocha/skip unless ^:replace is specified.
  • [X] Default like src-paths, skip-meta not honored
  • [X] Line number of exceptions, right now it shows the test var. (maybe ok because it’s in the stack trace?)
  • [X] Use FIPP/puget for colored pretty printing
  • [X] repl/config merge overwrites values from tests.edn / extra-vals with default-config
  • [X] prettier diffing of (is (= ))
  • [X] Make zero-assertion check work with –fail-fast
  • [X] Add to tools.deps wiki
  • [X] Improve matcher-combinator support, maybe as a separate package/plugin?
  • [X] look at keyword inheritance for making the reporters more generic
  • [X] watch/api mode: undef removed vars
  • [X] Use `#kaocha/v1 {}` instead of `#kaocha {}`
  • [X] Prevent output of matcher-combinators
  • [X] Watch source paths
  • [X] skip / focus by metadata :skip-meta :exclude-meta :focus-meta
  • [X] skip / focus nss/vars. :skip-meta :exclude-meta :focus-meta
  • [X] profiling
  • [X] capture output
  • [X] wrap reporters in t/with-test-out
  • [X] signal load/parse errors as test failures
  • [X] print config / test plan / test result
  • [X] provide #kaocha convenience reader literal for config file
  • [X] watch: re-run failed tests first

Upstream issues

Updates to pom continually adds superfluous newlines under Java 9

https://dev.clojure.org/jira/browse/TDEPS-29

Classpath mess

Kaocha relies on the ability to dynamically add directories to the classpath. There are a number of libraries that allow you to do this, but none of them work reliably across build tools and java versions.

I signalled this on ClojureVerse but got very little usable input. Still the post is useful for illustrating the problem. I might post there if I manage to better understand how this should be solved.

Java 9 introduced the Jigsaw project which introduces a new first class “module” concept to Java, and a “module path”. Modules export package and depend on other modules. In a post-jigsaw world modules can only “see” types from modules they explicitly depend upon. This might be part of the problem.

Java 9 also contains Reflection changes, which makes the approach that Pomegranate/Dynapath used before (making URLClassLoader modifiable) no longer possible. In Java 9 they cause a warning, in Java 10 they fail irrevocably. This is also a result of Jigsaw.

How classes are found (oracle.com)

Actually not that much useful stuff in here

Good general overview of Jigsaw

  • Class::forName continues to work if the package of the class that is requested (I think? “The provider class”) is known to the ContextClassLoader. (i.e. thread-bound?) -> on top of this code that reflects upon some type is considered in a module which can read the module that contains the type
  • Every classloader has its own “unnamed” module
  • Classes loaded from the regular classpath end up in the unnamed module of the AppClassLoader

Java 9 Modules (blog.andresteingress.com)

Looking into what Clojure does

2c2ed386ed0f6f875342721bdaace908e298c7f3 http://github.com/clojure/clojure/commit/2c2ed386ed0f6f875342721bdaace908e298c7f3 “hotfix for classloading”

http://github.com/clojure/clojure/commit/a4142dc9c2e107528b90bb885ed5e8d92fd3f621 “made eval use ephemeral classloader in all cases”

Huzzaaah seems using the Compiler/LOADER rather than the contextclassloader seems to fix this.

New test types

  • implement -run and -load
  • add specs
  • derive :kaocha.testable.type/leaf
  • handle load errors (in load and run!)
  • set metadata (if applicable)
  • dots
  • documentation
  • test with junit.xml
  • check file/line in test failures
  • result counts in final result

Docker images

function push_clojure_image() {
docker pull $IMAGE
docker run -i -t $IMAGE clojure -e '(println (System/getProperty "java.runtime.name") (System/getProperty "java.runtime.version") "\nClojure" (clojure-version))'
docker run -i -t $IMAGE node --version
docker tag $IMAGE lambdaisland/clojure:openjdk${JDK}
docker push lambdaisland/clojure:openjdk${JDK}
}

IMAGE=docker.nextjournal.com/environment@sha256:3cf693dd52db9898f844c5b6c64a124d9c0612708e95dc1b1064408381be14d9
JDK=11
push_clojure_image

IMAGE=docker.nextjournal.com/environment@sha256:e1fb6255d3b71afe9655193ac57f76be6f74960e3df9d63ac6bfbd208fd20cad
JDK=9
push_clojure_image

IMAGE=docker.nextjournal.com/environment@sha256:32c81619aecfd70363c1341f17dd99701fbd157ea797a7880fabd8e71bca3b6e
JDK=8
push_clojure_image
  • pull image from nextjournal : docker pull <image>
  • tag it : docker tag <nj_image> lambdaisland/clojure:openjdk<version>
  • push it : docker push !$

Links

https://github.com/nubank/emidje

Missing tests

  • [ ] src/kaocha/watch.clj
  • [ ] src/kaocha/config.clj
  • [ ] src/kaocha/hierarchy.clj
  • [ ] src/kaocha/classpath.clj
  • [ ] src/kaocha/load.clj
  • [ ] src/kaocha/specs.clj
  • [ ] src/kaocha/stacktrace.clj
  • [ ] src/kaocha/plugin/profiling.clj
  • [ ] src/kaocha/plugin/print_invocations.clj
  • [ ] src/kaocha/plugin/alpha/info.clj
  • [ ] src/kaocha/plugin/alpha/xfail.clj
  • [ ] src/kaocha/plugin.clj
  • [ ] src/kaocha/assertions.clj
  • [ ] src/kaocha/report/progress.clj
  • [ ] src/kaocha/monkey_patch.clj
  • [ ] src/kaocha/type.clj
  • [ ] src/kaocha/matcher_combinators.clj
  • [ ] src/kaocha/jit.clj