Enable programmatic running of benchmarks

[ktoso]

As discussed in #27 (comment) we sometimes may want to runMain com.example.MyRunner in order to get programatic access to the test results (send them over somewhere etc).

This does not currently work, and we'll get

[info] # JMH 1.5 (released 5 days ago)
[info] # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/bin/java
[info] # VM options: <none>
[info] # Warmup: 1 iterations, single-shot each
[info] # Measurement: 1 iterations, single-shot each
[info] # Timeout: 10 min per iteration
[info] # Threads: 1 thread
[info] # Benchmark mode: Single shot invocation time
[info] # Benchmark: org.openjdk.jmh.samples.JMHSample_02_BenchmarkModes.measureSingleShot
[info]
[info] # Run progress: 100.00% complete, ETA 00:00:00
[info] # Fork: 1 of 1
[info] <failure>
[info]
[info] java.lang.IllegalArgumentException: Benchmark does not match a class
[info]  at org.openjdk.jmh.util.ClassUtils.loadClass(ClassUtils.java:90)
[info]  at org.openjdk.jmh.runner.BaseRunner.runBenchmark(BaseRunner.java:214)
[info]  at org.openjdk.jmh.runner.BaseRunner.runBenchmarks(BaseRunner.java:111)
[info]  at org.openjdk.jmh.runner.ForkedRunner.run(ForkedRunner.java:51)
[info]  at org.openjdk.jmh.runner.ForkedMain.main(ForkedMain.java:80)
[info] Caused by: java.lang.ClassNotFoundException: org.openjdk.jmh.samples.generated.JMHSample_02_BenchmarkModes_measureSingleShot
[info]  at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
[info]  at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
[info]  at java.security.AccessController.doPrivileged(Native Method)
[info]  at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
[info]  at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
[info]  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
[info]  at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
[info]  at java.lang.Class.forName0(Native Method)
[info]  at java.lang.Class.forName(Class.java:191)
[info]  at org.openjdk.jmh.util.ClassUtils.loadClass(ClassUtils.java:72)
[info]  ... 4 more
[info]
[info]

Adding these:

./target/scala-2.10/classes/...

to the classpath should solve things.

[ktoso]

oh my glob, it's so easily solved. PR with examples in 10 minutes from now. // cc @sirthias

[sirthias]

Yes, I just realized myself.
All I need to add is a runMain in Compile <<= (runMain in Compile).dependsOn(compile in JmhKeys.Jmh) and things work for me.
Very nice!

[ktoso]

indeed :-)

It's also possible to mainClass in (Compile, run) := Some("com.example.CustomRunnerApp") in case you always use the same runner.

[sirthias]

Yup. I have a whole bunch of benchmarks however.
The only thing that doesn't work is setting mainClass to None and have autodetection present me with a nice selection UI. However, that'd be just icing on the cake. I'll gladly have it w/o icing... :)

[sirthias]

One more thing: Is the current fork in (Compile, run) := true still needed anymore?
Things appear to be ok for me without a fork in (Compile, runMain) := true.

[ktoso]

Resolved and documented - I'll push a 0.1.10 now.

[ktoso]

Thanks for the hint on forking... must look into it, would you mind copy pasting this into a new issue? Thanks!

[sirthias]

Excellent, thanks a lot for being so responsive!
sbt-jmh is great stuff!

As for the "fork ticket", here it is: #30