@params.txt parsing is various types of broken #250

gdevenyi · 2019-04-18T15:13:07Z

I'm trying to move my variables defined at the top of pipeline into a config file so I can offer differently configured versions.

At first glance, I expected to be able to just move the variables into a file and add "-p" to the front, and everything would go.

So, a variable defined like this:

-p BEASTMODEL_DIR="${System.getenv().QUARANTINE_PATH}/resources/mni_icbm152_nlin_sym_09c_minc2"

Fails:

Could not understand command ${System.getenv().QUARANTINE_PATH}/resources/mni_icbm152_nlin_sym_09c_minc2 or find it as a file

Changing to the bash style:

-p BEASTMODEL_DIR="${QUARANTINE_PATH}/resources/mni_icbm152_nlin_sym_09c_minc2"

Still fails:

Could not understand command ${QUARANTINE_PATH}/resources/mni_icbm152_nlin_sym_09c_minc2 or find it as a file

Changing it to a real path:

-p BEASTMODEL_DIR="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2"

Still fails with java errors:

java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:109)
	at org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:131)
Caused by: java.io.FileNotFoundException: /opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2 (Is a directory)
	at java.io.FileInputStream.open0(Native Method)
	at java.io.FileInputStream.open(FileInputStream.java:195)
	at java.io.FileInputStream.<init>(FileInputStream.java:138)
	at groovy.util.CharsetToolkit.<init>(CharsetToolkit.java:71)
	at org.codehaus.groovy.runtime.ResourceGroovyMethods.newReader(ResourceGroovyMethods.java:1572)
	at org.codehaus.groovy.runtime.ResourceGroovyMethods.getText(ResourceGroovyMethods.java:600)
	at org.codehaus.groovy.runtime.dgm$962.doMethodInvoke(Unknown Source)
	at org.codehaus.groovy.reflection.GeneratedMetaMethod$Proxy.doMethodInvoke(GeneratedMetaMethod.java:73)
	at org.codehaus.groovy.runtime.metaclass.MethodMetaProperty$GetBeanMethodMetaProperty.getProperty(MethodMetaProperty.java:76)
	at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:64)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:296)
	at bpipe.Runner.loadPipelineSrc(Runner.groovy:662)
	at bpipe.Runner$loadPipelineSrc.callStatic(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:56)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:194)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:222)
	at bpipe.Runner.main(Runner.groovy:377)
	... 6 more

The text was updated successfully, but these errors were encountered:

ssadedin · 2019-04-18T21:32:20Z

Can you show the whole command line you are using? Those failures look a bit like what I would expect if the parameters are inserted after the script file rather than before. The general form has to be:

bpipe run <params here> pipeline.groovy <inputs here>

Can you confirm that your form follows the above?

gdevenyi · 2019-04-19T00:53:48Z

I can confirm I'm running it that way:

bpipe run @/data/chamal/projects/gabriel/src/minc-bpipe-library/default2.conf -n300 ~/projects/src/minc-bpipe-library/pipeline.bpipe ../subject*

gdevenyi · 2019-04-19T01:03:32Z

Moving the -n300 had the same result

ssadedin · 2019-04-21T01:23:10Z

Very strange, I can't pick what's wrong.

The reading of the parameters from the file is being done under the hood by the same library that parses the command arguments (not Bpipe doing that explicitly). The behavior of that library is that it literally takes whatever is in the file and injects it as command line arguments.

So it ought to work if the contents of your file is like:

-p BEASTMODEL_DIR="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2"

Note the -p is necessary at the start.

Obviously, it's critical that there are no spaces or anything that would count as a space in there. Otherwise, I'm not sure what's wrong!

ssadedin · 2019-04-21T01:28:18Z

NB: it would be interesting to know if this test passes for you:

https://github.com/ssadedin/bpipe/tree/master/tests/params_from_file

It is not demonstrating ideal usage (the -p should be inside the file!) but it will establish if there's something else really strange interfering (your shell? environment? etc).

gdevenyi · 2019-04-22T00:50:13Z

20:47:32 [/scratch/bpipe/tests/params_from_file] master ± bash run.sh

ERROR: Failed to find expected parameter text in output

20:49:40 [:/scratch/bpipe/tests/params_from_file] master(+5/-1) ± cat test.out
java.lang.ClassNotFoundException: bpipe.Runner
        at org.codehaus.groovy.tools.RootLoader.findClass(RootLoader.java:179)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at org.codehaus.groovy.tools.RootLoader.loadClass(RootLoader.java:151)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:99)
        at org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:131)

ssadedin · 2019-04-22T01:55:54Z

Hmm, that's a problem, but not the one I expected! Just checking the obvious, did you build the code yet?

As in, in the top level directory:

./gradlew dist

gdevenyi · 2019-04-22T18:17:24Z

I used my bpipe install 0.9.9.6 which has been functioning for me and the 20+ folks who use my pipelines written in bpipe otherwsie :)

gdevenyi · 2019-04-23T13:11:52Z

0.9.9.7 release has the same bug.

In case it may be relevant:

± java -version
java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)

gdevenyi · 2019-05-02T13:40:41Z

Following up here. Cloning master and building bpipe from source results in a the test passing, so I'm learning towards this bug being an issue with the bpipe distribution version (+ maybe an interaction with my java)

gdevenyi · 2019-05-28T19:32:42Z

Hi, just following up again.

The downloadable bpipe version is broken, but if I build locally these tests work. Any ideas why?

gdevenyi · 2019-08-01T20:07:41Z

Followup here.

The test passes for 0.9.9.7.

If I use the "recommended format"

-p BEASTMODEL_DIR="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2"
-p BEASTMODEL_NAME="mni_icbm152_t1_tal_nlin_sym_09c"
-p BEASTLIBRARY_DIR="/opt/quarantine/resources/BEaST_libraries/combined"
-p DEFACEMODEL="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c"
-p REGISTRATIONMODEL="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c.mnc"
-p REGISTRATIONBRAINMASK="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c_mask.mnc"
-p REGISTRATIONHEADMASK="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c_headmask.mnc"
-p REGISTRATIONOUTLINE="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c_outline.mnc"
-p REGISTRATIONANTIMASK="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c_antimask.mnc"

bpipe run @/data/chamal/projects/gabriel/src/minc-bpipe-library/default2.conf -n300 ~/projects/src/minc-bpipe-library/pipeline.bpipe ~/projects/personal/raw_data/minc/*mnc

Fails with error:

java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:109)
	at org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:131)
Caused by: java.io.FileNotFoundException: /opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2 (Is a directory)
	at java.io.FileInputStream.open0(Native Method)
	at java.io.FileInputStream.open(FileInputStream.java:195)
	at java.io.FileInputStream.<init>(FileInputStream.java:138)
	at groovy.util.CharsetToolkit.<init>(CharsetToolkit.java:71)
	at org.codehaus.groovy.runtime.ResourceGroovyMethods.newReader(ResourceGroovyMethods.java:1572)
	at org.codehaus.groovy.runtime.ResourceGroovyMethods.getText(ResourceGroovyMethods.java:600)
	at org.codehaus.groovy.runtime.dgm$962.doMethodInvoke(Unknown Source)
	at org.codehaus.groovy.reflection.GeneratedMetaMethod$Proxy.doMethodInvoke(GeneratedMetaMethod.java:73)
	at org.codehaus.groovy.runtime.metaclass.MethodMetaProperty$GetBeanMethodMetaProperty.getProperty(MethodMetaProperty.java:76)
	at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:64)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:296)
	at bpipe.Runner.loadPipelineSrc(Runner.groovy:672)
	at bpipe.Runner$loadPipelineSrc.callStatic(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:56)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:194)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:222)
	at bpipe.Runner.main(Runner.groovy:381)
	at bpipe.Runner9.main(Runner9.java:47)
	... 6 more

If instead I setup like the test:

BEASTMODEL_DIR="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2"
BEASTMODEL_NAME="mni_icbm152_t1_tal_nlin_sym_09c"
BEASTLIBRARY_DIR="/opt/quarantine/resources/BEaST_libraries/combined"
DEFACEMODEL="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c"
REGISTRATIONMODEL="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c.mnc"
REGISTRATIONBRAINMASK="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c_mask.mnc"
REGISTRATIONHEADMASK="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c_headmask.mnc"
REGISTRATIONOUTLINE="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c_outline.mnc"
REGISTRATIONANTIMASK="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c_antimask.mnc"

And run like the test:

bpipe run -p@/data/chamal/projects/gabriel/src/minc-bpipe-library/default2.conf -n300 ~/projects/src/minc-bpipe-library/pipeline.bpipe ~/projects/personal/raw_data/minc/*mnc

We get success:

====================================================================================================
|                              Starting Pipeline at 2019-08-01 16:04                               |
====================================================================================================

=============================== Stage convert (2015_02_02_DLG_t1w_2) ===============================

=============================== Stage convert (2018_07_16_GAD_t1w_5) ===============================

gdevenyi · 2019-09-30T17:30:01Z

Hi,

I'm following up again here because I would like to change my pipeline to use configuration file options but I am not comfortable doing so until the difference between the docs and the functional implementation is resolved.

The working version (and the in-repo test code) uses [email protected] while the documentation specifies @params.txt directly. Which is considered correct? If the later, it is broken during use.

gdevenyi · 2020-12-03T00:58:53Z

Followup here, documentation is still unclear.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

@params.txt parsing is various types of broken #250

@params.txt parsing is various types of broken #250

gdevenyi commented Apr 18, 2019

ssadedin commented Apr 18, 2019

gdevenyi commented Apr 19, 2019

gdevenyi commented Apr 19, 2019

ssadedin commented Apr 21, 2019

ssadedin commented Apr 21, 2019

gdevenyi commented Apr 22, 2019

ssadedin commented Apr 22, 2019

gdevenyi commented Apr 22, 2019

gdevenyi commented Apr 23, 2019

gdevenyi commented May 2, 2019

gdevenyi commented May 28, 2019

gdevenyi commented Aug 1, 2019

gdevenyi commented Sep 30, 2019

gdevenyi commented Dec 3, 2020

@params.txt parsing is various types of broken #250

@params.txt parsing is various types of broken #250

Comments

gdevenyi commented Apr 18, 2019

ssadedin commented Apr 18, 2019

gdevenyi commented Apr 19, 2019

gdevenyi commented Apr 19, 2019

ssadedin commented Apr 21, 2019

ssadedin commented Apr 21, 2019

gdevenyi commented Apr 22, 2019

ssadedin commented Apr 22, 2019

gdevenyi commented Apr 22, 2019

gdevenyi commented Apr 23, 2019

gdevenyi commented May 2, 2019

gdevenyi commented May 28, 2019

gdevenyi commented Aug 1, 2019

gdevenyi commented Sep 30, 2019

gdevenyi commented Dec 3, 2020