-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
make preparelocal use S3 for tarball and unify with --dryrun #6544
Comments
should be enough to go in this order:
|
this relates with #7461 |
|
I would like to explain how we move input files (wrapper, scripts, and user sandbox) in crab systems first. In the normal job submission:
|
So, to unify This is why I said earlier that I want to deprecate the behavior, not remove the command. This is possible and easy to do in client side. Thanks to Dario again for Basically, For the server side, well..a lot of code changes needed, obviously the I will write the detail down tomorrow. |
And of course thanks to Stefano for the original ideas on how to unify both commands and the attempted of improving the |
sounds like you know more than me about this matter now :-)
|
Here is the pointer to the code: These PR changes 3 things:
Because I separated sandbox from |
@belforte I have moved this task to Todo in "CRAB Work Planner". |
I looked at the code and have only some minor questions.
|
OK. Let's deploy the change to DagmanCreator ASAP so we can push new client in production. |
back to this. Code from Wa is in https://github.com/novicecpp/CRABServer/tree/get_sandbox_in_schedd |
doing 4. I found a bug in #8740 which was only affecting sandbox existance check in new code. But I have also found that we also still need |
let's go with a:
|
at the moment |
I decided to go in steps.
@aspiringmind-code @novicecpp I will gladly get your advice if you feel like suggesting something different |
currently preparelocal creates a tarball with all needed stuff for executing the job wrapper
(inside DagmanCreator) called InputFiles.tar.gz and sends it to the schedd, from where crab preparelocal
fetches it to create local directory where to run the job.
Such tarball should be transferred via S3 cache instead, and possibly with same code as for --dryrun.
Even better, --dryrun should be executed inside the directory created by preparelocal, and should not be part of the submit command.
something like:
Also, currently in the schedd there are both
InputFiles.tar.gz
andinput_files.tar.gz
! pretty damn confusingDifficulty is to have a way to implement a piece at a time, w/o breaking things.
The text was updated successfully, but these errors were encountered: