make preparelocal use S3 for tarball and unify with --dryrun

[belforte]

currently preparelocal creates a tarball with all needed stuff for executing the job wrapper
(inside DagmanCreator) called InputFiles.tar.gz and sends it to the schedd, from where crab preparelocal
fetches it to create local directory where to run the job.

Such tarball should be transferred via S3 cache instead, and possibly with same code as for --dryrun.

Even better, --dryrun should be executed inside the directory created by preparelocal, and should not be part of the submit command.

something like:

• submit --dry : does all things like submit, uploads all tarballs, but does not submit to schedd
• preparelocal : like now, creates local dir for a task submitted or submitted with --dry
• crab testrun (new command) runs in the local director for 10 events (like current submit --dry)

Also, currently in the schedd there are both InputFiles.tar.gz and input_files.tar.gz ! pretty damn confusing

Difficulty is to have a way to implement a piece at a time, w/o breaking things.

[belforte]

should be enough to go in this order:

1. make crab preparelocal use things from S3 and never talk to sched
2. implement crab testrun as something to do after crab preparelocal
3. change submit --dry to be stop after uploading and point user to crab testrun

[belforte]

this relates with #7461

[novicecpp]

The goals of this issue The things we agreed in the last meeting are:

• We can deprecate crab submit --dryrun behavior.
  • Current behavior is we upload "custom" InputFiles.tar.gz to S3 and set tasks status to UPLOADED (see DryRunUploader). Then, client pull this custom inputFiles.tar.gz, extract, and run locally. Each time it run will increase number of event it consumed and measure a time to give user idea how to adjust splitting parameter.
  • Users more likely want to check if crabConfig.py/PSet.py able to run in grid than see how long does it take (citation needed).
• Nothing change to preparelocal. Download the submitted file and prepare env for run locally. Note that to do preparelocal, user need to submit the real task with limited total job to run.

[novicecpp]

I would like to explain how we move input files (wrapper, scripts, and user sandbox) in crab systems first.

In the normal job submission:

• First, user submit the task, client upload the sandbox to S3 (s3://crabcache/<cernusername>/sandboxes/<sha1>.tar.gz).
• Second, TW create InputFiles.tar.gz as zipped executable to transfer to schedd later

  CRABServer/src/python/TaskWorker/Actions/DagmanCreator.py

  Lines 726 to 729 in 9b4679d

  	tf = tarfile.open('InputFiles.tar.gz', mode='w:gz')
  	try:
  	for ifname in inputFiles + subdags + ['input_args.json']:
  	tf.add(ifname)

  .
  • This include user sandbox.tar.gz where we download back from S3 then add to the list of InputFiles

    CRABServer/src/python/TaskWorker/Actions/DagmanCreator.py

    Lines 1239 to 1240 in 9b4679d

    	if kw['task'].get('tm_user_sandbox') == 'sandbox.tar.gz':
    	inputFiles.append('sandbox.tar.gz')

    List of files in InputFiles.tar.gz is described in here
• Third, the InputFiles.tar.gz will be "submitted" to schedd along with subdag.jdl (I do not know what it is) through transfer_input_files

  CRABServer/src/python/TaskWorker/Actions/DagmanSubmitter.py

  Line 540 in 9b4679d

  jobJDL["transfer_input_files"] = str(info['inputFilesString'])

  of condor submit command.
• Forth, in schedd, dag_bootstrap_startup.sh extract InputFiles.tar.gz to SPOOL_DIR

  CRABServer/scripts/dag_bootstrap_startup.sh

  Line 54 in 9b4679d

  TARBALL_NAME="InputFiles.tar.gz"

• Finally, when Dagman submit the job, it use transfer_input_files again but only upload a necessary files. You can see the list here and here

  CRABServer/src/python/TaskWorker/Actions/DagmanCreator.py

  Lines 537 to 552 in 9b4679d

  	info.setdefault("additional_input_file", "")
  	if os.path.exists("CMSRunAnalysis.tar.gz"):
  	info['additional_environment_options'] += 'CRAB_RUNTIME_TARBALL=local'
  	info['additional_input_file'] += ", CMSRunAnalysis.tar.gz"
  	else:
  	raise TaskWorkerException("Cannot find CMSRunAnalysis.tar.gz inside the cwd: %s" % os.getcwd())
  	if os.path.exists("TaskManagerRun.tar.gz"):
  	info['additional_environment_options'] += ' CRAB_TASKMANAGER_TARBALL=local'
  	else:
  	raise TaskWorkerException("Cannot find TaskManagerRun.tar.gz inside the cwd: %s" % os.getcwd())
  	if os.path.exists("sandbox.tar.gz"):
  	info['additional_input_file'] += ", sandbox.tar.gz"
  	info['additional_input_file'] += ", run_and_lumis.tar.gz"
  	info['additional_input_file'] += ", input_files.tar.gz"
  	info['additional_input_file'] += ", submit_env.sh"
  	info['additional_input_file'] += ", cmscp.sh"

[novicecpp]

So, to unify preparelocal and submit --dryrun,
I will make submit --dryrun to have the similar UX as preparelocal but without submitting to the schedd. This functionality is for users to test their new tasks before real crab submission (Thanks to Dario for the idea and pointout the pain of preaparelocal).

This is why I said earlier that I want to deprecate the behavior, not remove the command.

This is possible and easy to do in client side. Thanks to Dario again for crab recover command that use other command like crab status/kill behind the scence.

Basically, crab submit --dryrun will submit the task to let TW create the InputFiles.tar.gz, wait until task status change to UPLOADED and then continue to trigger preparelocal command.

For the server side, well..a lot of code changes needed, obviously the InputFiles.tar.gz need to upload to S3, and spliting sandbox from InputFiles.tar.gz to not reupload the sandbox again and again, especially the compaign-like task where it share sandbox.

I will write the detail down tomorrow.

[novicecpp]

And of course thanks to Stefano for the original ideas on how to unify both commands and the attempted of improving the --dryrun code (both client and server side) that I can reuse it to fix this issue.

[belforte]

sounds like you know more than me about this matter now :-)

subdag.jdl is used for automatic splitting, we do not want users to run automatic splitting machinery interactively via --dryrun

[novicecpp]

Here is the pointer to the code:
Server: PR #8645
Client: PR dmwm/CRABClient#5329

These PR changes 3 things:

• Separate sandbox.tar.gz from InputFiles.tar.gz
  • InputFiles.tar.gz is the file contains everything we need to run in schedd and work node.
  • This is needed to unify the code between submit, submit --dryrun, and preparelocal.
• crab submit --dryrun do preparelocal, but the task is not submit to schedd.
• preparelocal --jobid 1 simply call preparelocal and execute in one go
  • When I read the preparelocal code, I feel like it does not make any sense to have a different way to run the job. I might be wrong though.

Because I separated sandbox from InputFiles.tar.gz, I am not sure if some other commands or code have the assumption that it expects sandbox in there, beside the code I changed.
So, it needs more test. But the simple test is passed though https://gitlab.cern.ch/crab3/CRABServer/-/pipelines/7990590 ( the branch base on master 7f4e9eed).

[novicecpp]

@belforte I have moved this task to Todo in "CRAB Work Planner".
Feel free to bump priority as you see fit.

[belforte]

I looked at the code and have only some minor questions.
Time to lay out a plan. As a general strategy I'd rather break submit --dryrun for a while then mix old and new code. Cleaning up has always been difficult.

• 1. add support for checkS3Object in RESTCache . This can go in immediately and deploy new REST
• 2. add checkS3Object to ServerUtilities
• 3. Modify AdjustSites and dag_bootstrap_startup to download sandbox from S3. Can go in at any time
• 4. Modify DagmanCreator to stop downloading sandbox.tar.gz and upload slimmed InputFiles.tar.gz. Can go in at any time after 3.
• 5. could changes to Client go in here ? Need to test
• 6. Modify Handler to use new DryRun.py instead of DryRunUploader.py
• 7. Deploy new Client if not done at 5.
• 8. take this chance to also fix complete transitioning preparelocal to args via JSON CRABClient#5288 possibly adding the new tarball

[belforte]

OK. Let's deploy the change to DagmanCreator ASAP so we can push new client in production.

[belforte]

back to this. Code from Wa is in https://github.com/novicecpp/CRABServer/tree/get_sandbox_in_schedd
I have now imported that branch in my repo: https://github.com/belforte/CRABServer/tree/Sandbox-in-Sched-from-Wa

[belforte]

1 and 2 done in #8740
3 done in #8743

will do 4 once those are tested and merged
4 done in #8745

[belforte]

doing 4. I found a bug in #8740 which was only affecting sandbox existance check in new code.
Will fix in same PR as to push 4.

But I have also found that we also still need debug_files.tar.gz to be moved around via TW, which creates confusion. Should change AdjustSites.py to download that and create that debug_files directory at bootstrap time.
Better yet find a way to upload them as files to S3, like down for twlog, but that may require extensve changes to Client, ServerUtilities and RestCache. Current code supports uploading of a single tarball using objectype=debugfiles. But we want 3 separate files for config, pset and scriptExe

[belforte]

let's go with a:

• 4.5 expand debug_files.tar.gz in AdjustSites and stop handling it in DagmanCreator - remove debug_files handling in TW. Do it in scheduler #8747

[belforte]

at the moment preparelocal in current client does not work with new TW and new client does not work with current TW. I need to make at least one of them backward compatible.

[belforte]

I decided to go in steps.

• deploy a Client compatible with new handling of tarballs via S3 add support for tarballs in S3, not on webdir CRABClient#5340
• deploy TW changes done until now
• do further changes for dryrun and do not worry if it stays broken until client is updated. Or maybe simply deprecate the "run" part of submit --dryrun and tell uses to use preparelocal and run_job

@aspiringmind-code @novicecpp I will gladly get your advice if you feel like suggesting something different