Add option to run multiple jobs in a row #71
Conversation
| @@ -1186,12 +1186,13 @@ def main(): | |||
| GcpBatch.run_combine_results_on_cloud(gcs_bucket, gcs_prefix, results_dir, do_timeseries) | |||
| else: | |||
| parser = argparse.ArgumentParser() | |||
| parser.add_argument("project_filename") | |||
| parser.add_argument("project_filenames", help="Comma-separated list of project YAML files to run.") | |||
There was a problem hiding this comment.
maybe removing the help part from here as it is in the argument section as well?
There was a problem hiding this comment.
I'm not sure what you mean... But I like the help string here because it shows up when you run buildstock_gcp --help:
...
positional arguments:
project_filenames Comma-separated list of project YAML files to run.
job_identifiers Comma-separated list of job IDs to use. Optional override of gcp.job_identifier in your project file. Max 48 characters.
...
| job_IDs = len(project_filenames) * [None] | ||
| if args.job_identifiers: | ||
| job_IDs = args.job_identifiers.split(",") | ||
| if len(job_IDs) != n_projects: |
There was a problem hiding this comment.
Is this capturing the project_id issue? like trying to see what is the likelihood of this error happening?
There was a problem hiding this comment.
This is just checking that if you give a list of project files and a list of IDs, they're the same length.
| @@ -1186,12 +1186,13 @@ def main(): | |||
| GcpBatch.run_combine_results_on_cloud(gcs_bucket, gcs_prefix, results_dir, do_timeseries) | |||
There was a problem hiding this comment.
Maybe we move this into develop at this point instead of gcp?
| batch.push_image() | ||
| batch.run_batch() | ||
| batch.process_results() | ||
| for project_filename, job_ID in zip(project_filenames, job_IDs): |
There was a problem hiding this comment.
Project-Job Pair Processing continues all the way to post-processing and then the next project-job pair is picked up or once the results are avilable in gcs, this will pick up the next job?
There was a problem hiding this comment.
Right now, this is just running one job completely (waiting for post-processing to finish), then starting the next one. We could potentially start the second job while the first is in post-processing, but I'm not doing that here.
|
Minimum allowed coverage is Generated by 🐒 cobertura-action against 07ccfae |
Allows passing in multiple project files to
buildstock_gcp. The projects are all validated upfront (to catch any issues quickly) then run sequentially.Possible future improvements: