Job Concurrency - When running, block all others. How?

[dinis-rodrigues]

All my jobs have a unique tag {“single_run_per_job”: <job_name>} which prevents the same job from running more than once concurrently. In my dagster.yaml

tag_concurrency_limits:
      - key: single_run_per_job
        value:
          applyLimitPerUniqueValue: True
        limit: 1

What I would like to achieve is having a job that if it is running, no other job can be running as well.
Is this possible to achieve?

For example, I would like to have a job that performs optimize and vacuum to my tables, and I don't any other job running at the same time.

[OwenKephart]

Hi @dinis-rodrigues, unfortunately this sort of concurrency limit is not supported in a convenient way at the moment -- all tag keys are treated equivalently, and so assuming there's only a single tag key (e.g. single_run_per_job), there's no way to have an instance of one specific job prevent all other jobs from running while still allowing any other combination of jobs to proceed in parallel.

Any solution here would need to have one distinct tag key per job, e.g. something like each job having {f"{job_name}_concurrency_tag": True}, and then the vacuum job would need to have all of those tags. This has obvious drawbacks, but is likely the only workaround at the moment.

[dinis-rodrigues]

Not an ideal solution, but I need to implement this. Thank you!

[bernardocortez]

This is very handy. I have some job I would like to stall all others too. Is there a way for this already?

[garethbrickman]

Could you elaborate what you mean? Please feel free to use pseudo-code snippets to illustrate

[bernardocortez]

I would like to be able to launch a job that runs some schema migration upon deployment of a new version of my repository

[davidalbertonogueira]

Could you elaborate what you mean? Please feel free to use pseudo-code snippets to illustrate

@job(
    resource_defs=prod_adls2,
    op_retry_policy=default_retry_policy,
    tags={"single_run_per_job": "apply_schema_migrations"},
)
def apply_schema_migrations_job():
    schema_migrate()

Like this. I want to make sure this runs when i deploy, and i don't want any other job to start before this one finishes.

[garethbrickman]

I believe the only approach is as answered above: to try to use run queue prioritization rules.