how (not) to specify job runtimes

[andre-merzky]

Hi all,

we are running a private flux instance within a job allocation and scheduler smaller tasks for execution onto the allocation's compute nodes. It happens to be the case that we actually do not know the expected runtimes of the tasks, and can't even do an educated guess. For an example, for a recent biomolecular simulation workload we have been running, max observed runtimes have been 2 magnitudes larger than mean task runtime, meaning that we have to handle very long tail distributions. And in general, many of the science codes we run terminate once some science objective has been reached (some model trained, some convergence reached, some molecules travelled far enough, etc) which rarely translates into an exact runtime estimate...

Now it seems to be the case that the Flux jobspec requires to specify a task duration - the jobspec is declared invalid otherwise. As Flux seems to kill tasks after that duration expires, we can't specify something like a mean expected duration - so we seem to be left with the only option so specify very large task durations to ensure tasks can run to completion.

My questions:

• is my understanding of the duration parameter correct, or do I miss a flag to avoid task termination on expiration?
• would it be possible to accept jobspecs w/o task durations being specified? I guess this will likely impact the scheduling algorithm - but our arbitrary long task runtimes are not helping there either I guess... :-P
• alternatively, would a relaxation of the duration parameter to estimated runtime possible, to still inform the scheduling algorithm, but at the same time to overrun safely in case the application code needs to do so?

Many thanks, Andre.

[grondo]

Hi @andre-merzky, sorry the documentation for this jobspec attribute is not very easy to find. Here's the current description of the duration attribute from RFC 14:

The value of the duration attribute is a floating-point number greater than or equal to zero representing time span in seconds. A duration of 0 SHALL be interpreted as “unlimited” or “not set” by the system. The scheduler will make an effort to allocate the requested resources for the number of seconds specified in duration.

So if I understand your question correctly, I think you want to set a value of 0 for your jobs, which will be considered "unlimited" by Flux.

[andre-merzky]

Ah, that's a much easier and simpler answer than I expected - I indeed missed that documentation!

Thanks, we'll use that.

[dongahn]

If duration key is 0, Fluxion scheduler currently sets the span for the jobspan to be 7 days which is the compiled-in max.

• https://github.com/flux-framework/flux-sched/blob/065224e7ce72b575abf9e0694824231788de5ee2/resource/traversers/dfu_impl.hpp#L86

• https://github.com/flux-framework/flux-sched/blob/065224e7ce72b575abf9e0694824231788de5ee2/resource/config/system_defaults.hpp#L33

Two implications:

1. If you run your Flux instance within an HPC system like LC's Lassen, your job will never get the walltime signal as the batch job limit will be much less than 7 days.

2. If you are specializing the queuing policy of your flux instance to be backfilling-capable, this can potentially negatively impact the schedule effectiveness.

[dongahn]

Also when we introduce fluxion resource config file support, you will be able to impose a different system max than the compiled-in max as well.