Problem: Pipelines remaining idle even when there are many packages queued

[joel-simpson]

Enduro in the current DPS setup watches three directories:

dpj-ts
epj-ts
other-ts

It queues the packages as they appear in the directories. What seems to be an issue though, is the way these packages are distributed to the transfer slots. For example, if there are many epj-s queued before a batch of dpj-s, the dpj-s aren't picked up, even if the dpj pipeline is completeley idle.

The behaviour we're observing won't give us full utilization of all pipelines unless we somehow manage to distribute dpj-s and epj-s evenly in the queue, which is not really realistic. Consider the following scenario:

a dpj submission is queued
an epj submission is sent to archiving
the sip containing the “avleveringsliste” is queued in enduro, after all the dpj-s
epj sip creation does not start until the avl is archived, basically when the dpj submission is finished

The result is the epj and dpj submissions are prevented from running in parallell, and even if we manage to tweak it so both epj and dpj sips are created, we have to make sure they get queued evenly.

Ideally, Enduro should trigger transfers for a certain type of package, as long as that type of pipeline has free transfer slots, ignoring the transfer's overall position in the queue.