Propose queue changeTaskRunPriority

[lotas]

Proposal:

https://github.com/taskcluster/taskcluster-rfcs/blob/276a15216e8ebe3b7976f6770ad10813a32c55fb/rfcs/0190-queue-change-task-run-priority.md

[jcristau]

I was wondering if there's a case to be made for the source and target priorities being included in the scopes.

[lotas]

Good point!
source <> target would be 8 to 8 permutations, maybe too much?

We can, however, add more generic ones like lowering or raising the priority, so two separate scopes?
queue:lower-task-run-priority:<taskId>
queue:raise-task-run-priority:<taskId>

[Archaeopteryx]

Have there been considerations to set this at the task graph level?

• This should prioritize the whole graph and ensure its timely execution else potentially backlogged tasks need to be identified and configured manually.
• A release graph can dependent on the tasks in a different one, e.g. the tasks scheduled automatically for a push can be dependencies of release graph created by the hook (fired by cron job or manually) to actually do a release (which has additional tasks).

[jcristau]

Have there been considerations to set this at the task graph level?

* This should prioritize the whole graph and ensure its timely execution else potentially backlogged tasks need to be identified and configured manually.

* A release graph can dependent on the tasks in a different one, e.g. the tasks scheduled automatically for a push can be dependencies of release graph created by the hook (fired by cron job or manually) to actually do a release (which has additional tasks).

I would argue that task (or run) is the right granularity for the queue service. If we need to do this for a whole task group that can likely be built on top of the fine-grained API.

[lotas]

(sorry, pr was closed by accident)

[bhearsum]

Writing down a few things we talked about synchronously:

• If the run is where the adjusted priority is to be stored, we should do something to make sure reruns pick up the priority of the previous run (to make sure bumped priorities aren't lost on rerun).
• Here are a few concrete use cases for this:
  • We have a high priority patch (eg: a potential fix for a chemspill bug) running tasks on Try. We want to bump the priority of all tasks in that task group.
  • We have a chemspill release or high priority nightly to get out. We want to bump the priority of all tasks in one or more task groups.
  • We are waiting for a specific task in a low resource pool (eg: mac hardware). We want to bump the priority of that specific task.

[lotas]

thanks @bhearsum for the summary

* If the `run` is where the adjusted priority is to be stored, we should do something to make sure reruns pick up the priority of the previous run (to make sure bumped priorities aren't lost on rerun).

indeed, priority on reruns should be "inherited"

* Here are a few concrete use cases for this:
  * We have a high priority patch (eg: a potential fix for a chemspill bug) running tasks on Try. We want to bump the priority of all tasks in that task group.
  * We have a chemspill release or high priority nightly to get out. We want to bump the priority of all tasks in one or more task groups.
  * We are waiting for a specific task in a low resource pool (eg: mac hardware). We want to bump the priority of that specific task.

So for the first two scenarios I wonder, why not simply create whole task group with the highest priority already, since it is known in advance this is very important? Or try pushes with raised priority is not possible?

For the third example, it's worth adding that if there are multiple tasks pending with same priority, we would need to lower the priority for the "not so important" ones, to let important one go sooner

[bhearsum]

* Here are a few concrete use cases for this:
  * We have a high priority patch (eg: a potential fix for a chemspill bug) running tasks on Try. We want to bump the priority of all tasks in that task group.
  * We have a chemspill release or high priority nightly to get out. We want to bump the priority of all tasks in one or more task groups.
  * We are waiting for a specific task in a low resource pool (eg: mac hardware). We want to bump the priority of that specific task.

So for the first two scenarios I wonder, why not simply create whole task group with the highest priority already, since it is known in advance this is very important? Or try pushes with raised priority is not possible?

That's a good idea actually, yeah. We'd have to figure out how to wire that into UIs & CLIs, but that's a very tractable problem.

[lotas]

@bhearsum @Archaeopteryx @jcristau I have reworked this document a bit, since it is clear that operating on the task group level is very important.
It was also missed out by me initially that tasks runs could fail and new runs wouldn't necessarily pick up same priority.

I'm changing this to work on both task and task group levels, and it would do "best effort" to change priority of scheduled and not yet scheduled tasks

[bhearsum]

It's hard to be certain if we'll use them, but I like this break out of raising/lowering - nice addition!

[lotas]

now that I think about it.. maybe it's too dangerous. If someone can lower the priority of a task that means that high-priority tasks might be de-prioritized.

@jcristau suggested to include existing and new priority in the scope as well, maybe we should do that instead

queue:change-task-priority:<newPriority>:<existingPriority>:<taskId> // to allow easier wildcard scopes
queue:change-task-group-priority:<newPriority>/<taskGroupId>

wdyt?

[bhearsum]

I think either are fine, but it may affect who we grant these scopes to depending on the implementation.

[lotas]

I'm leaving audit side of this out of scope, since we don't have anything similar on the platform yet.

However, it would probably be necessary to leave traces in the form of publishing events.
I would probably add two new exchanges to notify whoever is interested in such changes

[lotas]

I know other events do not include this, but maybe we should attach actor/user to those events?

[petemoore]

Regarding scopes, if we can reuse existing scopes, to avoid privilege escalation, that would be good. In other words, in general, if your action has the effect of scheduling a task with a given priority, but you would normally not have the scopes required to execute a task in that queue with that same priority, this could be considered a form of privilege escalation. If execution with a given priority always requires a particular scope, they should be no way to subvert the system via a different path.

So e.g. how about in order to change priority of a task to priority <p> you need to have either scope (anyOf):

queue:create-task:<priority>:<taskQueueId>
queue:change-priority:<priority>:<taskQueueId>
queue:change-task-group-priority:<priority>:<schedulerId>/<taskGroupId>

I'm not too fond of a scope that just allows increasing or decreasing priority since there is no way to limit the priority to a particular value.

[bhearsum]

Regarding scopes, if we can reuse existing scopes, to avoid privilege escalation, that would be good. In other words, in general, if your action has the effect of scheduling a task with a given priority, but you would normally not have the scopes required to execute a task in that queue with that same priority, this could be considered a form of privilege escalation. If execution with a given priority always requires a particular scope, they should be no way to subvert the system via a different path.

I see what you're getting at, but "privilege escalation" seems a bit extreme. With just these new scopes you cannot affect what runs, just when it runs. I suppose it's theoretically possible that this could be exploited, but it seems rather farfetched.

So e.g. how about in order to change priority of a task to priority <p> you need to have either scope (anyOf):

queue:create-task:<priority>:<taskQueueId> queue:change-priority:<priority>:<taskQueueId> queue:change-task-group-priority:<priority>:<schedulerId>/<taskGroupId>

I think is probably fine. The new change-priority scopes allow for addressing the use case of granting someone just the ability to change priorities if it comes up.