Refinery: Add optional support for priorityClassName

[micah-adams-focused]

Is your feature request related to a problem? Please describe.

The honeycomb chart template allows for optional configuration of a priorityClassName in the daemonset.yaml manifest. This seems like a useful parameter to introduce for refinery resources as well, given the critical nature of scheduling refinery dependent resources on a cluster.

Describe the solution you'd like

It seems like an optional configuration for priorityClassName in the refinery manifests would be beneficial for scheduling critical refinery workloads. I'd like to see if we can support a similar pattern for setting this value in refinery deployments.

Describe alternatives you've considered

There is quite a bit of fine grained control that is already offered in the helm chart, and I can see how setting a mix of resource requests/limits, antiAffinity, disruption budgets, and node selectors can achieve more complex resiliency settings for a complex refinery deployment. I think the addition of supporting an optional priorityClassName can help when there needs to be a coarser grained setting for ensuring the scheduling of refinery based workloads, while not disrupting the other fine grain controls we already have in the chart.

Additional context

[JamieDanielson]

@micah-adams-focused that sounds like a reasonable idea. We are open to having this feature added to the Refinery chart, and will put this on our backlog. We would also welcome a PR from this if you'd like to work on it sooner!