[ENHANCEMENT] Refactor the job scheduler locking system to lock arbitrary resources

[downsrob]

Is your feature request related to a problem?
The job scheduler locking system is currently limited to locking on jobID using context objects from scheduled jobs, and is scoped to simply preventing concurrent execution of the same job. By allowing the jobID to be set when acquiring a job scheduler lock, the locking system may be extended to locking any provided string ID. The motivating use case for this extension is the Index State Management Shrink action (issue, PR), which needs to lock specific nodes to prevent multiple shrink actions from attempting to move and resize an index on the same node.

What solution would you like?

• Add a new LockService method to enable acquiring locks using any provided String Enables locking on an arbitrary lockID #164
• Rename the jobID to lockID throughout the job scheduler locking system to reflect the change in semantics
• Create a system for cleaning up stale locks. Normally, when a job scheduler job is deleted, the job lock will be deleted with it. With the locking system decoupled from jobs, stale locks which aren't deleted could persist in the lock index indefinitely.

What alternatives have you considered?
Adding an alternative lock specifically for resources. As the only difference between the two locking systems, resource and job, would be the semantics of what the lockID represents, it would be a confusing redundancy.

Creating a locking system within ISM to keep the job scheduler locking system limited to job locks. The proposed change provides a new and more powerful abstracted locking system which could be useful for any job scheduler user wishing to lock resources across jobs. This locking system could have been limited to ISM, but its use cases seem to be generic enough to warrant inclusion in the job scheduler.