Quiesce sagas when parking the rack for updates

[jgallagher]

Saga nodes are neither required nor generally expected to be compatible across different software release versions. Our current mupdate-driven process is to park the rack then mupdate all sleds. But if we park the rack while there are still live sagas, we could accidentally start up a new software release that tries to resume a saga that was serialized on the prior release.

One way to address this would be to add a manual step in the rack update process that checks for any sagas in such a state prior to performing the update. But this is kinda awkward: a running Nexus might start a new saga at any given time (e.g., kicked off by some RPW on a periodic timer), so a check for sagas would only be valid if all Nexus instances were stopped first. An optimistic attempt at this might look like:

• Shut down all Nexus instances
• Check for live sagas
• If any live sagas exist, restart the Nexus it's assigned to
• Once it completes, shut down that Nexus
• Check again to ensure no additional sagas were created while the prior one was running (repeat as needed)

It would probably be kinder to our updaters to build something into Nexus + omdb that allowed an updater to put Nexus into a "no new sagas are allowed to start; let me know when all existing sagas are complete" quiesced state. Then the update procedure could be updated to:

• Use omdb to quiesce Nexus sagas
• Park the rack
• Continue as we normally do