fleetctl: support overwrite command

[wuqixuan]

If unit file modified, can use "overwrite" command to update.
First stop and remove the old unit, and submit the new unit and restore
the status as the previous one.

Fixes #760

[bcwaldon]

Replace the description with this:

Overwrite stops (if needed), unload (if needed), and destroys, then submits, loads (if needed) and starts (if needed) the unit after replacing the unit file with that found on the local filesystem

[bcwaldon]

@wuqixuan you don't actually have to wait for units to transition through the loaded state on teardown or startup if that isn't the desired final state. You can trust that fleet will drive units towards the desired state, regardless of what intermediate state there may be.

[dzrw]

@bcwaldon Can you merge this? The changes you've requested (last year) just deal with copy, and not functionality. I'm sure the people who really want/need restart (overwrite) capabilities can deal with some bad text.

[esatterwhite]

As a user of fleet, I think the primary use case here would be rolling update that didn't have to bring down each instance of a running container. If I have service@1, service@2, and service@3 I don't want to have to stop all of them. I'd really want to update the unit file inplace, And then restart them 1 by 1. And honestly, I'd be happy to do the restarting manually, that isn't difficult to do.

But having to stop & destroy all of the running services before an update can be applied is a bit of a show stopper.

[jonboulle]

To recap:

• the fleet agent already supports "replacing" units of a given name by comparing their current local state with that of the unit in the registry:

  fleet/agent/reconcile.go

  Line 256 in 81d6c4a

  if cJHash != dJHash {

• hence, there's no need for fleetctl to do individual stop/destroy/load operations, as this PR currently does (indeed, that would not really be any different to just doing a fleetctl stop foo && fleetctl destroy foo && fleetctl load foo && fleetctl start foo(
• what we need to do is allow fleetctl to overwrite a unit in-place - the agent will then do the necessary work to actually replace that unit on the system
• in terms of user experience with fleetctl, it could behave similarly to how start/stop do today, polling by default (unless --no-block is passed):

  fleet/fleetctl/start.go

  Line 77 in 81d6c4a

  if !sharedFlags.NoBlock {

  
  but instead of only testing for the unit to reach a given state, it would also need to check that the unit's hash reaches the desired value (of the replacement)

[kayrus]

I've implemented bash based POC which works with the fleet v0.11.5.

Didn't test it with templates.

[antrik]

See also #1442 for a slightly different approach (regarding user interface) for this functionality, which seems worth considering.

[jonboulle]

Per OOB discussion, let's proceed with a --replace (or --overwrite) flag for start/submit/load. It'll probably look quite similar to #1442 but needs a bit more work to ensure we can sanely block (looking for the new hash) etc

[tixxdz]

Closing this in favor of #1509

@wuqixuan thank you for the patch, could you followup or test #1509 , thanks!