How to update unit? (question related to fleetctl)

[ernado]

For example, when I run

~/fleet$ fleetctl submit stun.service 
WARNING: Unit stun.service in registry differs from local unit file stun.servic

I expect that unit will be updated, but it stays the same version.

Currently I reached the desired effect with

ernado@node:~/fleet$ fleetctl destroy stun.service 
Destroyed stun.service
ernado@node:~/fleet$ fleetctl submit stun.service
ernado@node:~/fleet$ fleetctl start stun.service 
Unit stun.service launched on xxxxx.../xx.xx.xx.xx

[bcwaldon]

@ernado It seems like you've answered your own question. What would you prefer to see?

[ernado]

@bcwaldon
I prefer something like

fleetctl upgrade stun.service
# or
fleetctl submit stun.service --force 

to make it with one command and/or some related docs.
For me it is not very obvious.

Or error instead of warning. Currently it feels like unit gets updated.
Thank you.

[lamielle]

Was there some out-of-band resolution on this request/question? Right now I'm developing a few systemd units. It's really painful to have to go through the "destroy, stop, load, start" cycle over and over. A fleetctl command that facilitates a quicker edit/run cycle would be very useful.

I've tried writing a single command that chains together a sequence of fleetctrl commands, but it seems they run too quickly and commands after "destroy" don't seem to take effect.

Surely Fleet/CoreOS devs have a better cycle than this when developing unit files... right?

[ernado]

Maybe it is not very common use-case and CoreOS team think that full cycle of "destroy, submit, start" is acceptable.

@lamielle, I've described the desired behavior of offered command "upgrade".
Are you agree with it, or maybe you have some additions?
We can develop together a proposal for needed command, providing detailed description.

[lamielle]

Given that Fleet explicitly doesn't support "rolling upgrades" I wonder if naming the command 'upgrade' will be contentious? Additionally, I'm all for keeping tools like fleetctl simple, so adding one more top level command may not be desirable by the Fleet team. Maybe a flag to one of the existing commands (like @ernado suggests) would be better.

It would be nice to hear from @bcwaldon about the following problem. If it is simply a bug in the client then addressing this issue would satisfy my dev workflow needs AND add no additional complexity to the fleetctl too:

• Edit a service file (redis.service as an example)

• Run the following: fleetctl destroy redis.service && fleetctl submit redis.service && fleetctl start redis.service

• Note on the CoreOS machine that the commands didn't appear to have any effect. The update unit was neither started nor submitted. However, the status reported by fleetctl appears to be correct:

  $ fleetctl destroy redis.service && fleetctl submit redis.service && fleetctl start redis.service
  Destroyed redis.service
  Triggered global unit redis.service start
  $ fleetctl list-units
  UNIT                    MACHINE                         ACTIVE  SUB
  redis.service           4a2d1c6f.../172.17.8.101        active  running
  $ fleetctl list-unit-files
  UNIT                    HASH    DSTATE          STATE           TARGET
  redis.service           7c9e2b0 launched        -               -
  

• However, if I run each command individually, the changes DO take effect, the unit is destroyed, re-submitted, and started.

Just being able to "chain" the fleetctl commands using shell && operators would address my biggest complaint. Sure, a built-in command would be nice, but it seems addressing this problem with the tool might be a good first step.

@ernado would this approach help you out at all?

[ernado]

@lamielle, it would be nice to somehow "chain" commands, ie run them synchronously. I'm pretty sure it will help in our use-case.
On other side, it may be more difficult to implement command chaining than one command, that solves only "upgrade" use-case.

[lamielle]

You may be right: the fleetctl is probably not running these synchronously and that may be the core issue with this. Which, to me, indicates a problem with the system as a whole: I would think that, if a client submits requests very quickly to the fleet daemon, that they would queue up and execute one-by-one on the host running the daemon. The results I see when chaining produce no logs (via journalctl -f) on the host related to fleet/systemd. But at this point I'm speculating on how fleet works under the hood.

As for a new command, what about "fleetctl restart --update "? This says "restart the specified unit AND since you passed --update, the client will destroy/submit the unit as well". Without the --update flag, fleetctl would ONLY stop/start the unit. The unit itself would not be destroyed and re-submitted. I like this approach since it supports two use cases with a single top level command. So you can both restart AND update the unit+restart.

What do you think, @ernado?

[bcwaldon]

@lamielle To address your issue with incorrect behavior of fleetctl destroy redis.service && fleetctl submit redis.service && fleetctl start redis.service: this should definitely work as you expect. If you are see otherwise, please file a bug.

While I'm more than happy to support what users actually want to do, we need to do it the correct way. Simply making fleetctl redeploy an alias for fleetctl destroy; fleetctl start isn't going to add a lot of value, since it cannot be implemented as an atomic operation. The API does not support changing unit file contents in-place. I honestly haven't thought through how to implement this properly yet, but I do want to chime in and say that if we can do it correctly, then I don't mind supporting it.

[bcwaldon]

@lamielle ...and @jonboulle just pointed out to me that fleetctl destroy isn't actually guaranteeing that your units shutdown before the command exists, which means you are likely encountering #866

[phemmer]

Running into this issue myself. I consider the ability to update unit files without service interruption to be fairly critical.
Lets say I've got a unit file called [email protected], which is running 3 instances ([email protected], [email protected], [email protected]). If I want to update the unit file I have to stop all 3 instances first. The way I'd like to do this is to update the unit file, restart one instance, verify it came up properly, proceed to the next instance, repeat.

This is type of action is fully supported by systemd. It's only fleet which prevents it.

[jonboulle]

@phemmer can you explain how fleet is preventing it? Essentially, you can achieve what you want today with the following, right?

fleetctl destroy [email protected]
fleetctl start [email protected]
.. wait ...
fleetctl destroy [email protected]
fleetctl start [email protected]
.. wait ...
fleetctl destroy [email protected]
fleetctl start [email protected]

[vaijab]

@jonboulle Yes, one can use the above example to achieve what @phemmer is trying to do.

The issue I have is that sometimes stop / start is enough, but other times unit file is changed and destroy / start is required.

[ernado]

@jonboulle, @vaijab

Sometimes I am unable to start units that are dead even without changing unit files and I am forced to destroy / start. Is it ok?

[jonboulle]

@ernado Yeah, that's a slightly separate issue related to what fleetctl start actually means - which, right now, is "set the target state of this unit to launched" (so if the target state is already launched, it's effectively a no-op).

We have been kicking around some ideas about clarifying this and perhaps tweaking the behaviour slightly to make it more intuitive - I thought there were some issues but perhaps not yet /cc @bcwaldon

[ernado]

@jonboulle
Maybe it is possible to make more verbose output in cases:

• start when unit is dead
• start when unit file need update

And updated docs.

Is it ok to keep this issue opened?

[jonboulle]

@ernado I agree, that's something we need to clear up too. Let's leave it open at least until we're confident we have other issues to track the next steps.

[irony]

fleetctl submit -f (or fleetctl start -f) would be great to update existing unit..

[vaijab]

fleetctl submit -f (or fleetctl start -f) would be great to update existing unit..

What should fleet do in this case? Restart service?

[irony]

fleetctl start -f web@1 

would refresh the unit file and restart the unit..

it would send the updated unit file (including template) and then restart the unit. when debugging startup of units it's very common to do fleetctl destroy web@1 && fleet submit web@1 && fleet start web@1 (to avoid the WARNING: Unit [email protected] in registry differs from local unit file [email protected])

[mauritsl]

I wrote a shell script that does the trick for me.

#!/usr/bin/env bash
fleetctl stop $1.service
fleetctl stop $1-discovery.service
fleetctl destroy $1.service
fleetctl destroy $1-discovery.service
UNIT_RUNNING=1
while [ $UNIT_RUNNING -ne 0 ]; do
  UNIT_RUNNING=`fleetctl list-units | grep $1.service | wc -l`;
  sleep 2;
  echo "Waiting for units to stop...";
done
fleetctl submit $1.service
fleetctl submit $1-discovery.service
fleetctl load $1.service
fleetctl load $1-discovery.service
fleetctl start $1.service
fleetctl start $1-discovery.service
sleep 1;
fleetctl list-units

Save the script with vi ~/restart.sh and use ~/restart.sh beanstalkd to restart a service. I've included lines to restart a -discovery (sidekick) service as well, but you can leave that lines out.

[bcwaldon]

See #760

[celevra]

what if i want to update the unit file but don't want to restart the service? Could be there are dependencies and i only want to update the unit file for the next reboot