Add maintenance mode hooks

[silviabotros]

In the interest of more automation, it would be great if when a server is placed in maintenance mode within orchestrator, to also support a hook there. That way placing a server in maintenance does all the work of notifying a chat room, disabling the server undergoing maintenance in any load balancers and even killing connections if we decide to add that to said hook

[shlomi-noach]

Thoughts for cnsideration:

• I'm guessing such hooks need to be synchronous
• should we wait indefinitely until a hook returns?
• should we abort operation if a hook results with error?

End-maintenance is sometimes aborted. For example, the user ctrl-c an
operation in command line mode. There is a background polling that clears
maintenance flag. Should this also invoke post-maintenance ? Synchronously?
What happens on failure?

On Friday, October 2, 2015, Silvia Botros [email protected] wrote:

In the interest of more automation, it would be great if when a server is
placed in maintenance mode within orchestrator, to also support a hook
there. That way placing a server in maintenance does all the work of
notifying a chat room, disabling the server undergoing maintenance in any
load balancers and even killing connections if we decide to add that to
said hook

—
Reply to this email directly or view it on GitHub
#114.

[silviabotros]

We can add a timeout for the hook after which orchestrator shows a message that the hook failed, log a more detailed stack as to what line in the script failed. I wouldn't want this to be a blocker to making it clear this server was intended to be in maintenance so marking maintenance mode in orchestrator only is I think ok (at least shows that someone tried.....and it didn't go 100% as planned)

yes if the reaping maintenances is defined/configured, it should also run the hooks when the maintenance status is reaped. that way the same channels are utilized. I think it is fair to say that as long as behavior on hook failure is properly documented, people can decide whether the hook should be trivial like chat ops..or more involved like actually calling out to load balancers and making changes there...

[shlomi-noach]

Need to rewrite EndMaintenance() such that it invokes an 'onEndMaintenance' hook.

EndMaintenanceByInstanceKey() should do a two-step:

• read-maintenance-key
• EndMaintenance()

ExpireMaintenance() should do a two-step:

• read maintenance keys
• one by one issue EndMaintenance()

Make sure to handle the end_timestamp column value correctly (can use GREATEST() function)

[shlomi-noach]

omg this is almost two years old. Might actually do this.

[silviabotros]

woohoo!

Silvia Botros

On August 30, 2016 at 1:05:59 PM, Shlomi Noach ([email protected])
wrote:

omg this is almost two years old. Might actually do this.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#114 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABQwyjO_eSDk0RaxCh9ouiO3JRI5fjtVks5qlGL0gaJpZM4GIOJa
.