nymms should gracefully fail without permissions

[sethlindberg]

I wanted to check out the ami! It looks pretty cool - only I hadn't set up an IAM profile yet and I noticed in syslog this:

Oct 18 23:07:19 ip-10-20-242-215 kernel: [9675236.422718] init: nymms-scheduler main process ended, respawning
Oct 18 23:07:19 ip-10-20-242-215 kernel: [9675236.890671] init: nymms-reactor main process (12129) terminated with status 1
Oct 18 23:07:19 ip-10-20-242-215 kernel: [9675236.890734] init: nymms-reactor main process ended, respawning
Oct 18 23:07:20 ip-10-20-242-215 kernel: [9675237.036472] init: nymms-probe main process (12130) terminated with status 1

it seems like it'd be an easy fix to have nymms fail a bit more gracefully if it doesn't have permissions to do what it needs to do.

[phobologic]

So this is probably where I need a little documentation. Right now the logs for the daemons goes straight to stdout, mostly because I've been developing for ubuntu/upstart. If you look in /var/log/upstart you'll see a log per daemon that has the actual exception output. To be fair I should probably catch that output and make it a bit more user friendly, so I'll definitely keep that in mind.

As far as the messages you're seeing in syslog - unless I force messages into syslogd, and those messages are then routed by the hosts syslog configuration into the syslog logfile, there's not a ton I can do. The messages you're seeing there are from upstart (which is ubuntu's new init) which has pretty high priority to share it's messages throughout the logs in most cases.

In the end I want the daemon to die and then be automatically restarted by upstart (which is what you're seeing there) whenever there's a catastrophic problem. That way when/if you do provide a role (or user credentials for a user with the correct permissions) for the instance it will pick it up on the next restart by upstart.

I'm not currently sure if there's anyway for me to get boto to reload it's various credential endpoints WITHOUT restarting the process, which is what I'd need for the daemons to continue running after a failure like this.

Let me know what you think, and thanks for the first non-me bug :)

[sethlindberg]

Hmm, ok. It looks like upstart restarts the service 11 times in quick succession and then fails (over the course of 3 seconds) - is this what you want?

Are you tabling amazon linux/redhat support for now? (that would make sense)

[phobologic]

Yeah, those are the defaults for Upstart. It becomes more useful when there's an actual failure that can be recovered from.

I'm only tabling redhat support in as much that I don't have the experience in creating packages for that OS. I may eventually do that - but for now I'm sticking with whats easiest. If you want to create a RedHat package/startup scripts go for it!