Runbook for handling uptime alerts #116
Conversation
| kubectl --as cluster-admin get xvshn[TAB here for specific service] | egrep $instanceNamespace_generated_chars # also describe to read what happened | ||
| ---- | ||
|
|
||
| .Check logs of our comp-functions |
There was a problem hiding this comment.
I doubt this is helpful to anyone who's never worked with our AppCat.
Also the comp-functions don't necessarily say anything about why a specific service is not working.
| kubectl -n syn-crossplane logs deployments/function-appcat-aeb2dbb03cf6 # <--- this number changes regularly | ||
| ---- | ||
|
|
||
| For stuck resources, You can create dummy label on object and then rollout restart crossplane function-appcat and provider-kubernetes. |
There was a problem hiding this comment.
Someone who has never used Crossplane might not know what a "stuck" resource is.
| ---- | ||
| #$instanceNamespace_generated_chars can be obtained in a way: `echo vshn-postgresql-my-super-prod-5jfjn | rev | cut -d'-' -f1 | rev` ===> 5jfjn | ||
| kubectl --as cluster-admin get objects | egrep $instanceNamespace_generated_chars # here look for False objects and describe them to find out what is wrong | ||
| kubectl --as cluster-admin get xvshn[TAB here for specific service] | egrep $instanceNamespace_generated_chars # also describe to read what happened |
There was a problem hiding this comment.
I'd probably go with describe from the beginning and not just mention it in a side note. It shows more information and the events.
There was a problem hiding this comment.
I changed to that and also added a real life example, so it's going to be easier for engineers to go through our abstractions
There was a problem hiding this comment.
I still shows as kubectl --as cluster-admin get xvshn for me not kubectl --as cluster-admin describe xvshn
wejdross
force-pushed
the
runbook_sla_alert
branch
from
1e2a23b to
619e004
Compare
Summary
Checklist
change,decision,requirement/quality,requirement/functional,dependencyas they show up in the changelog