adding safeguard alert

[viennaa]

• there was no critical alerting when vROps is gone entirely
• playbook still needs to be added here before merging

The last part is crucial, please add the playbook before merging!

[sapcc-bot]

Failed to validate the Prometheus rules. Details. Readme.

[richardtief]

I would vote for adding the absent condition to the alert above. There is a VropsAPIDown alert already. The condition is not appropriate anymore, too.

(
  sum by (target) (vrops_api_response) 
  / 
  count by (target) (vrops_api_response) > 500
)
or absent(vrops_api_response)

[viennaa]

Initially I thought the same, however I would not like to add it to the existing alert anymore, since it uses $labels.target in meta and summary, which won't be filled in case of the absent condition hitting.

[viennaa]

Another thought: Sunny wanted this as a warning. So we can give it the same name with warning, different messages and would be good to go.

[himanip94]

I am not sure why are we keeping the alert as a warning if all collectors are down. Alternatively, we could change the previous alert to warning and set the new one to critical, using the same alert name?

[viennaa]

To me both of them are worth critical. The recent case was without noticing since October, 9th.
I would generally tend to increase the for condition to 2h here, so it is a rather slow firing alert.

Sunny's argument was, that this can easily fire during maintenance activities. If these activities would take more than 2h, it would be wise, to silence the alert beforehand. I can give a quick guidance how to do this preventatively.

[himanip94]

I agree to silencing alerts before maintenance activities, that way can keep both as critical.

[sunnyrarora]

Silencing is one way, but here i believe we are increasing every next alert to critical in that sense, it will delete purpose of monitoring. InfraOps onduty primary task is to handle MIMs, i dont think it would be wise to involve him/her on troubelshooting monitoring tasks. here secondary can assist better. thats my two cents, you are free to keep it primary, but again one needs to sure if they are ready to handle it.

[viennaa]

Just to keep in mind: there will be no critical alerts fired from affected vc if a node is down or anything else happens. Technically nothing in production is affected at this time, but if something would be affected from then on, no alerting takes place.

Maybe you can take this into your meeting and come back with a decision here.

[viennaa]

any decision?

[himanip94]

Will discuss this in Tuesday monitoring call