Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rate-limiter in email send causes high-latency alerts #2688

Open
bboreham opened this issue Jul 1, 2020 · 1 comment
Open

Rate-limiter in email send causes high-latency alerts #2688

bboreham opened this issue Jul 1, 2020 · 1 comment
Labels
chore refinement/improvement of end user functionality; making things work better monitoring Changes to how we monitor Weave Cloud itself

Comments

@bboreham
Copy link

bboreham commented Jul 1, 2020

The rate-limiter seems to be doing its job, to slow down people trying to spam the sign-in page.
However this then shows up as a high-latency alert, e.g. on 2020-06-27:

2:46 AM
Weave CloudAPP Instance: Weave Cloud (prod)
(UsersLatency - WARNING FIRING) default/users: high latency
Impact: Users have a very slow experience logging in
99th percentile latency: 1.650
View firing alerts Playbook Dashboard
containerName: users
description: The users service has a 99th-quantile latency of more than 0.5 seconds for 5m.
@bboreham bboreham added monitoring Changes to how we monitor Weave Cloud itself chore refinement/improvement of end user functionality; making things work better labels Jul 1, 2020
@bboreham bboreham changed the title Rate-limited in email send causes high-latency alerts Rate-limiter in email send causes high-latency alerts Jul 1, 2020
@bboreham bboreham mentioned this issue Jul 30, 2020
Closed
@bboreham
Copy link
Author

Something similar happened 2021-04-23 https://weaveworks.slack.com/archives/C0CG6Q4BG/p1619179896270400

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
chore refinement/improvement of end user functionality; making things work better monitoring Changes to how we monitor Weave Cloud itself
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bboreham