Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redis error during high traffic times #233

Open
brostwalt87 opened this issue Mar 2, 2023 · 0 comments
Open

Redis error during high traffic times #233

brostwalt87 opened this issue Mar 2, 2023 · 0 comments

Comments

@brostwalt87
Copy link

We are currently running authservice using AWS Elasticache as a backing store. During high traffic hours, we often see this issue in the logs

[2023-03-02 15:14:44.482] [console] [trace] hmget: redis connection closed error, retrying
[2023-03-02 15:14:44.482] [console] [trace] hmget: redis connection closed error, retrying
[2023-03-02 15:14:44.483] [console] [trace] hmget: redis connection closed error, retrying
[2023-03-02 15:14:44.483] [console] [error] hmget: redis connection closed error, throwing error

I have looked into different ways of tweaking the Elasticache server, including upgrading the instance to cache.m5.large. So pure resource consumption does not seem to be an issue. The elasticache server is running 2 replicas - a primary and secondary. Also, authservice is running a deployment with 2 pods, but has the ability to autoscale to 3. Resources do not seem to be an issue there

Also we have the following set on redis/elasticache:
appendonly no
maxmemory-policy allkeys-lru

Would like to see if there is a way to either optimize redis or increase the number of retries via a variable that can be added in at runtime

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant