Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Low Grade Memory Growth in Cache Controller Refresh #122

Open
josephpeacock opened this issue Jul 20, 2020 · 0 comments
Open

[BUG] Low Grade Memory Growth in Cache Controller Refresh #122

josephpeacock opened this issue Jul 20, 2020 · 0 comments
Labels
bug Something isn't working
Milestone

Comments

@josephpeacock
Copy link
Collaborator

josephpeacock commented Jul 20, 2020

Describe the bug
The updateCacheController hook is designed to delete and refresh the cache controllers for all tracked remote clusters. Right now, the goroutines allocated for the cache controllers aren't being released when this happens. Instead, they're being parked (https://golang.org/src/runtime/proc.go). Work is needed to figure out why these goroutines are being parked and resolve that issue. Made #123 to address the symptoms, but needs work to identify and fix the root cause.

Steps To Reproduce

Run admiral with at least one remote cluster (the more there are, the faster the leak) and pprof enabled, and take goroutine dumps periodically. You will see large (and increasing) numbers of parked goroutines and a corresponding memory increase.

Expected behavior
Admiral is able to refresh its controllers without seeing a permanent increase in parked goroutines or memory usage.

@josephpeacock josephpeacock added the bug Something isn't working label Jul 20, 2020
@benchi benchi added this to the v1.2 milestone Nov 23, 2020
@benchi benchi changed the title [BUG] Fix Memory Leak in Cache Controller Refresh [BUG] Low Grade Memory Growth in Cache Controller Refresh Nov 23, 2020
@benchi benchi modified the milestones: v1.2, v1.3 Nov 23, 2020
itsLucario pushed a commit to itsLucario/admiral that referenced this issue Aug 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants