-
-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[🐛 Bug]: Selenium HUB + selenium-chrome-node memory not releasing when using proxy #1924
Comments
@gerardo8al, thank you for creating this issue. We will troubleshoot it as soon as we can. Info for maintainersTriage this issue by using labels.
If information is missing, add a helpful comment and then
If the issue is a question, add the
If the issue is valid but there is no time to troubleshoot it, consider adding the
If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C),
add the applicable
After troubleshooting the issue, please add the Thank you! |
Please try the latest release, I remember there were some memory fixes. |
Tried latest release. Same results. Also noticed, when memory starts going up and not released, my current sessions start throwing this error:
while new ones that get picked up after throw the |
I misread the use case. Have you checked what uses all that memory when you run with a proxy? Also, why don't you set the proxy to the container instead? |
Have not set up the proxy on container level because I have different proxies for different use cases, and these are set at the application level based on some logic. I have not yet checked what is using up the memory. Any tips on how to check this? 😬 🙏 |
Go inside the container and inspect the processes. |
These are the running processes. I see plenty of chrome processes so this worries me a bit...
This is my
Not sure how to interpret this further or continue debugging, suggestions appreciated. |
Why are there so many Chrome processes open? Are you not closing the session? Does this happen when you do not use the proxy? |
Not sure why the processes are kept open. I just see this behaviour in the proxy version. The closing of sessions is the same in the versions with and without proxy. The code that does the crawling is the same for both versions, the only difference is in the driver code for proxy. |
I see. I am trying to figure out how to help because something in the proxy is causing Chrome to stay open. There is little we can do. You should probably run the whole browser (non-headless) and see what is happening. Aside from that, I am out of ideas. |
I also encountered this issue and eventually found out that the reason was a conflict between headless and proxy with auth |
I am closing this because there is nothing for us to do. |
Hello @gerardo8al did you find a solution since ? |
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
What happened?
My setup using Selenium Hub with 4(min) replicas of selenium-chrome-node is not releasing memory when using proxy with authentication and eventually becomes unusable.
A little bit more details about the use case.... Every hour I have a process that triggers tasks that open around 400 to 600 URLs in total. It takes around 10 minutes to run them all.
I need to add an authenticated proxy to my selenium. When I deploy my proxy branch, during the first few triggers it takes around the same time, but it gets slower and slower, eventually leading to the crash explained below.
Once I add the proxy with authentication (link of what I did here), after some time (varies but usually around the 7th trigger), the memory of my chrome pods stops being freed and it eventually leads to the next crawlings not being able to create new sessions and throwing
SessionNotCreatedException
.Below code snippets relevant to this:
WebDriver without proxy
WebDriver with proxy
Simplified example of how my workers open the URLs
Grafana dashboards
The dashboard below shows memory usage WITHOUT proxy. (note: for this test triggered more than once an hour sometimes to see if memory usage would increase)
Grafana dashboard showing memory usage WITH proxy
(note: for this test we triggered more than once an hour sometimes to see if memory usage would increase. Memory stopped going down after 6th trigger)
Command used to start Selenium Grid with Docker
Relevant log output
Operating System
Ubuntu 20.04.5 LTS (Focal Fossa)
Docker Selenium version (tag)
4.10.0-20230607
The text was updated successfully, but these errors were encountered: