[🐛 Bug]: Selenium HUB + selenium-chrome-node memory not releasing when using proxy #1924

gerardo8al · 2023-08-24T09:49:00Z

What happened?

My setup using Selenium Hub with 4(min) replicas of selenium-chrome-node is not releasing memory when using proxy with authentication and eventually becomes unusable.

A little bit more details about the use case.... Every hour I have a process that triggers tasks that open around 400 to 600 URLs in total. It takes around 10 minutes to run them all.

I need to add an authenticated proxy to my selenium. When I deploy my proxy branch, during the first few triggers it takes around the same time, but it gets slower and slower, eventually leading to the crash explained below.

Once I add the proxy with authentication (link of what I did here), after some time (varies but usually around the 7th trigger), the memory of my chrome pods stops being freed and it eventually leads to the next crawlings not being able to create new sessions and throwing SessionNotCreatedException.

Below code snippets relevant to this:

WebDriver without proxy

from selenium import webdriver
from selenium.webdriver.remote.webdriver import WebDriver
from app import settings


class Driver(WebDriver):
    """Webdriver with all necessary options set."""

    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument("--ignore-ssl-errors=yes")
    chrome_options.add_argument("--ignore-certificate-errors")
    chrome_options.add_argument("--blink-settings=imagesEnabled=false")
    chrome_options.add_argument(
        "--disable-dev-shm-usage"
    )  # This is a workaround until solved. See: https://stackoverflow.com/a/53970825

   chrome_options.add_argument("--headless=new")

    def __init__(self) -> None:
        """Instance driver will all necessary options."""
        super().__init__(command_executor=settings.COMMAND_EXECUTOR_URL, options=self.chrome_options)

WebDriver with proxy

import zipfile

from selenium import webdriver
from selenium.webdriver.remote.webdriver import WebDriver

from app import settings


class Driver(WebDriver):
    """Webdriver with all necessary options set."""

    manifest_json = """
    {
        "version": "1.0.0",
        "manifest_version": 2,
        "name": "Chrome Proxy",
        "permissions": [
            "proxy",
            "tabs",
            "unlimitedStorage",
            "storage",
            "<all_urls>",
            "webRequest",
            "webRequestBlocking"
        ],
        "background": {
            "scripts": ["background.js"]
        },
        "minimum_chrome_version":"22.0.0"
    }
    """

    background_js = """
        var config = {
                mode: "fixed_servers",
                rules: {
                singleProxy: {
                    scheme: "http",
                    host: "%s",
                    port: parseInt(%s)
                },
                bypassList: ["localhost"]
                }
            };
        chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});
        function callbackFn(details) {
            return {
                authCredentials: {
                    username: "%s",
                    password: "%s"
                }
            };
        }
        chrome.webRequest.onAuthRequired.addListener(
                    callbackFn,
                    {urls: ["<all_urls>"]},
                    ['blocking']
        );
    """ % (
        settings.PROXY_HOST,
        settings.PROXY_PORT,
        settings.PROXY_USER,
        settings.PROXY_PASSWORD,
    )

    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument("--ignore-ssl-errors=yes")
    chrome_options.add_argument("--ignore-certificate-errors")
    chrome_options.add_argument("--blink-settings=imagesEnabled=false")
    chrome_options.add_argument(
        "--disable-dev-shm-usage"
    )  # This is a workaround until solved. See: https://stackoverflow.com/a/53970825
     chrome_options.add_argument("--headless=new")

    def __init__(self) -> None:
        """Instance driver will all necessary options."""
        self._prepare_chromedriver(settings.USE_PROXY) #settings.USE_PROXY is True
        super().__init__(command_executor=settings.COMMAND_EXECUTOR_URL, options=self.chrome_options)

    def _prepare_chromedriver(self, use_proxy: bool = False) -> None:
        if use_proxy:
            pluginfile = "/tmp/proxy_auth_plugin.zip"

            with zipfile.ZipFile(pluginfile, "w") as zp:
                zp.writestr("manifest.json", self.manifest_json)
                zp.writestr("background.js", self.background_js)
            self.chrome_options.add_extension(pluginfile)

Simplified example of how my workers open the URLs

with Driver() as driver:
	for url of urls:
		driver.get(url)
		# Do some logic with result

Grafana dashboards

The dashboard below shows memory usage WITHOUT proxy. (note: for this test triggered more than once an hour sometimes to see if memory usage would increase)

Grafana dashboard showing memory usage WITH proxy
(note: for this test we triggered more than once an hour sometimes to see if memory usage would increase. Memory stopped going down after 6th trigger)

Command used to start Selenium Grid with Docker

My kubernetes cluster is deployed using the helm chart provided on [https://www.selenium.dev/docker-selenium](https://www.selenium.dev/docker-selenium) with the values provided below:

selenium-grid:
  global:
    seleniumGrid:
      imageTag: 4.10.0-20230607
  autoscaling:
    enabled: true
  ingress:
    enabled: false
  chromeNode:
    replicas: 4
    maxReplicaCount: 5
  firefoxNode:
    enabled: false
    deploymentEnabled: false
  edgeNode:
    deploymentEnabled: false
    enabled: false

Relevant log output

2023-08-22T09:31:32+02:00	Traceback (most recent call last):
2023-08-22T09:31:32+02:00	  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
2023-08-22T09:31:32+02:00	    R = retval = fun(*args, **kwargs)
2023-08-22T09:31:32+02:00	                 ^^^^^^^^^^^^^^^^^^^^
2023-08-22T09:31:32+02:00	  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery.py", line 249, in _inner
2023-08-22T09:31:32+02:00	    reraise(*exc_info)
2023-08-22T09:31:32+02:00	  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/_compat.py", line 60, in reraise
2023-08-22T09:31:32+02:00	    raise value
2023-08-22T09:31:32+02:00	  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery.py", line 244, in _inner
2023-08-22T09:31:32+02:00	    return f(*args, **kwargs)
2023-08-22T09:31:32+02:00	           ^^^^^^^^^^^^^^^^^^
2023-08-22T09:31:32+02:00	  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
2023-08-22T09:31:32+02:00	    return self.run(*args, **kwargs)
2023-08-22T09:31:32+02:00	           ^^^^^^^^^^^^^^^^^^^^^^^^^
2023-08-22T09:31:32+02:00	  File "/app/celery_tasks.py", line 87, in crawl_p_l
2023-08-22T09:31:32+02:00	    loop.run_until_complete(get_urls)
2023-08-22T09:31:32+02:00	  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
2023-08-22T09:31:32+02:00	    return future.result()
2023-08-22T09:31:32+02:00	           ^^^^^^^^^^^^^^^
2023-08-22T09:31:32+02:00	  File "/app/middleware/crawling/base.py", line 69, in start_crawling_session
2023-08-22T09:31:32+02:00	    await get_urls_with_selenium()
2023-08-22T09:31:32+02:00	  File "/app/middleware/crawling/base.py", line 104, in crawl
2023-08-22T09:31:32+02:00	    with Driver() as driver:
2023-08-22T09:31:32+02:00	         ^^^^^^^^
2023-08-22T09:31:32+02:00	  File "/app/utilities/driver.py", line 81, in __init__
2023-08-22T09:31:32+02:00	    super().__init__(command_executor=settings.COMMAND_EXECUTOR_URL, options=self.chrome_options)
2023-08-22T09:31:32+02:00	  File "/usr/local/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 286, in __init__
2023-08-22T09:31:32+02:00	    self.start_session(capabilities, browser_profile)
2023-08-22T09:31:32+02:00	  File "/usr/local/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 378, in start_session
2023-08-22T09:31:32+02:00	    response = self.execute(Command.NEW_SESSION, parameters)
2023-08-22T09:31:32+02:00	               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-08-22T09:31:32+02:00	  File "/usr/local/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 440, in execute
2023-08-22T09:31:32+02:00	    self.error_handler.check_response(response)
2023-08-22T09:31:32+02:00	  File "/usr/local/lib/python3.11/site-packages/selenium/webdriver/remote/errorhandler.py", line 245, in check_response
2023-08-22T09:31:32+02:00	    raise exception_class(message, screen, stacktrace)
2023-08-22T09:31:32+02:00	selenium.common.exceptions.SessionNotCreatedException: Message: Could not start a new session. java.util.concurrent.TimeoutException

Operating System

Ubuntu 20.04.5 LTS (Focal Fossa)

Docker Selenium version (tag)

4.10.0-20230607

The text was updated successfully, but these errors were encountered:

github-actions · 2023-08-24T09:49:12Z

@gerardo8al, thank you for creating this issue. We will troubleshoot it as soon as we can.

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

diemol · 2023-08-24T09:53:14Z

Please try the latest release, I remember there were some memory fixes.

gerardo8al · 2023-08-24T12:59:28Z

Tried latest release. Same results.

Also noticed, when memory starts going up and not released, my current sessions start throwing this error:

selenium.common.exceptions.WebDriverException: Message: unknown error: session deleted because of page crash

while new ones that get picked up after throw the SessionNotCreated

diemol · 2023-08-24T13:21:29Z

I misread the use case. Have you checked what uses all that memory when you run with a proxy?

Also, why don't you set the proxy to the container instead?

gerardo8al · 2023-08-24T14:07:35Z

Have not set up the proxy on container level because I have different proxies for different use cases, and these are set at the application level based on some logic.

I have not yet checked what is using up the memory. Any tips on how to check this? 😬 🙏

diemol · 2023-08-24T14:49:41Z

Go inside the container and inspect the processes.

gerardo8al · 2023-08-25T13:08:40Z

These are the running processes. I see plenty of chrome processes so this worries me a bit...

seluser@selenium-chrome-node/$ ps -eaf
UID          PID    PPID  C STIME TTY          TIME CMD
seluser        1       0  0 02:44 ?        00:00:00 bash /opt/bin/entry_point.sh
seluser        9       1  0 02:44 ?        00:00:10 /usr/bin/python3 /usr/bin/supervisord --configuration /etc/supervisord.conf
seluser       11       9  0 02:44 ?        00:00:00 bash /opt/bin/start-xvfb.sh
seluser       12       9  0 02:44 ?        00:00:00 bash /opt/bin/start-vnc.sh
seluser       13       9  0 02:44 ?        00:00:00 bash /opt/bin/start-novnc.sh
seluser       14       9  0 02:44 ?        00:00:00 bash -c /opt/bin/start-selenium-node.sh; EXIT_CODE=$?; kill -s SIGINT `cat /var/run/supervisor/supervisord.pid`; exit $EXIT_CODE
seluser       16      11  0 02:44 ?        00:00:00 /bin/sh /usr/bin/xvfb-run --server-num=99 --listen-tcp --server-args=-screen 0 1360x1020x24 -fbdir /var/tmp -dpi 96 -listen tcp -noreset -ac +extension RANDR /usr
seluser       21      13  0 02:44 ?        00:00:00 bash /opt/bin/noVNC/utils/novnc_proxy --listen 7900 --vnc localhost:5900
seluser       24      14  0 02:44 ?        00:00:00 /bin/bash /opt/bin/start-selenium-node.sh
seluser       33      16  0 02:44 ?        00:00:24 Xvfb :99 -screen 0 1360x1020x24 -fbdir /var/tmp -dpi 96 -listen tcp -noreset -ac +extension RANDR -auth /tmp/xvfb-run.iTGMgS/Xauthority
seluser       36       1  0 02:44 ?        00:00:00 pulseaudio -D --exit-idle-time=-1
seluser       41      21  0 02:44 ?        00:00:02 python3 -m websockify --web /opt/bin/noVNC/utils/../ 7900 localhost:5900
seluser       48      16  0 02:44 ?        00:00:04 /usr/bin/fluxbox -display :99.0
seluser       73      24  0 02:44 ?        00:03:05 java -Dwebdriver.http.factory=jdk-http-client -jar /opt/selenium/selenium-server.jar --ext /opt/selenium/selenium-http-jdk-client.jar node --bind-host false --con
seluser      101      12  0 02:44 ?        00:00:13 x11vnc -usepw -forever -shared -rfbport 5900 -rfbportv6 5900 -display :99.0
seluser    59834       1  0 07:06 ?        00:00:43 /opt/google/chrome/chrome --no-sandbox --allow-pre-commit-input --blink-settings=imagesEnabled=false --disable-background-networking --disable-client-side-phishin
seluser    59840   59834  0 07:06 ?        00:00:00 cat
seluser    59841   59834  0 07:06 ?        00:00:00 cat
seluser    59843       1  0 07:06 ?        00:00:00 /opt/google/chrome/chrome_crashpad_handler --monitor-self --monitor-self-annotation=ptype=crashpad-handler --database=/home/seluser/.config/google-chrome/Crash Re
seluser    59845       1  0 07:06 ?        00:00:00 /opt/google/chrome/chrome_crashpad_handler --no-periodic-tasks --monitor-self-annotation=ptype=crashpad-handler --database=/home/seluser/.config/google-chrome/Cra
seluser    59850   59834  0 07:06 ?        00:00:00 /opt/google/chrome/chrome --type=zygote --no-zygote-sandbox --no-sandbox --enable-logging --headless=new --log-level=0 --crashpad-handler-pid=59843 --enable-crash
seluser    59851   59834  0 07:06 ?        00:00:00 /opt/google/chrome/chrome --type=zygote --no-sandbox --enable-logging --headless=new --log-level=0 --crashpad-handler-pid=59843 --enable-crash-reporter=, --noerrd
seluser    59852   59851  0 07:06 ?        00:00:00 /opt/google/chrome/nacl_helper --no-sandbox
seluser    60581       1  0 07:07 ?        00:00:39 /opt/google/chrome/chrome --no-sandbox --allow-pre-commit-input --blink-settings=imagesEnabled=false --disable-background-networking --disable-client-side-phishin
seluser    60585   60581  0 07:07 ?        00:00:00 cat
seluser    60586   60581  0 07:07 ?        00:00:00 cat
seluser    60588       1  0 07:07 ?        00:00:00 /opt/google/chrome/chrome_crashpad_handler --monitor-self --monitor-self-annotation=ptype=crashpad-handler --database=/home/seluser/.config/google-chrome/Crash Re
seluser    60590       1  0 07:07 ?        00:00:00 /opt/google/chrome/chrome_crashpad_handler --no-periodic-tasks --monitor-self-annotation=ptype=crashpad-handler --database=/home/seluser/.config/google-chrome/Cra
seluser    60595   60581  0 07:07 ?        00:00:00 /opt/google/chrome/chrome --type=zygote --no-zygote-sandbox --no-sandbox --enable-logging --headless=new --log-level=0 --crashpad-handler-pid=60588 --enable-crash
seluser    60596   60581  0 07:07 ?        00:00:00 /opt/google/chrome/chrome --type=zygote --no-sandbox --enable-logging --headless=new --log-level=0 --crashpad-handler-pid=60588 --enable-crash-reporter=, --noerrd
seluser    60597   60596  0 07:07 ?        00:00:00 /opt/google/chrome/nacl_helper --no-sandbox
seluser    60637   60596  0 07:07 ?        00:00:00 /opt/google/chrome/chrome --type=utility --utility-sub-type=storage.mojom.StorageService --lang=en-US --service-sandbox-type=utility --no-sandbox --disable-dev-sh
seluser    63575       1  0 07:10 ?        00:00:34 /opt/google/chrome/chrome --no-sandbox --allow-pre-commit-input --blink-settings=imagesEnabled=false --disable-background-networking --disable-client-side-phishin
seluser    63579   63575  0 07:10 ?        00:00:00 cat
seluser    63580   63575  0 07:10 ?        00:00:00 cat
seluser    63582       1  0 07:10 ?        00:00:00 /opt/google/chrome/chrome_crashpad_handler --monitor-self --monitor-self-annotation=ptype=crashpad-handler --database=/home/seluser/.config/google-chrome/Crash Re
seluser    63584       1  0 07:10 ?        00:00:00 /opt/google/chrome/chrome_crashpad_handler --no-periodic-tasks --monitor-self-annotation=ptype=crashpad-handler --database=/home/seluser/.config/google-chrome/Cra
seluser    63589   63575  0 07:10 ?        00:00:00 /opt/google/chrome/chrome --type=zygote --no-zygote-sandbox --no-sandbox --enable-logging --headless=new --log-level=0 --crashpad-handler-pid=63582 --enable-crash
seluser    63590   63575  0 07:10 ?        00:00:00 /opt/google/chrome/chrome --type=zygote --no-sandbox --enable-logging --headless=new --log-level=0 --crashpad-handler-pid=63582 --enable-crash-reporter=, --noerrd
seluser    63591   63590  0 07:10 ?        00:00:00 /opt/google/chrome/nacl_helper --no-sandbox
seluser    63646   63590  0 07:10 ?        00:00:00 /opt/google/chrome/chrome --type=utility --utility-sub-type=storage.mojom.StorageService --lang=en-US --service-sandbox-type=utility --no-sandbox --disable-dev-sh
seluser    71731       1  3 08:05 ?        00:00:42 /opt/google/chrome/chrome --no-sandbox --allow-pre-commit-input --blink-settings=imagesEnabled=false --disable-background-networking --disable-client-side-phishin
seluser    71741   71731  0 08:05 ?        00:00:00 cat
seluser    71742   71731  0 08:05 ?        00:00:00 cat
seluser    71751       1  0 08:05 ?        00:00:00 /opt/google/chrome/chrome_crashpad_handler --monitor-self --monitor-self-annotation=ptype=crashpad-handler --database=/home/seluser/.config/google-chrome/Crash Re
seluser    71753       1  0 08:05 ?        00:00:00 /opt/google/chrome/chrome_crashpad_handler --no-periodic-tasks --monitor-self-annotation=ptype=crashpad-handler --database=/home/seluser/.config/google-chrome/Cra
seluser    71758   71731  0 08:05 ?        00:00:00 /opt/google/chrome/chrome --type=zygote --no-zygote-sandbox --no-sandbox --enable-logging --headless=new --log-level=0 --crashpad-handler-pid=71751 --enable-crash
seluser    71759   71731  0 08:05 ?        00:00:00 /opt/google/chrome/chrome --type=zygote --no-sandbox --enable-logging --headless=new --log-level=0 --crashpad-handler-pid=71751 --enable-crash-reporter=, --noerrd
seluser    71780   71759  0 08:05 ?        00:00:00 /opt/google/chrome/nacl_helper --no-sandbox
seluser    82512       1  8 08:21 ?        00:00:37 /opt/google/chrome/chrome --no-sandbox --allow-pre-commit-input --blink-settings=imagesEnabled=false --disable-background-networking --disable-client-side-phishin
seluser    82519   82512  0 08:21 ?        00:00:00 cat
seluser    82520   82512  0 08:21 ?        00:00:00 cat
seluser    82568       1  0 08:21 ?        00:00:00 /opt/google/chrome/chrome_crashpad_handler --monitor-self --monitor-self-annotation=ptype=crashpad-handler --database=/home/seluser/.config/google-chrome/Crash Re
seluser    82585       1  0 08:21 ?        00:00:00 /opt/google/chrome/chrome_crashpad_handler --no-periodic-tasks --monitor-self-annotation=ptype=crashpad-handler --database=/home/seluser/.config/google-chrome/Cra
seluser    82605   82512  0 08:21 ?        00:00:00 /opt/google/chrome/chrome --type=zygote --no-zygote-sandbox --no-sandbox --enable-logging --headless=new --log-level=0 --crashpad-handler-pid=82568 --enable-crash
seluser    82606   82512  0 08:21 ?        00:00:00 /opt/google/chrome/chrome --type=zygote --no-sandbox --enable-logging --headless=new --log-level=0 --crashpad-handler-pid=82568 --enable-crash-reporter=, --noerrd
seluser    82650   63589  0 08:21 ?        00:00:00 /opt/google/chrome/chrome --type=gpu-process --no-sandbox --disable-dev-shm-usage --enable-logging --headless=new --log-level=0 --ozone-platform=headless --use-an
seluser    82670       0  0 08:21 pts/0    00:00:00 /bin/bash
seluser    82681   82606  0 08:21 ?        00:00:00 /opt/google/chrome/nacl_helper --no-sandbox
seluser    82691   60595  0 08:21 ?        00:00:00 /opt/google/chrome/chrome --type=gpu-process --no-sandbox --disable-dev-shm-usage --enable-logging --headless=new --log-level=0 --ozone-platform=headless --use-an
seluser    82692   59850  0 08:21 ?        00:00:00 /opt/google/chrome/chrome --type=gpu-process --no-sandbox --disable-dev-shm-usage --enable-logging --headless=new --log-level=0 --ozone-platform=headless --use-an
seluser    82731   71758  0 08:21 ?        00:00:00 /opt/google/chrome/chrome --type=gpu-process --no-sandbox --disable-dev-shm-usage --enable-logging --headless=new --log-level=0 --ozone-platform=headless --use-an
seluser    82750   71759  0 08:21 ?        00:00:00 /opt/google/chrome/chrome --type=utility --utility-sub-type=storage.mojom.StorageService --lang=en-US --service-sandbox-type=utility --no-sandbox --disable-dev-sh
seluser    82751   59851  0 08:21 ?        00:00:00 /opt/google/chrome/chrome --type=utility --utility-sub-type=storage.mojom.StorageService --lang=en-US --service-sandbox-type=utility --no-sandbox --disable-dev-sh
seluser    82782   82606  0 08:21 ?        00:00:00 /opt/google/chrome/chrome --type=utility --utility-sub-type=storage.mojom.StorageService --lang=en-US --service-sandbox-type=utility --no-sandbox --disable-dev-sh
seluser    84414       1  5 08:22 ?        00:00:21 /opt/google/chrome/chrome --no-sandbox --allow-pre-commit-input --blink-settings=imagesEnabled=false --disable-background-networking --disable-client-side-phishin
seluser    84418   84414  0 08:22 ?        00:00:00 cat
seluser    84419   84414  0 08:22 ?        00:00:00 cat
seluser    84421       1  0 08:22 ?        00:00:00 /opt/google/chrome/chrome_crashpad_handler --monitor-self --monitor-self-annotation=ptype=crashpad-handler --database=/home/seluser/.config/google-chrome/Crash Re
seluser    84423       1  0 08:22 ?        00:00:00 /opt/google/chrome/chrome_crashpad_handler --no-periodic-tasks --monitor-self-annotation=ptype=crashpad-handler --database=/home/seluser/.config/google-chrome/Cra
seluser    84428   84414  0 08:22 ?        00:00:00 /opt/google/chrome/chrome --type=zygote --no-zygote-sandbox --no-sandbox --enable-logging --headless=new --log-level=0 --crashpad-handler-pid=84421 --enable-crash
seluser    84429   84414  0 08:22 ?        00:00:00 /opt/google/chrome/chrome --type=zygote --no-sandbox --enable-logging --headless=new --log-level=0 --crashpad-handler-pid=84421 --enable-crash-reporter=, --noerrd
seluser    84431   84429  0 08:22 ?        00:00:00 /opt/google/chrome/nacl_helper --no-sandbox
seluser    84452   84429  0 08:22 ?        00:00:00 /opt/google/chrome/chrome --type=utility --utility-sub-type=storage.mojom.StorageService --lang=en-US --service-sandbox-type=utility --no-sandbox --disable-dev-sh
seluser    86746       1  9 08:24 ?        00:00:25 /opt/google/chrome/chrome --no-sandbox --allow-pre-commit-input --blink-settings=imagesEnabled=false --disable-background-networking --disable-client-side-phishin
seluser    86759   86746  0 08:24 ?        00:00:00 cat
seluser    86760   86746  0 08:24 ?        00:00:00 cat
seluser    86805       1  0 08:24 ?        00:00:00 /opt/google/chrome/chrome_crashpad_handler --monitor-self --monitor-self-annotation=ptype=crashpad-handler --database=/home/seluser/.config/google-chrome/Crash Re
seluser    86821       1  0 08:24 ?        00:00:00 /opt/google/chrome/chrome_crashpad_handler --no-periodic-tasks --monitor-self-annotation=ptype=crashpad-handler --database=/home/seluser/.config/google-chrome/Cra
seluser    86828   86746  0 08:24 ?        00:00:00 /opt/google/chrome/chrome --type=zygote --no-zygote-sandbox --no-sandbox --enable-logging --headless=new --log-level=0 --crashpad-handler-pid=86805 --enable-crash
seluser    86829   86746  0 08:24 ?        00:00:00 /opt/google/chrome/chrome --type=zygote --no-sandbox --enable-logging --headless=new --log-level=0 --crashpad-handler-pid=86805 --enable-crash-reporter=, --noerrd
seluser    86900   86829  0 08:24 ?        00:00:00 /opt/google/chrome/nacl_helper --no-sandbox
seluser    86941   86829  0 08:24 ?        00:00:00 /opt/google/chrome/chrome --type=utility --utility-sub-type=storage.mojom.StorageService --lang=en-US --service-sandbox-type=utility --no-sandbox --disable-dev-sh
seluser    89568       1  8 08:26 ?        00:00:10 /opt/google/chrome/chrome --no-sandbox --allow-pre-commit-input --blink-settings=imagesEnabled=false --disable-background-networking --disable-client-side-phishin
seluser    89597   89568  0 08:26 ?        00:00:00 cat
seluser    89598   89568  0 08:26 ?        00:00:00 cat
seluser    89640       1  0 08:26 ?        00:00:00 /opt/google/chrome/chrome_crashpad_handler --monitor-self --monitor-self-annotation=ptype=crashpad-handler --database=/home/seluser/.config/google-chrome/Crash Re
seluser    89646       1  0 08:26 ?        00:00:00 /opt/google/chrome/chrome_crashpad_handler --no-periodic-tasks --monitor-self-annotation=ptype=crashpad-handler --database=/home/seluser/.config/google-chrome/Cra
seluser    89652   89568  0 08:26 ?        00:00:00 /opt/google/chrome/chrome --type=zygote --no-zygote-sandbox --no-sandbox --enable-logging --headless=new --log-level=0 --crashpad-handler-pid=89640 --enable-crash
seluser    89653   89568  0 08:26 ?        00:00:00 /opt/google/chrome/chrome --type=zygote --no-sandbox --enable-logging --headless=new --log-level=0 --crashpad-handler-pid=89640 --enable-crash-reporter=, --noerrd
seluser    89673   82605  4 08:27 ?        00:00:05 /opt/google/chrome/chrome --type=gpu-process --no-sandbox --disable-dev-shm-usage --enable-logging --headless=new --log-level=0 --ozone-platform=headless --use-an
seluser    89701   86828  4 08:27 ?        00:00:06 /opt/google/chrome/chrome --type=gpu-process --no-sandbox --disable-dev-shm-usage --enable-logging --headless=new --log-level=0 --ozone-platform=headless --use-an
seluser    89707   84428  1 08:27 ?        00:00:01 /opt/google/chrome/chrome --type=gpu-process --no-sandbox --disable-dev-shm-usage --enable-logging --headless=new --log-level=0 --ozone-platform=headless --use-an
seluser    89717   89653  0 08:27 ?        00:00:00 /opt/google/chrome/nacl_helper --no-sandbox
seluser    89784   89653  0 08:27 ?        00:00:00 /opt/google/chrome/chrome --type=utility --utility-sub-type=storage.mojom.StorageService --lang=en-US --service-sandbox-type=utility --no-sandbox --disable-dev-sh
seluser    90604   89652  1 08:27 ?        00:00:01 /opt/google/chrome/chrome --type=gpu-process --no-sandbox --disable-dev-shm-usage --enable-logging --headless=new --log-level=0 --ozone-platform=headless --use-an
seluser    93744   86746  4 08:29 ?        00:00:00 /opt/google/chrome/chrome --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=none --no-sandbox --disable-dev-shm-u
seluser    93753   82512  8 08:29 ?        00:00:00 /opt/google/chrome/chrome --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=none --no-sandbox --disable-dev-shm-u
seluser    93775   71731  7 08:29 ?        00:00:00 /opt/google/chrome/chrome --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=none --no-sandbox --disable-dev-shm-u
seluser    93778   84414  7 08:29 ?        00:00:00 /opt/google/chrome/chrome --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=none --no-sandbox --disable-dev-shm-u
seluser    93794   60581  0 08:29 ?        00:00:00 /proc/self/exe --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=none --no-sandbox --disable-dev-shm-usage --enab
seluser    93811   59834  0 08:29 ?        00:00:00 /proc/self/exe --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=none --no-sandbox --disable-dev-shm-usage --enab
seluser    93812   89568  0 08:29 ?        00:00:00 /proc/self/exe --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=none --no-sandbox --disable-dev-shm-usage --enab
seluser    93818   82670  0 08:29 pts/0    00:00:00 ps -eaf
seluser    93821   63575  0 08:29 ?        00:00:00 /proc/self/exe --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=none --no-sandbox --disable-dev-shm-usage --enab

This is my top output. What stands out is PID 67 with VIRT 3869948 but still says %MEM 1.9. Other than that I am not sure what to look for.

top - 13:20:25 up 46 min,  0 users,  load average: 17.21, 10.89, 7.85
Tasks: 106 total,   6 running,  99 sleeping,   0 stopped,   1 zombie
%Cpu(s): 37.7 us, 27.8 sy,  0.0 ni, 30.2 id,  2.2 wa,  0.0 hi,  2.0 si,  0.2 st
MiB Mem :  15986.3 total,   3338.0 free,   9368.3 used,   3279.9 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   6228.4 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     67 seluser   20   0 3869948 307980  28124 S   3.4   1.9   1:37.55 java
   5070 seluser   20   0   32.5g 187364 138760 R   0.9   1.1   0:49.27 chrome
  72355 seluser   20   0   32.5g 187176 141160 D   4.9   1.1   0:12.67 chrome
   6089 seluser   20   0   32.5g 185328 139680 S   3.7   1.1   0:49.49 chrome
   8244 seluser   20   0   32.5g 179976 138136 S   1.8   1.1   0:41.33 chrome
  62267 seluser   20   0   32.5g 173816 135140 S   2.1   1.1   0:18.73 chrome
  57526 seluser   20   0   32.5g 172488 134072 S   3.7   1.1   0:22.05 chrome
  67793 seluser   20   0   32.5g 170928 132052 D   4.6   1.0   0:14.18 chrome
  73736 seluser   20   0   32.5g 169316 127284 D  10.1   1.0   0:08.82 chrome
  76153 seluser   20   0   32.3g  77328  65108 S   3.7   0.5   0:00.17 chrome
  76201 seluser   20   0   32.3g  77044  64980 S   4.6   0.5   0:00.15 chrome
  76012 seluser   20   0   32.3g  76920  65160 S   0.9   0.5   0:00.16 chrome
     39 seluser   20   0  171112  65756  41280 S   1.5   0.4   0:09.50 Xvfb
  73751 seluser   20   0   32.3g  60980  49596 S   0.6   0.4   0:00.70 chrome
   5085 seluser   20   0   32.3g  60884  49496 S   0.0   0.4   0:00.52 chrome
   6104 seluser   20   0   32.3g  60800  49412 S   0.0   0.4   0:00.44 chrome
   8268 seluser   20   0   32.3g  60756  49368 S   0.0   0.4   0:00.41 chrome
  72436 seluser   20   0   32.3g  60676  49292 S   0.0   0.4   0:00.52 chrome
  62330 seluser   20   0   32.3g  60604  49220 S   0.0   0.4   0:00.82 chrome
  57541 seluser   20   0   32.3g  60516  49132 S   0.0   0.4   0:00.84 chrome
  67877 seluser   20   0   32.3g  60220  48840 S   0.0   0.4   0:00.94 chrome
   8267 seluser   20   0   32.3g  60096  48660 S   0.0   0.4   0:00.08 chrome
  62329 seluser   20   0   32.3g  59944  48512 S   0.0   0.4   0:00.09 chrome
   5084 seluser   20   0   32.3g  59928  48496 S   0.0   0.4   0:00.11 chrome
   6103 seluser   20   0   32.3g  59868  48436 S   0.0   0.4   0:00.09 chrome
  57540 seluser   20   0   32.3g  59652  48220 S   0.0   0.4   0:00.15 chrome
  73750 seluser   20   0   32.3g  59440  48008 S   0.0   0.4   0:00.12 chrome
  72435 seluser   20   0   32.3g  59432  47996 S   0.0   0.4   0:00.07 chrome
  67876 seluser   20   0   32.3g  59380  47944 S   0.0   0.4   0:00.11 chrome
  76209 seluser   20   0   32.3g  56156  44916 R   1.8   0.3   0:00.06 chrome
  72526 seluser   20   0   32.3g  55624  40904 S   2.4   0.3   0:01.62 chrome
  74011 seluser   20   0   32.3g  55308  40608 S   3.1   0.3   0:01.15 chrome
  67928 seluser   20   0   32.3g  54844  40516 S   0.0   0.3   0:03.12 chrome
  62396 seluser   20   0   32.3g  54632  40776 S   0.0   0.3   0:04.43 chrome
  74455 seluser   20   0   32.3g  54620  40484 S   2.4   0.3   0:01.01 chrome
  56201 seluser   20   0   32.3g  52888  39228 S   0.0   0.3   0:00.31 chrome
  12611 seluser   20   0   32.3g  50280  36608 S   0.0   0.3   0:00.49 chrome
  27678 seluser   20   0   32.3g  47920  34280 S   0.3   0.3   0:00.56 chrome
  76213 seluser   20   0   32.2g  46856  35796 R   2.8   0.3   0:00.09 chrome
   8292 seluser   20   0   32.3g  44136  32376 S   0.6   0.3   0:00.72 chrome
   6134 seluser   20   0   32.3g  42924  31116 S   0.0   0.3   0:00.87 chrome
  72512 seluser   20   0   32.3g  41872  30136 S   0.0   0.3   0:00.11 chrome
  57567 seluser   20   0   32.3g  41820  30100 S   0.0   0.3   0:00.59 chrome
  62432 seluser   20   0   32.3g  41380  29608 S   0.0   0.3   0:00.30 chrome
   5108 seluser   20   0   32.3g  41256  29520 S   0.0   0.3   0:00.74 chrome
  73774 seluser   20   0   32.3g  40380  28600 S   0.3   0.2   0:00.21 chrome
  68006 seluser   20   0   32.3g  39928  28172 S   0.0   0.2   0:00.32 chrome
     11 seluser   20   0   33820  23832   9220 S   0.0   0.1   0:03.50 supervisord
     43 seluser   20   0   28380  18136   8460 S   0.0   0.1   0:00.38 python3
     94 seluser   20   0   42476  15408   8244 S   0.6   0.1   0:01.83 x11vnc
  75420 seluser   20   0   32.1g  14788  12576 S   8.6   0.1   0:04.07 chromedriver
     50 seluser   20   0   25132  12672  10528 S   0.3   0.1   0:01.64 fluxbox
     38 seluser   20   0  280204  12352  10424 S   0.0   0.1   0:00.02 pulseaudio
  73752 seluser   20   0   32.0g   3948   3592 S   0.0   0.0   0:00.00 nacl_helper
  56028 seluser   20   0    9848   3924   3304 S   0.0   0.0   0:00.41 bash
   6111 seluser   20   0   32.0g   3908   3552 S   0.0   0.0   0:00.00 nacl_helper
   8269 seluser   20   0   32.0g   3900   3548 S   0.0   0.0   0:00.00 nacl_helper
  57542 seluser   20   0   32.0g   3840   3488 S   0.0   0.0   0:00.00 nacl_helper
  67914 seluser   20   0   32.0g   3840   3488 S   0.0   0.0   0:00.00 nacl_helper
   5086 seluser   20   0   32.0g   3828   3480 S   0.0   0.0   0:00.00 nacl_helper
  62343 seluser   20   0   32.0g   3812   3460 S   0.0   0.0   0:00.00 nacl_helper
  75060 seluser   20   0   11820   3760   3020 R   0.6   0.0   0:00.25 top
  72473 seluser   20   0   32.0g   3716   3360 S   0.0   0.0   0:00.01 nacl_helper
     25 seluser   20   0    9584   3584   3284 S   0.0   0.0   0:00.00 start-selenium-

Not sure how to interpret this further or continue debugging, suggestions appreciated.

diemol · 2023-08-25T15:02:16Z

Why are there so many Chrome processes open? Are you not closing the session? Does this happen when you do not use the proxy?

gerardo8al · 2023-08-25T15:23:54Z

Not sure why the processes are kept open. I just see this behaviour in the proxy version.

The closing of sessions is the same in the versions with and without proxy. The code that does the crawling is the same for both versions, the only difference is in the driver code for proxy.

diemol · 2023-08-25T21:49:25Z

I see. I am trying to figure out how to help because something in the proxy is causing Chrome to stay open.

There is little we can do. You should probably run the whole browser (non-headless) and see what is happening.

Aside from that, I am out of ideas.

Air-Zhuang · 2023-09-11T09:12:27Z

I also encountered this issue and eventually found out that the reason was a conflict between headless and proxy with auth

diemol · 2023-10-17T08:03:13Z

I am closing this because there is nothing for us to do.

Lukas-drz · 2023-11-22T22:50:46Z

Hello @gerardo8al did you find a solution since ?
I have a similar deployment and similar error.
Thanks

github-actions · 2023-12-23T00:16:32Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

gerardo8al added the needs-triaging label Aug 24, 2023

diemol added R-awaiting-answer and removed needs-triaging labels Aug 24, 2023

diemol closed this as not planned Won't fix, can't repro, duplicate, stale Oct 17, 2023

github-actions bot locked and limited conversation to collaborators Dec 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[🐛 Bug]: Selenium HUB + selenium-chrome-node memory not releasing when using proxy #1924

[🐛 Bug]: Selenium HUB + selenium-chrome-node memory not releasing when using proxy #1924

gerardo8al commented Aug 24, 2023

github-actions bot commented Aug 24, 2023

diemol commented Aug 24, 2023

gerardo8al commented Aug 24, 2023

diemol commented Aug 24, 2023

gerardo8al commented Aug 24, 2023

diemol commented Aug 24, 2023

gerardo8al commented Aug 25, 2023

diemol commented Aug 25, 2023

gerardo8al commented Aug 25, 2023

diemol commented Aug 25, 2023

Air-Zhuang commented Sep 11, 2023

diemol commented Oct 17, 2023

Lukas-drz commented Nov 22, 2023

github-actions bot commented Dec 23, 2023

[🐛 Bug]: Selenium HUB + selenium-chrome-node memory not releasing when using proxy #1924

[🐛 Bug]: Selenium HUB + selenium-chrome-node memory not releasing when using proxy #1924

Comments

gerardo8al commented Aug 24, 2023

What happened?

WebDriver without proxy

WebDriver with proxy

Simplified example of how my workers open the URLs

Grafana dashboards

Command used to start Selenium Grid with Docker

Relevant log output

Operating System

Docker Selenium version (tag)

github-actions bot commented Aug 24, 2023

diemol commented Aug 24, 2023

gerardo8al commented Aug 24, 2023

diemol commented Aug 24, 2023

gerardo8al commented Aug 24, 2023

diemol commented Aug 24, 2023

gerardo8al commented Aug 25, 2023

diemol commented Aug 25, 2023

gerardo8al commented Aug 25, 2023

diemol commented Aug 25, 2023

Air-Zhuang commented Sep 11, 2023

diemol commented Oct 17, 2023

Lukas-drz commented Nov 22, 2023

github-actions bot commented Dec 23, 2023