An error occurred while retrieving available packages

[johnne]

Description

I'm getting an error message saying "An error occurred while retrieving available packages" as well as "I know you want to give up, but wait a bit longer.." which never ends. In my terminal I get

[D 2023-05-05 07:59:37.313 ServerApp.mamba_gator] ...
[D 2023-05-05 07:59:39.573 ServerApp.mamba_gator] Accepting token-authenticated connection from 127.0.0.1
[D 2023-05-05 07:59:39.592 ServerApp.mamba_gator] 202 GET /conda/tasks/3?1683266378066 (127.0.0.1) 19.02ms
[E 2023-05-05 07:59:39.626 ServerApp.mamba_gator] Error for task {'type': 'InvalidVersion', 'error': "Invalid version: '2022.02.02.gr3.9'", 'message': 'InvalidVersion("Invalid version: \'2022.02.02.gr3.9\'")', 'traceback': ['  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/handlers.py", line 96, in execute_task\n    result = await f(*args)\n             ^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/handlers.py", line 380, in update_available\n    answer = await env_manager.list_available()\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/envmanager.py", line 692, in list_available\n    packages = await current_loop.run_in_executor(None, format_packages, data)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/concurrent/futures/thread.py", line 58, in run\n    result = self.fn(*self.args, **self.kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/envmanager.py", line 662, in format_packages\n    version = parse(entry.get("version", ""))\n              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/packaging/version.py", line 52, in parse\n    return Version(version)\n           ^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/packaging/version.py", line 198, in __init__\n    raise InvalidVersion(f"Invalid version: \'{version}\'")\n']}.
[D 2023-05-05 07:59:40.603 ServerApp.mamba_gator] Accepting token-authenticated connection from 127.0.0.1
[D 2023-05-05 07:59:40.604 ServerApp.mamba_gator] {'type': 'InvalidVersion', 'error': "Invalid version: '2022.02.02.gr3.9'", 'message': 'InvalidVersion("Invalid version: \'2022.02.02.gr3.9\'")', 'traceback': ['  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/handlers.py", line 96, in execute_task\n    result = await f(*args)\n             ^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/handlers.py", line 380, in update_available\n    answer = await env_manager.list_available()\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/envmanager.py", line 692, in list_available\n    packages = await current_loop.run_in_executor(None, format_packages, data)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/concurrent/futures/thread.py", line 58, in run\n    result = self.fn(*self.args, **self.kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/envmanager.py", line 662, in format_packages\n    version = parse(entry.get("version", ""))\n              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/packaging/version.py", line 52, in parse\n    return Version(version)\n           ^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/packaging/version.py", line 198, in __init__\n    raise InvalidVersion(f"Invalid version: \'{version}\'")\n']}
[E 2023-05-05 07:59:40.605 ServerApp.mamba_gator] {
      "Host": "localhost:8890",
      "Accept": "*/*",
      "Referer": "http://localhost:8890/lab",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/112.0"
    }

I'm able to run both conda search --json and mamba search --json which I saw could be a problem in another issue (#127)

Reproduce

1. Activate Conda environment with jupyterlab and gator_lab installed
2. Run jupyter lab

Expected behavior

No error messages

Context

• Python package version: Version: 3.11.3 Build: h1456518_0_cpython
• Extension version: Version: 5.2.0 Build: pyhd8ed1ab_0

jupyter labextension list
Config option `kernel_spec_manager_class` not recognized by `ListLabExtensionsApp`.
[W 2023-05-05 08:02:54.466 LabApp] Config option `kernel_spec_manager_class` not recognized by `LabApp`.
JupyterLab v3.6.3
/Users/johnlarsson/mambaforge/envs/jupyter/share/jupyter/labextensions
        jupyterlab_pygments v0.2.2 enabled OK (python, jupyterlab_pygments)
        nbdime-jupyterlab v2.2.0 enabled OK
        jupyterlab_templates v0.4.0 enabled OK (python, jupyterlab_templates)
        @mamba-org/gator-lab v3.1.0 enabled OK (python, mamba_gator)
        @voila-dashboards/jupyterlab-preview v2.2.0 enabled OK (python, voila)
        @jupyterlab/github v3.0.1 enabled OK (python, jupyterlab_github)
        @jupyterlab/git v0.41.0 enabled OK (python, jupyterlab-git)
        @jupyter-widgets/jupyterlab-manager v5.0.7 enabled OK (python, jupyterlab_widgets)

Other labextensions (built into JupyterLab)
   app dir: /Users/johnlarsson/mambaforge/envs/jupyter/share/jupyter/lab
        @quarto/jupyterlab-quarto v0.1.42 enabled OK

• General information:

conda info 

     active environment : jupyter
    active env location : /Users/johnlarsson/mambaforge/envs/jupyter
            shell level : 2
       user config file : /Users/johnlarsson/.condarc
 populated config files : /Users/johnlarsson/mambaforge/.condarc
                          /Users/johnlarsson/.condarc
          conda version : 23.3.1
    conda-build version : not installed
         python version : 3.10.9.final.0
       virtual packages : __archspec=1=arm64
                          __osx=13.3.1=0
                          __unix=0=0
       base environment : /Users/johnlarsson/mambaforge  (writable)
      conda av data dir : /Users/johnlarsson/mambaforge/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/bioconda/osx-arm64
                          https://conda.anaconda.org/bioconda/noarch
          package cache : /Users/johnlarsson/mambaforge/pkgs
                          /Users/johnlarsson/.conda/pkgs
       envs directories : /Users/johnlarsson/mambaforge/envs
                          /Users/johnlarsson/.conda/envs
               platform : osx-arm64
             user-agent : conda/23.3.1 requests/2.28.2 CPython/3.10.9 Darwin/22.4.0 OSX/13.3.1
                UID:GID : 502:20
             netrc file : None
           offline mode : False

Command Line Output

[D 2023-05-05 07:59:37.313 ServerApp.mamba_gator] ...
[D 2023-05-05 07:59:39.573 ServerApp.mamba_gator] Accepting token-authenticated connection from 127.0.0.1
[D 2023-05-05 07:59:39.592 ServerApp.mamba_gator] 202 GET /conda/tasks/3?1683266378066 (127.0.0.1) 19.02ms
[E 2023-05-05 07:59:39.626 ServerApp.mamba_gator] Error for task {'type': 'InvalidVersion', 'error': "Invalid version: '2022.02.02.gr3.9'", 'message': 'InvalidVersion("Invalid version: \'2022.02.02.gr3.9\'")', 'traceback': ['  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/handlers.py", line 96, in execute_task\n    result = await f(*args)\n             ^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/handlers.py", line 380, in update_available\n    answer = await env_manager.list_available()\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/envmanager.py", line 692, in list_available\n    packages = await current_loop.run_in_executor(None, format_packages, data)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/concurrent/futures/thread.py", line 58, in run\n    result = self.fn(*self.args, **self.kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/envmanager.py", line 662, in format_packages\n    version = parse(entry.get("version", ""))\n              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/packaging/version.py", line 52, in parse\n    return Version(version)\n           ^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/packaging/version.py", line 198, in __init__\n    raise InvalidVersion(f"Invalid version: \'{version}\'")\n']}.
[D 2023-05-05 07:59:40.603 ServerApp.mamba_gator] Accepting token-authenticated connection from 127.0.0.1
[D 2023-05-05 07:59:40.604 ServerApp.mamba_gator] {'type': 'InvalidVersion', 'error': "Invalid version: '2022.02.02.gr3.9'", 'message': 'InvalidVersion("Invalid version: \'2022.02.02.gr3.9\'")', 'traceback': ['  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/handlers.py", line 96, in execute_task\n    result = await f(*args)\n             ^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/handlers.py", line 380, in update_available\n    answer = await env_manager.list_available()\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/envmanager.py", line 692, in list_available\n    packages = await current_loop.run_in_executor(None, format_packages, data)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/concurrent/futures/thread.py", line 58, in run\n    result = self.fn(*self.args, **self.kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/envmanager.py", line 662, in format_packages\n    version = parse(entry.get("version", ""))\n              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/packaging/version.py", line 52, in parse\n    return Version(version)\n           ^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/packaging/version.py", line 198, in __init__\n    raise InvalidVersion(f"Invalid version: \'{version}\'")\n']}
[E 2023-05-05 07:59:40.605 ServerApp.mamba_gator] {
      "Host": "localhost:8890",
      "Accept": "*/*",
      "Referer": "http://localhost:8890/lab",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/112.0"
    }

Browser Output

Paste the output from your browser Javascript console here.

[fcollonval]

Thanks for reporting the issue @johnne

The error originated from

gator/mamba_gator/envmanager.py

Line 662 in 0279570

version = parse(entry.get("version", ""))

The easy fix would be to ignore invalid version number. Then a follow up could be to handle invalid version.

Would you be willing to open a PR? The first step would be to wrap the version parsing with a try-except:

try:
    version = parse(entry.get("version", "")) 
except InvalidVersion:
    name = entry.get("name")
    version = entry.get("version")
    self.log.warning(f"Unable to parse version '{version}' of '{name}'")
    continue

With InvalidVersion to be imported from packaging.version.

[johnne]

Thank you! I edited the installation I have and that seems to get rid of the error message.

I will open a PR

[fcollonval]

Thanks for trying the fix. Would you mind posting here the log about which package failed to be parsed?

[johnne]

unparsed_versions.txt

This is what I get when running jupyter lab --debug and doing grep "Unable to parse version" from the debug log.

[fcollonval]

Thanks a lot @johnne

So as expected most packages failing are none-python ones as they do not require to conform to Python version syntax. And in recent version of packaging those are not allowed. The fix you made (thanks again) will avoid the bug but prevent to handle those packages. Would you be willing to do a follow-up PR to handle version for those packages?

Basically we should internalize the old LegacyVersion from packaging that was the fallback when failing to parse a version string.

See: https://github.com/pypa/packaging/blob/20.9/packaging/version.py

The code was

    try:
        return Version(version)
    except InvalidVersion:
        return LegacyVersion(version)

So the follow-up PR would be to change the try-except introduced in #202 by:

raw_version = entry.get("version", "")
try:
    version = parse(raw_version) 
except InvalidVersion:
    name = entry.get("name")
    msg = f"Falling back to 'LegacyVersion' for version '{raw_version}' of '{name}'"
    self.log.debug(msg)
    version = LegacyVersion(raw_version)

and copying the definition of LegacyVersion (and all the code used by it) from https://github.com/pypa/packaging/blob/20.9/packaging/version.py

Would you be willing to do the follow-up PR?

FYI I released 5.2.1 with your fix.