Recommendations for caching pipx installs in GitHub Actions

[KyleKing]

pipx appears to manages a number of directories (https://pypa.github.io/pipx/how-pipx-works/). Is there a recommendation on what to cache in GtiHub Actions to speed up CI? For example, this snippet adapted from setup-python where I would like to cache the poetry install managed by pipx:

steps:
- uses: actions/checkout@v3
- name: Install poetry
  run: pipx install poetry==1.6.1

- uses: actions/cache@v3
  with:
    path: ${{ ??? }}
    key: ${{ runner.os }}-${{ matrix.python-version }}-pipx-poetry-1.6.1

- uses: actions/setup-python@v4
  with:
    python-version: '3.9'
    cache: 'poetry'
- run: poetry install
- run: poetry run pytest

[uranusjr]

At work we cache the entire ~/.local/pipx directory and that seems to work, or at least I never heard infra folks complain about it. Are there blockers to doing that?

[KyleKing]

That generally seems to work and I ran into issues if attempting to also cache PIPX_BIN_DIR

This is a simplified variation of what I'm using:

---

Edit: I have a better version below that can handle any runner OS and saves the bin dir

[KyleKing]

Update: you can skip the pipx install step on a cache hit by running ensurepath after. This is an updated action that is a simplified version of what appears to work for me:

name: Setup Pipx Action

inputs:
  os:
    default: ubuntu-22.04
  pipx-packages:
    default: ""
  python-version:
    required: true

runs:
  using: composite
  steps:
    - name: Setup Local Variables
      id: variable-setup
      run: |
        echo "pipx-home=$PIPX_HOME" >> $GITHUB_OUTPUT
        echo "pipx-bin-dir=$PIPX_BIN_DIR" >> $GITHUB_OUTPUT

        export BASE64_PIPX_ID=$(echo -n '${{ inputs.pipx-packages }}' | base64)
        echo "cache-key-pipx=pipx-${{ inputs.os }}-${{ inputs.python-version }}-$BASE64_PIPX_ID" >> $GITHUB_OUTPUT
      shell: bash

    - name: Cache pipx
      id: cache-pipx
      # cache doesn't have permission to extract to "opt/"
      if: ${{ inputs.os != 'macos-latest' }}
      uses: actions/cache@v3
      with:
        path: |
          ${{ steps.variable-setup.outputs.pipx-home }}
          ${{ steps.variable-setup.outputs.pipx-bin-dir }}
        key: ${{ steps.variable-setup.outputs.cache-key-pipx }}

    - name: Install Poetry ${{ inputs.poetry-version }} and '${{ inputs.pipx-packages }}'
      if: ${{ steps.cache-pipx.outputs.cache-hit != 'true' }}
      run: |
        for tool in ${{ inputs.pipx-packages }}; do;
          pipx install $tool
        done
      shell: bash

    - name: Ensure Pipx Paths
      run: |
        pipx ensurepath
        # Optionally check which packages were restored from the cache:
        pipx list
      shell: bash

[adam-grant-hendry]

pipx is a pre-installed package manner on all GitHub Actions runners. Hence, $PIPX_HOME and $PIPX_BIN_DIR are already set when your workflow starts.

Echoing these in a job step, you can see what some of them are for different OSs:

Ubuntu-22.04:
  $PIPX_HOME=/opt/pipx
  $PIPX_BIN_DIR=/opt/pipx_bin
Windows Server 2022:
  $PIPX_HOME=C:\Program Files (x86)\pipx
  $PIPX_BIN_DIR=C:\Program Files (x86)\pipx_bin
macOS 12:
  $PIPX_HOME=/usr/local/opt/pipx
  $PIPX_BIN_DIR=/usr/local/opt/pipx_bin

The fact that GitHub Actions doesn't explain in its documentation that it overrides these is unfortunate. Hence, simply using the default ~/.local/pipx and ~/.local/bin (see pipx installation options) won't work.

Also unfortunately, since some of these are located in /opt, you won't have permissions save there, which causes action/cache to fail if you use that for path. Also see here.

An alternative might be to copy files from /opt to a folder in your ${{ github.workspace }} and cache that path (with sudo cp). You would then need to copy those files back into /opt at the beginning of the workflow. Artifacts might be a better choice here (there's also a 10GB limit for caches).

A second option would be to install pipx at the user level, in which case the paths should be ~/.local/pipx and ~/.local/bin by default. This comes at the cost of an re-install of a program given to you by GitHub Actions, but should only have a cost hit whenever you add something to pipx. Again, consider the 10GB limit of caches.

[webknjaz]

pipx is a pre-installed package manner on all GitHub Actions runners. Hence, $PIPX_HOME and $PIPX_BIN_DIR are already set when your workflow starts.

Isn't this only the case in the GH Runners, but not self-hosted? Cross-platform use probably needs fallbacks.

[tautologicc]

Self-hosted runners also come with pipx installed.

[armand-sauzay]

Commenting on the above since when caching poetry dependencies using the vanilla setup here, installing poetry itself usually becomes the slowest step.

Any recommendation on caching pipx installs that would work nicely with this setup?

[uranusjr]

Did you actually try any of the solutions above? I fail to see how this is different.

[armand-sauzay]

Do you happen to have a minimal example of how caching ~/.local/pipx work for you?

By default runners do not seem to have the required permissions to be able to save things which are located in /opt so I don't see any time improvement on the action run I think

[uranusjr]

If your CI does not support caching things in certain locations, set a custom PIPX_HOME and other variables to things are installed into a location that can be cached.

[armand-sauzay]

For the context, in my particular case, I was trying to speed up the pipx install step - step 1/2 in the snippet below. Because of the caching, actions/setup-python is almost instantaneous but the pipx install of poetry (which you need to have to be able to cache in setup-python) takes ~10 seconds. I tried a couple of solutions that I expand on below but none seem to really show any performance gain vs the 10 seconds that pipx install poetry takes.

    - name: Install poetry
      shell: bash
      run: pipx install poetry
    - name: Setup python
      uses: actions/setup-python@v4
      with:
        cache: "poetry"
        python-version: ${{ inputs.python-version }}

I tried two solutions but both were slower than the actual pipx install:
Solution 1/2 --> By copying stuff to a location that can be cached and then restoring (roughly 20-30 seconds slower)

    - name: Restore Poetry from Cache
      id: cache-poetry
      uses: actions/cache@v4
      with:
        path: ${{ github.workspace }}/pipx_cache
        key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}

    - name: Copy pipx installs to workspace if not cached
      if: steps.cache-poetry.outputs.cache-hit != 'true'
      shell: bash
      run: |
        mkdir -p ${{ github.workspace }}/pipx_cache/home
        mkdir -p ${{ github.workspace }}/pipx_cache/bin
        sudo cp -r $PIPX_HOME ${{ github.workspace }}/pipx_cache/home
        sudo cp -r $PIPX_BIN_DIR ${{ github.workspace }}/pipx_cache/bin
    - name: Copy pipx installs back if cached
      if: steps.cache-poetry.outputs.cache-hit == 'true'
      shell: bash
      run: |
        sudo cp -r ${{ github.workspace }}/pipx_cache/home/* $PIPX_HOME
        sudo cp -r ${{ github.workspace }}/pipx_cache/bin/* $PIPX_BIN_DIR
        pipx ensurepath
        pipx list
    - name: Ensure Pipx Paths
      shell: bash
      if: hashFiles('poetry.lock') != ''
      run: |
        pipx ensurepath
        pipx list
    - name: Install poetry
      if: hashFiles('poetry.lock') != '' && steps.cache-pipx.outputs.cache-hit != 'true'
      shell: bash
      run: pipx install poetry

Solution 2/2 I also tried installing with pip instead of pipx, but same it did not result in any performance gain (same as pipx install ~10seconds)

    - name: Setup python
     uses: actions/setup-python@v4
     with:
       python-version: ${{ inputs.python-version }}
       cache: "pip"
   - name: Install Poetry
     shell: bash
     run: pipx install poetry
     run: |
       pip install poetry

Any other recommendation to speed up the install?

[sdb9696]

It seems that solution 1/2 Copy pipx installs to workspace if not cached copies the files to the cache location before poetry has been installed. I think that step should go after the Install poetry step.

N.B. I was able to get this working as per #1051 (reply in thread) by setting the $PIPX_HOME, $PIPX_BIN_DIR and $PIPX_MAN_DIR environment variables to a custom location in a step prior to the pipx-cache step. Also adding the new cached $PIPX_BIN_DIR to the $GITHUB_PATH. One of the nice things about this way is that the cache only contains the poetry install as opposed to any other pipx applications installed in the runner image. Something to be aware of though if the ci relies on any other pipx pre-installed apps.

I think the poetry.lock file doesn't need to be part of the cache-key because this cache is the poetry install itself rather than the application poetry is installing. I used the poetry and pipx version in the cache-key.

Finally to get this working on windows I set a path separator environment variable base on the runner.os and built the paths with that. There may be a nicer way of making it os independent but that's all I could come up with.

Working custom action below:

---
name: Setup Environment
description: Install requested pipx dependencies, configure the system python, and install poetry and the package dependencies

inputs:
  poetry-install-options:
    default: ""
  poetry-version:
    default: 1.8.2
  python-version:
    required: true

runs:
  using: composite
  steps:
    - uses: "actions/setup-python@v5"
      id: setup-python
      with:
        python-version: "${{ inputs.python-version }}"

    - name: Setup pipx environment Variables
      id: pipx-env-setup
      # pipx default home and bin dir are not writable by the cache action
      # so override them here and add the bin dir to PATH for later steps.
      # This also ensures the pipx cache only contains poetry
      run: |
        SEP="${{ !startsWith(runner.os, 'windows') && '/' || '\\' }}"
        PIPX_CACHE="${{ github.workspace }}${SEP}pipx_cache"
        echo "pipx-cache-path=${PIPX_CACHE}" >> $GITHUB_OUTPUT
        echo "pipx-version=$(pipx --version)" >> $GITHUB_OUTPUT
        echo "PIPX_HOME=${PIPX_CACHE}${SEP}home" >> $GITHUB_ENV
        echo "PIPX_BIN_DIR=${PIPX_CACHE}${SEP}bin" >> $GITHUB_ENV
        echo "PIPX_MAN_DIR=${PIPX_CACHE}${SEP}man" >> $GITHUB_ENV
        echo "${PIPX_CACHE}${SEP}bin" >> $GITHUB_PATH
      shell: bash

    - name: Pipx cache
      id: pipx-cache
      uses: actions/cache@v4
      with:
        path: ${{ steps.pipx-env-setup.outputs.pipx-cache-path }}
        key: ${{ runner.os }}-python-${{ steps.setup-python.outputs.python-version }}-pipx-${{ steps.pipx-env-setup.outputs.pipx-version }}-poetry-${{ inputs.poetry-version }}

    - name: Install poetry
      if: steps.pipx-cache.outputs.cache-hit != 'true'
      id: install-poetry
      shell: bash
      run: |-
        pipx install poetry==\ --python "${{ steps.setup-python.outputs.python-path }}"

    - name: Read poetry cache location
      id: poetry-cache-location
      shell: bash
      run: |-
        echo "poetry-venv-location=$(poetry config virtualenvs.path)" >> $GITHUB_OUTPUT

    - uses: actions/cache@v4
      name: Poetry cache
      with:
        path: |
          ${{ steps.poetry-cache-location.outputs.poetry-venv-location }}
        key: ${{ runner.os }}-python-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('poetry.lock') }}-options-${{ inputs.poetry-install-options }}

    - name: "Poetry install"
      shell: bash
      run: |
        poetry install ${{ inputs.poetry-install-options }}