Unable to find image 'ghcr.io/pypa/gh-action-pypi-publish:61da13deb5f5124fb1536194f82ed3d9bbc7e8f3'

[cpswan]

After bumping to v1.12.0 publishing to TestPyPI is failing:

Unable to find image 'ghcr.io/pypa/gh-action-pypi-publish:61da13deb5f5124fb1536194f82ed3d9bbc7e8f3' locally
docker: Error response from daemon: manifest unknown.

Full logs at the workflow run

Here's the relevant section of my workflow (which is pretty much boilerplate):

  publish-to-testpypi:
    name: Publish package to TestPyPI
    needs:
    - build
    runs-on: ubuntu-latest
    environment:
      name: testpypi
      url: https://test.pypi.org/p/atsdk
    permissions:
      id-token: write  # IMPORTANT: mandatory for trusted publishing
    steps:
    - name: Download all the dists
      uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8
      with:
        name: python-package-distributions
        path: dist/
    - name: Publish distribution to TestPyPI
      uses: pypa/gh-action-pypi-publish@61da13deb5f5124fb1536194f82ed3d9bbc7e8f3 # v1.12.0
      with:
        skip-existing: true
        attestations: true
        repository-url: https://test.pypi.org/legacy/

If I was to take a wild guess at why this is happening, it would be permissions for the new Docker images, which might be available to testers, but not yet public.

[cpswan]

Ah... taking another look I've got a new theory...

I'm pinning the action to it's git hash, in order to keep OpenSSF Scorecard checks happy. But it's using the hash as a tag, and there isn't a container image with that tag.

[webknjaz]

Thanks for the report! We didn't take hashes into account in #230 and images with those versions aren't published, and the runtime logic doesn't fall back to local builds.

cc @br3ndonland

[webknjaz]

UPD: I published an image with another tag and in #287 (reply in thread) we confirmed that it fixes the immediate problem.

I'm going to keep this issue open for now to keep it on my radar and think about a more long-term solution.

[webknjaz]

UPD: while releasing v1.12.1, I remembered to also produce an image for the commit hash before pushing the tag/branches — #296.

Long-term, I want to integrate the disconnected part of the process more to avoid human factor-caused mistakes.

[webknjaz]

As for possibly, doing a fallback, here's what should help implement the required check: https://checkson.io/blog/monitoring-docker-images/.

[DinisCruz]

I had the same issue today with a GH Action that has been working for quite a while

the first error I got was a bit misleading since it said "docker: invalid reference format: repository name (owasp-sbot/OSBot-GitHub-Actions) must be lowercase." (see https://github.com/owasp-sbot/OSBot-Utils/actions/runs/11723153140/job/32656045035#step:4:209 )

once I made that lowercase, I got another error saying "Unable to find image 'ghcr.io/owasp-sbot/osbot-github-actions:dev' locally" (see https://github.com/owasp-sbot/OSBot-Utils/actions/runs/11723801117/job/32656257025#step:4:209 )

my temporary workaround was to change in my GH Action

uses: pypa/gh-action-pypi-publish@release/v1

into

uses: pypa/[email protected]

[blink1073]

I'm seeing a similar situation when publishing to test pypi: "docker: Error response from daemon: Head "https://ghcr.io/v2/blink1073/drivers-github-tools/manifests/INTPYTHON-406": denied." My workaround was to pin the action as well.

https://github.com/blink1073/django-mongodb/actions/runs/11727412713/job/32668549951

[webknjaz]

These two last comments are actually a different issue and I saw two other reports filed separately (#291 / #299). I discovered that this was caused by calling the action from other composite actions.
I attempted fixing it in v1.12.1 and it didn't work because GitHub messes up the context variables we rely on to produce the image reference which consists of an action repo slug and the tag.
This was never encouraged, tested or intended use anyway. It just happened to work with the older structure.
I also discovered that people often misuse the action in an insecure/discouraged manner (like putting the build step into the same job where they elevate privileges with id-token: write).

So with that in mind, I'm thinking I'll document such use as known to be unsupported (as opposed to leading people to believe that it was ever encouraged in the first place).

[parth-kulkarni1]

I had the same issue today with a GH Action that has been working for quite a while

the first error I got was a bit misleading since it said "docker: invalid reference format: repository name (owasp-sbot/OSBot-GitHub-Actions) must be lowercase." (see https://github.com/owasp-sbot/OSBot-Utils/actions/runs/11723153140/job/32656045035#step:4:209 )

once I made that lowercase, I got another error saying "Unable to find image 'ghcr.io/owasp-sbot/osbot-github-actions:dev' locally" (see https://github.com/owasp-sbot/OSBot-Utils/actions/runs/11723801117/job/32656257025#step:4:209 )

my temporary workaround was to change in my GH Action

uses: pypa/gh-action-pypi-publish@release/v1

into

uses: pypa/[email protected]

If anyone is also having an error when switching to v1.11.0 of ERROR InvalidDistribution: Unknown distribution format: 'XXXXX.whl.publish.attestation', make sure you set attestations: false within your release step. See #283 for more info.

[webknjaz]

@parth-kulkarni1 no, split the job into two instead.

[ludwiglierhammer]

I had a similar issue today:

After bumping to v1.12.2 publishing to PyPi is failing:

Unable to find image 'ghcr.io/pypa/gh-action-pypi-publish:15c56dba361d8335944d31a2ecd17d700fc7bcbc' locally
docker: Error response from daemon: Get "https://ghcr.io/v2/": dial tcp 54.185.253.63:443: connect: connection refused.

The full logs.

Here is the section of my workflow:

jobs:
  build-n-publish-pypi:
    name: Build and publish Python 🐍 distributions 📦 to PyPI
    environment: production
    permissions:
      # IMPORTANT: this permission is mandatory for trusted publishing
      id-token: write
    runs-on: ubuntu-latest
    steps:
      - name: Harden Runner
        uses: step-security/harden-runner@91182cccc01eb5e619899d80e4e971d6181294a7 # v2.10.1
        with:
          disable-sudo: true
          egress-policy: block
          allowed-endpoints: >
            files.pythonhosted.org:443
            github.com:443
            pypi.org:443
            upload.pypi.org:443
      - name: Checkout Repository 
        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
      - name: Set up Python3
        uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
        with:
          python-version: "3.x"
      - name: Install packaging libraries
        run: |
          python -m pip install -r ci/requirements/requirements_ci.txt
      - name: Build a binary wheel and a source tarball
        run: |
          python -m flit build
      - name: Publish distribution 📦 to PyPI
        uses: pypa/gh-action-pypi-publish@15c56dba361d8335944d31a2ecd17d700fc7bcbc # v1.12.2

Publishing with a previous version v1.11.0 was working. This is the logs.

Many thanks for your help 🙏

[webknjaz]

@ludwiglierhammer it's failing because you configured your runner to block connections to GHCR and it prevents the network image download. You'll have to figure out how to allowlist it.

[br3ndonland]

Re: #290 (comment)

Ah... taking another look I've got a new theory...

I'm pinning the action to it's git hash, in order to keep OpenSSF Scorecard checks happy. But it's using the hash as a tag, and there isn't a container image with that tag.

Re: #290 (comment)

Thanks for the report! We didn't take hashes into account in #230 and images with those versions aren't published, and the runtime logic doesn't fall back to local builds.

cc @br3ndonland

@cpswan thanks for reporting this issue, and sorry for the inconvenience. As you mention, using the Git SHA is a valid (and secure) way to reference the action. We didn't catch this use case initially.

Thankfully it's a simple fix. I've opened #301 to resolve the issue.

Just a note in case other users land here - GitHub Actions only supports references by the full 40 character SHA. If users try to reference the action by a short SHA like 1234567, they will get an error like, "Unable to resolve action pypa/gh-action-pypi-publish@1234567, the provided ref 1234567 is the shortened version of a commit SHA, which is not supported. Please use the full commit SHA 1234567890123456789012345678901234567890 instead."

[ludwiglierhammer]

@ludwiglierhammer it's failing because you configured your runner to block connections to GHCR and it prevents the network image download. You'll have to figure out how to allowlist it.

@webknjaz Thanks for your hint. I added ruf-repo-cdn.sigstore.dev:443 to the allowed endpoints and the workflow passes. Has anybody an idea why? I just copied this new line from another workflow.

[br3ndonland]

@ludwiglierhammer couldn't you just add ghcr.io:443 to the allowed-endpoints used with the step-security/harden-runner action?

[webknjaz]

@ludwiglierhammer previously, the docker container was pre-built before all the steps in your job. Currently, it's not built but downloaded from GHCR, but the corresponding network access is happening when the job reaches the step with this action. So earlier, your hardening was being applied after the network access related to building the image (so it was before any network traffic was being blocked) and now, it's happening after the firewall is in place.