Build Docker image and push to GHCR #230

br3ndonland · 2024-04-19T01:51:01Z

Description

Closes #58

Up to this point, the project has been set up as a Docker action referencing the Dockerfile.

Lines 86 to 88 in 3fbcf7c

    
           runs: 
        
             using: docker 
        
             image: Dockerfile

The downside to using the Dockerfile for the action is that the Docker image must be built every time the action is used (#58).

This PR will set up the project to build the Docker image and push it to GitHub Container Registry (GHCR). This change will speed up user workflows every time the action is used because the workflows will simply pull the Docker image from GHCR instead of building again.

Changes

Build container image with GitHub Actions

This PR will build Docker images with the Docker CLI (docker build). Builds will include inline cache metadata so layers can be reused by future builds.

This PR only proposes to build container images for x86_64 (linux/amd64) because GitHub Actions Linux runners currently only support x86_64 CPU architectures (actions/runner-images#5631), and this project only supports GitHub Actions Linux runners. The README explains:

Since this GitHub Action is docker-based, it can only be used from within GNU/Linux based jobs in GitHub Actions CI/CD workflows. This is by design and is unlikely to change due to a number of considerations we rely on.

Push container image to GHCR

The workflow will log in to GHCR using the built-in GitHub token and push the Docker image. Workflow runs triggered by pull requests will build the Docker image and run the smoke tests but will not push the Docker image.

Update action to pull container image from GHCR

Docker actions support pulling in pre-built Docker images by supplying a registry address to the image: key. The downside to this syntax is that there's no way to specify the correct Docker tag because the GitHub Actions image: and uses: keys don't accept any context. For example, if a user's workflow has uses: pypa/gh-action-pypi-publish@release/v1.8, then the action should pull in a Docker image built from the release/v1.8 ref, something like ghcr.io/pypa/gh-action-pypi-publish:release-v1.8 (Docker tags can't have /).

# this works but the image tag can't be customized
runs:
  using: docker
  image: docker://ghcr.io/pypa/gh-action-pypi-publish:release-v1.8

# this doesn't work because `image:` doesn't support context
runs:
  using: docker
  image: docker://ghcr.io/pypa/gh-action-pypi-publish:${{ github.action_ref }}

The workaround is to switch the top-level action.yml to a composite action that then calls the Docker action, substituting the correct image name and tag.

Originally I thought that I'd have a workflow where I trigger a release, that release adds a commit that hardcodes an update to action.yml with the "future" version tag, tags that commit and pushes it (post docker publish). It wouldn't be on the main branch, the tags would be on the orphaned leaves.

This looks like a better idea so far. Thanks again!

pypa#230 (comment)

br3ndonland · 2024-04-26T23:12:16Z

That sounds great. Take your time. Thanks for your consideration.

If you do decide to accept this change, I'm happy to help maintain the workflows in the future. Feel free to mention me @br3ndonland and I will help address any issues that come up.

pypa#230 (comment)

webknjaz · 2024-05-16T15:56:33Z

Thanks! I've hit "rebase" on the UI to get this on top of the recent changes/linting/lockfile bumps but haven't yet looked into it deeper.

.github/workflows/build-and-push-docker-image.yml

.github/actions/run-docker-container/action.yml

.github/workflows/build-and-push-docker-image.yml

.github/workflows/self-smoke-test-action.yml

.pre-commit-config.yaml

Dockerfile

action.yml

.github/actions/run-docker-container/action.yml

action.yml

webknjaz · 2024-05-16T20:01:27Z

I'm done with the initial review. More is needed, but I'd rather accept what I can through separate PRs to make this one smaller. And the suggested refactoring could be done in parallel. I think that generating the file is a good idea. It should be possible to write the file without bringing in the PyYAML dependency. But it's not that easy for reading it. Can we make use of yq somehow, and convert YAML to JSON this way, maybe?

br3ndonland · 2024-05-27T20:43:49Z

@webknjaz thank you for your detailed review. I've addressed most of your comments so far.

webknjaz · 2024-05-29T14:40:43Z

Commits like 213c885ac41d769527ac150e2e633bb1ccd886d4 aren't really necessary since Git would automatically absorb changes applied separately. FYI. In fact, this one may have a harmful effect — when the label change is merged into the default branch, and this one rebased on top, Git will keep the removal commit and attempt deleting the label 🤷‍♂️

Anyway, we'll address this later on.

create-docker-action.py

action.yml

requirements/github-actions.txt

action.yml

create-docker-action.py

webknjaz · 2024-05-29T20:03:55Z

@br3ndonland I'm sure my review is incomplete, but hopefully it gives you enough ideas to try out until the next time.

I think it'd be nice to get this in before #236 if at all possible.

webknjaz · 2024-05-29T20:28:13Z

@br3ndonland could you also rebase this branch locally? The GH button doesn't work, meaning there's going to be some conflicts to resolve.

actions/checkout#27 (comment) actions/checkout#1108

pypa#230 (comment) Co-authored-by: Sviatoslav Sydorenko (Святослав Сидоренко) <[email protected]>

Co-authored-by: Sviatoslav Sydorenko (Святослав Сидоренко) <[email protected]>

for more information, see https://pre-commit.ci

pypa#230 (comment)

br3ndonland · 2024-11-04T22:22:45Z

When workflow_dispatch is triggered, and I type in 1.12.0, I understand that it'll publish ghcr.io/pypa/gh-action-pypi-publish:v1.12.0, ghcr.io/pypa/gh-action-pypi-publish:v1.12 and ghcr.io/pypa/gh-action-pypi-publish:v1.

At your request (#230 (comment)), the workflow_dispatch menu has two fields:

Git ref (branch or tag)
Docker image tag

After navigating to the action, there will be a "Run workflow" button. To publish an image for a specific release, v1.13.0 for example:

Git ref - v1.13.0 (tag)
Docker image tag - v1.13.0

But how about ghcr.io/pypa/gh-action-pypi-publish:release-v1 and ghcr.io/pypa/gh-action-pypi-publish:unstable-v1? What's the process for those?

Currently the workflow will automatically build and push Docker images whenever new commits are pushed to unstable/* or release/* branches.

gh-action-pypi-publish/.github/workflows/build-and-push-docker-image.yml

Lines 5 to 8 in 36965cb

    
           on:   # yamllint disable-line rule:truthy 
        
             pull_request: 
        
             push: 
        
               branches: ["release/*", "unstable/*"]

Should the case of uses: pypa/gh-action-pypi-publish@unstable/v1 not hit the GHCR image and build the image like it does now? Does set_image() need to be modified to implement this?

Why wouldn't you want to push a Docker image for unstable/v1? I wouldn't want that as a user. It seems like it would just introduce more confusion and complication.

Should the case of uses: pypa/gh-action-pypi-publish@release/v1 hit the :v1 image, or should the image generation workflow produce one more tag? It looks like these two bits aren't cohesive: reading the diff once again, it looks like if I enter release/v1.11.0, it'll produce ghcr.io/pypa/gh-action-pypi-publish:release-v1, ghcr.io/pypa/gh-action-pypi-publish:release-v1.11 and ghcr.io/pypa/gh-action-pypi-publish:release-v1.11.0 which would then not match references like uses: pypa/[email protected].

The case of uses: pypa/gh-action-pypi-publish@release/v1 will use the code on the release/v1 branch and will pull the Docker image built from that same branch.

if I enter release/v1.11.0

Your repo does not have any branches or tags named release/v1.11.0, so why would you do that?

If you're publishing a Docker image for a specific release, enter the name of the tag as I said above.

Git ref - v1.13.0 (tag)
Docker image tag - v1.13.0

uses: pypa/[email protected] will then pull the Docker image built from the v1.13.0 Git tag.

webknjaz · 2024-11-05T19:55:15Z

When workflow_dispatch is triggered, and I type in 1.12.0, I understand that it'll publish ghcr.io/pypa/gh-action-pypi-publish:v1.12.0, ghcr.io/pypa/gh-action-pypi-publish:v1.12 and ghcr.io/pypa/gh-action-pypi-publish:v1.

At your request (#230 (comment)), the workflow_dispatch menu has two fields:
1. Git ref (branch or tag)

2. Docker image tag
After navigating to the action, there will be a "Run workflow" button. To publish an image for a specific release, v1.13.0 for example:
1. Git ref - v1.13.0 (tag)

2. Docker image tag - v1.13.0

Are you sure you're looking at the latest PR diff? Right now, when I look at https://github.com/pypa/gh-action-pypi-publish/pull/230/files#diff-63d86cca28be8586869affd2baf5d48489229f5d9b4684c11beb787eee6f614cR9-R14, I see a single input — tag. Is it possible that you might have a branch with something different?

Though, reading through the workflow, the docker image tag is computed from the tag input, which makes sense to me.

But how about ghcr.io/pypa/gh-action-pypi-publish:release-v1 and ghcr.io/pypa/gh-action-pypi-publish:unstable-v1? What's the process for those?

Currently the workflow will automatically build and push Docker images whenever new commits are pushed to unstable/* or release/* branches.

gh-action-pypi-publish/.github/workflows/build-and-push-docker-image.yml

Lines 5 to 8 in 36965cb

on: # yamllint disable-line rule:truthy

pull_request:

push:

branches: ["release/*", "unstable/*"]

Oh… That's the bit I've been overlooking.

Should the case of uses: pypa/gh-action-pypi-publish@unstable/v1 not hit the GHCR image and build the image like it does now? Does set_image() need to be modified to implement this?

Why wouldn't you want to push a Docker image for unstable/v1? I wouldn't want that as a user. It seems like it would just introduce more confusion and complication.

Oh, I was just writing down my confusion. I didn't look into the PR in a while, so I didn't remember all the context and got lost. It makes sense now.

Should the case of uses: pypa/gh-action-pypi-publish@release/v1 hit the :v1 image, or should the image generation workflow produce one more tag? It looks like these two bits aren't cohesive: reading the diff once again, it looks like if I enter release/v1.11.0, it'll produce ghcr.io/pypa/gh-action-pypi-publish:release-v1, ghcr.io/pypa/gh-action-pypi-publish:release-v1.11 and ghcr.io/pypa/gh-action-pypi-publish:release-v1.11.0 which would then not match references like uses: pypa/[email protected].

The case of uses: pypa/gh-action-pypi-publish@release/v1 will use the code on the release/v1 branch and will pull the Docker image built from that same branch.

if I enter release/v1.11.0

Your repo does not have any branches or tags named release/v1.11.0, so why would you do that?

If you're publishing a Docker image for a specific release, enter the name of the tag as I said above.
1. Git ref - v1.13.0 (tag)

2. Docker image tag - v1.13.0
uses: pypa/[email protected] will then pull the Docker image built from the v1.13.0 Git tag.

This was very helpful to refresh my understanding of the bits and pieces of the updates in the PR. Thank you!

I think I'm ready to merge it finally. I'll then try it out and let you know if something is still uncertain.

webknjaz · 2024-11-05T20:50:48Z

I just pushed release/v1 and realized that people will be hitting a non-existing image if they attempt a release during the next few minutes… Hopefully, not many will hit this. For any following pushes, they'll just be getting a slightly outdated image, which should be mostly fine except for when the older image is incompatible.

This is a follow-up for #230, which renamed the workflow filename.

webknjaz · 2024-11-06T09:15:36Z

@br3ndonland here are some regressions reported:

action pinned to a commit hash: Unable to find image 'ghcr.io/pypa/gh-action-pypi-publish:61da13deb5f5124fb1536194f82ed3d9bbc7e8f3' #290
self-hosted runner: After this update, my github actions are broken with below logs #289
nested composite action: No Such File or Directory #291
GitHub Enterprise: Unable to use v1.12.0 on Github Enterprise #292

Could you help me figure them out?

action.yml

webknjaz · 2024-11-06T19:44:55Z

v1.12.1 addresses immediate problems from the list above. Long-term, the release process will need to be improved and made more cohesive.

PR pypa#230 updated the action to pull Docker images from GHCR instead of building Docker images each time the workflow runs. As part of this PR, a new GitHub Actions workflow was added that builds Docker images and pushes them to GitHub Container Registry (GHCR). Actions can be referenced in various ways. The Docker build workflow covers most of the action references, but does not push Docker images tagged with the Git commit ID (Git SHA). This commit will add Docker tags for referencing the action with a Git SHA. GitHub Actions only supports references by the full 40 character SHA. If users try to reference the action by a short SHA like `1234567`, they will get an error like, "Unable to resolve action `pypa/gh-action-pypi-publish@1234567`, the provided ref `1234567` is the shortened version of a commit SHA, which is not supported. Please use the full commit SHA `1234567890123456789012345678901234567890` instead." pypa#230 pypa#290 https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/using-pre-written-building-blocks-in-your-workflow#using-shas

br3ndonland · 2024-11-12T04:08:59Z

@br3ndonland here are some regressions reported:

action pinned to a commit hash: Unable to find image 'ghcr.io/pypa/gh-action-pypi-publish:61da13deb5f5124fb1536194f82ed3d9bbc7e8f3' #290

self-hosted runner: After this update, my github actions are broken with below logs #289

nested composite action: No Such File or Directory #291

GitHub Enterprise: Unable to use v1.12.0 on Github Enterprise #292

Could you help me figure them out?

@webknjaz thanks for handling the user feedback so far. Here are some updates to the list you provided.

action pinned to a commit hash: Unable to find image 'ghcr.io/pypa/gh-action-pypi-publish:61da13deb5f5124fb1536194f82ed3d9bbc7e8f3' #290
- Comment: Unable to find image 'ghcr.io/pypa/gh-action-pypi-publish:61da13deb5f5124fb1536194f82ed3d9bbc7e8f3' #290 (comment)
- Fix: Tag Docker images with Git SHA #301
self-hosted runner: After this update, my github actions are broken with below logs #289
- Fix: ✨ Pre-install Python if there's none #294
nested composite action: No Such File or Directory #291
- I agree with your comment (v1.12.0 and newer releases don't support nested invocations from composite actions #299 (comment)) that 🧪 Use prefetched action to make trampoline #293 does not necessarily fix this problem.
- Possible duplicate: v1.12.0 and newer releases don't support nested invocations from composite actions #299
- Possible duplicate: v1.12 still failing with error similar to #291 #300
- Related: Build Docker image and push to GHCR #230 (comment)
- Fix: Support nested composite actions #302
GitHub Enterprise: Unable to use v1.12.0 on Github Enterprise #292
- There isn't enough information in Unable to use v1.12.0 on Github Enterprise #292 to fix the issue, but it was possibly occurring because GitHub Enterprise modifies $GITHUB_SERVER_URL. If so, the fix would have been a simple one-line change to the actions/checkout step in the composite action to set github-server-url: https://github.com.
```
- name: Check out action repo
  uses: actions/checkout@v4
  with:
    github-server-url: https://github.com
    path: action-repo
    ref: ${{ steps.set-repo-and-ref.outputs.ref }}
    repository: ${{ steps.set-repo-and-ref.outputs.repo }}
```
- Fix: 🧪 Use prefetched action to make trampoline #293 (the second checkout step may not have been necessary because we can use github.action_path)

webknjaz reviewed Apr 25, 2024

View reviewed changes

.github/workflows/build-and-push-docker-image.yml Show resolved Hide resolved

webknjaz mentioned this pull request Apr 26, 2024

[Roadmap] Implement PEP 740 pypi/warehouse#15871

Closed

25 tasks

br3ndonland added a commit to br3ndonland/gh-action-pypi-publish that referenced this pull request Apr 26, 2024

Separate docker login and docker push

bb0315f

pypa#230 (comment)

webknjaz pushed a commit to br3ndonland/gh-action-pypi-publish that referenced this pull request May 16, 2024

Separate docker login and docker push

3384fad

pypa#230 (comment)

webknjaz force-pushed the ghcr branch from bb0315f to 3384fad Compare May 16, 2024 15:55