Refactor CI workflow

[danielhollas]

Overview

The goal of the PR is twofold:

• Improve CI runtimes (currently the overall workflow takes ~1h40min!)
• Reduce complexity

In the end, I mostly ended up reverting things back to where they were roughly around PR #372, while (hopefully) keeping the current modularity. Closes #414

Major changes

• The main change here is to stop uploading Docker images as artifacts: Instead we upload them to ghcr.io straightaway. This already brings a lot of simplification and time reduction -- the only "disadvantage" is that the PRs cannot be open from forks, but as discussed in Slow build test and push #414, this is more of a feature rather than a bug.

• The second approach I took was to reduce our reliance on the self-hosted ARM64 runner, which is a limited resource (can only run one job at a time). In this PR, the arm64 image is build on ubuntu GHA runner with the help of QEMU virtualization. This is a bit slower then building natively, but not by that much (12min versus 9min). The other big advantage of this is that it reduces a lot of complexity, since we don't need to create / manage manifests manually, and simply let the docker/bake-action take care of the multiplatform build and upload. We can re-evaluate this in a future PR.

• We first build the amd64 images and immediately test full-stack. This overall takes only 6 minutes, so any obvious issues should be caught early.
  I believe some of the improvements here should be ported to QEApp and aiida-core repos as well.

Final results

• The workflow now takes 20 minutes. The first amd64 build and full-stack tests take only around 6 minutes, so any issues are surfaced early.
• The overall diff here in terms of lines (+551/-502) is a bit misleading, since I've added a lot of explanation and example inputs / outputs in the auxiliary shell scripts. The total lines of the yaml files is actually +352/-470 +320/-470.

Future work

• Revamp release procedure so that we do not have to push to main
• Reconsider buidling of arm64 image on self-hosted runner, if GHA turns out to be unreliable. On some commits on this PR the build took over 1h instead of 12 minutes.
• Speed up the build by optimizing what we put in the image. I have some ideas but I'll keep them for a separate PR.

[unkcpz]

Thanks!
Just careful that I don't want to go back to the original situation that tests for all 4*2 images are just in one GH workflow (or two for two architectures). The bake can happened in one workflow but tests are better separated.

[danielhollas]

Yes, tests should be run separately and in parallel. But the current build strategy is extremely wasteful, since we're not reusing the previous layers (e.g. the base image is build 4 times, unless I am missing something).

[unkcpz]

(e.g. the base image is build 4 times, unless I am missing something).

That is correct! Initially I want to keep things as separated as possible and notice in the end every build cost < mins so didn't put effort to combine the built processes back to one.
It definitely worth to try. Let me know if you encounter any problem and sorry I delayed on starting to work on this.

[danielhollas]

@mbercx would you mind installing jq on the ARM runner? Thx

[danielhollas]

@unkcpz I've pushed a fix to dodo.py for building locally. Can you try it out by running doit build? Here's what I get after that

$ docker images
aiidalab/full-stack              2024.1017.post145.dev0_8f8a30e   1b069e00e91e   About a minute ago   2.54GB
aiidalab/lab                     2024.1017.post145.dev0_8f8a30e   af94ef50d9cc   About a minute ago   2.19GB
aiidalab/base-with-services      2024.1017.post145.dev0_8f8a30e   e6be8727a8d1   4 minutes ago        2.54GB
aiidalab/base                    2024.1017.post145.dev0_8f8a30e   dba20a49b4f7   34 minutes ago       2.16GB

The image tag (determined by VERSION variable) is automatically generated from git commit.

Otherwise I think I replied to all your comments, thank you for a detailed review! Once I get a green light from you I will merge and then make a release for the aiida v2.5 image.

[unkcpz]

Can you try it out by running doit build?

Thanks for the fix. This works for me locally.

But the command README seems not work doit build --arch=arm64 --version=my-version. And I specify with --targe base but it build all. Besides this, I am good will all other changes.

[unkcpz]

Before merge this can we make a release with aiida-core==2.5.1 first?

[danielhollas]

But the command README seems not work doit build --arch=arm64 --version=my-version. And I specify with --targe base but it build all.

This should now be fixed, can you try again? Note that the doit tests command is still broken, but this will require a bit more change to fix, I'll open an issue.

[danielhollas]

I've temporarily switched to buildjet runners (and you can see that the tests pass now @unkcpz).
My plan is to merge this and try making a release again so we don't have to wait till the ARM64 runners are fixed.

[unkcpz]

My plan is to merge this and try making a release again so we don't have to wait till the ARM64 runners are fixed.

Make sense to me. And I think we really need to move to buildjet.

[danielhollas]

@unkcpz can you approve so I can merge? (I could bypass it but don't want to :-) )

[unkcpz]

All good. This is much clear! Thanks. @danielhollas (and I know how difficult to figure out all the details 😄 ) I'll let you write up for a detail commit message, 153 commits seems a legend PR.