CI: test Docker ubuntu-small and ubuntu-full on linux/arm64 as well

[rouault]

No description provided.

[pjonsson]

I broke libjxl for linux/arm64 with my changes, sorry about that.

I don't know anything about multi-platform builds, but you also changed something with the image digest for arm64. Would reverting commit 4a37432 also have fixed that issue?

[rouault]

but you also changed something with the image digest for arm64. Would reverting commit 4a37432 also have fixed that issue?

I'm not sure why I had to change this part to be honest. I presume this is due to having switched between Docker versions. My current one is "Docker version 24.0.7, build 24.0.7-0ubuntu2~20.04.1"
This version doesn't like things like "FROM arm64v8/ubuntu:24.04@sha256:c7a252e466fe410e855ae2b0ffd33aef074b26371df66df474386a98ab79f60e AS runner" :

docker.io/arm64v8/ubuntu:24.04
sha256:c7a252e466fe410e855ae2b0ffd33aef074b26371df66df474386a98ab79f60e
 --build-arg PROJ_VERSION=49906f73a3be1cdf11f5b679ae5bef627cdabeec --build-arg GDAL_VERSION=b9f419dd37387de8c3ad668c5cbea79e64174dc0 --build-arg GDAL_REPOSITORY=OSGeo/gdal --build-arg WITH_DEBUG_SYMBOLS=yes --build-arg BUILDKIT_INLINE_CACHE=1 --build-arg WITH_CCACHE=1 --build-arg GDAL_RELEASE_DATE=20241012 --build-arg RSYNC_REMOTE=rsync://127.0.0.1:23985/gdal-docker-cache/osgeo/gdal:ubuntu-small --network host --build-arg TARGET_ARCH=arm64 --build-arg TARGET_BASE_IMAGE=arm64v8/ubuntu:24.04@sha256:c7a252e466fe410e855ae2b0ffd33aef074b26371df66df474386a98ab79f60e
ERRO[0000] failure getting variant                       error="getCPUInfo for pattern: Cpu architecture: not found"
[+] Building 0.6s (6/6) FINISHED                                                                                                                                                                           
 => [internal] load .dockerignore                                                                                                                                                                     0.1s
 => => transferring context: 2B                                                                                                                                                                       0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                  0.0s
 => => transferring dockerfile: 12.31kB                                                                                                                                                               0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                                                                                                            0.4s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:865e5dd094beca432e8c0a1d5e1c465db5f998dca4e439981029b3b81fb39ed5                                                                       0.0s
 => [internal] load metadata for docker.io/library/ubuntu:24.04                                                                                                                                       0.0s
 => ERROR [internal] load metadata for docker.io/arm64v8/ubuntu:24.04@sha256:c7a252e466fe410e855ae2b0ffd33aef074b26371df66df474386a98ab79f60e                                                         0.0s
------
 > [internal] load metadata for docker.io/arm64v8/ubuntu:24.04@sha256:c7a252e466fe410e855ae2b0ffd33aef074b26371df66df474386a98ab79f60e:
------
Dockerfile:216
--------------------
 214 |     
 215 |     # Build final image
 216 | >>> FROM $TARGET_BASE_IMAGE AS runner
 217 |     
 218 |     USER root
--------------------
error: failed to solve: rpc error: code = Unknown desc = arm64v8/ubuntu:24.04@sha256:c7a252e466fe410e855ae2b0ffd33aef074b26371df66df474386a98ab79f60e: unable to fetch descriptor () which reports content 

[pjonsson]

Edit: clarified that I meant no longer support using Docker < 23.0 for building the GDAL images.

I get the same with Docker 27.3. The image digest thing works if I revert your change and 4a37432 though:

Using PROJ_DATUMGRID_LATEST_LAST_MODIFIED=last-modified: Wed, 28 Aug 2024 13:26:27 GMT
Using PROJ_VERSION=54a7662dbc52963da70303ac8c9124966f4cf6b4
Using GDAL_VERSION=7c54387bf26e04acfcf33b195aa9de8abe4707b0
Using GDAL_REPOSITORY=OSGeo/gdal
Using GDAL_RELEASE_DATE=20241013
a0b5f16d7ca7   osgeo/gdal:gdal_rsync_daemon                               "/bin/sh -c 'rsync -…"   4 days ago   Up 4 days                                        gdal_rsync_daemon
Fetching digest for ubuntu:24.04 linux/arm64...
sha256:f294a2d1660d4091e58b9b09fca12e4fdb5de4bac6cdcddedef1cca07d758497
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 23.94kB done
#1 DONE 0.0s

#2 resolve image config for docker-image://docker.io/docker/dockerfile:1
#2 DONE 0.9s

#3 docker-image://docker.io/docker/dockerfile:1@sha256:865e5dd094beca432e8c0a1d5e1c465db5f998dca4e439981029b3b81fb39ed5
#3 CACHED

#4 [internal] load metadata for docker.io/library/ubuntu:24.04@sha256:f294a2d1660d4091e58b9b09fca12e4fdb5de4bac6cdcddedef1cca07d758497
#4 ...

#5 [internal] load metadata for docker.io/library/ubuntu:24.04
#5 DONE 0.4s

#4 [internal] load metadata for docker.io/library/ubuntu:24.04@sha256:f294a2d1660d4091e58b9b09fca12e4fdb5de4bac6cdcddedef1cca07d758497
#4 DONE 1.2s

#6 [internal] load .dockerignore
#6 transferring context: 2B done
#6 DONE 0.0s

#7 [builder  1/27] FROM docker.io/library/ubuntu:24.04@sha256:ab64a8382e935382638764d8719362bb50ee418d944c1f3d26e0c99fae49a345
#7 DONE 0.0s

#8 [runner  1/15] FROM docker.io/library/ubuntu:24.04@sha256:f294a2d1660d4091e58b9b09fca12e4fdb5de4bac6cdcddedef1cca07d758497
#8 CACHED

#9 [internal] load build context
#9 transferring context: 246B done
#9 DONE 0.0s

#10 [builder  2/27] RUN echo arm64
#10 CACHED

#11 [builder  3/27] COPY ./bh-set-envvars.sh /buildscripts/bh-set-envvars.sh
#11 CACHED

#12 [builder  4/27] RUN rm -f /etc/apt/apt.conf.d/docker-clean     && echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache     && echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
#12 CACHED

#13 [runner  2/15] RUN date
#13 0.049 exec /bin/sh: exec format error
#13 ERROR: process "/bin/sh -c date" did not complete successfully: exit code: 1

#14 [builder  5/27] RUN --mount=type=cache,target=/var/cache/apt,sharing=locked     --mount=type=cache,target=/var/lib/apt,sharing=locked     . /buildscripts/bh-set-envvars.sh     && if test "arm64" != ""; then     rm -f /etc/apt/sources.list /etc/apt/sources.list.d/ubuntu.sources     && echo "deb [arch=amd64] http://us.archive.ubuntu.com/ubuntu/ noble main restricted universe" >> /etc/apt/sources.list     && echo "deb [arch=amd64] http://us.archive.ubuntu.com/ubuntu/ noble-updates main restricted universe" >> /etc/apt/sources.list     && echo "deb [arch=amd64] http://us.archive.ubuntu.com/ubuntu/ noble-backports main restricted universe" >> /etc/apt/sources.list     && echo "deb [arch=amd64] http://security.ubuntu.com/ubuntu noble-security main restricted universe" >> /etc/apt/sources.list     && echo "deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ noble main restricted universe" >> /etc/apt/sources.list     && echo "deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ noble-updates main restricted universe" >> /etc/apt/sources.list     && echo "deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ noble-security main restricted universe" >> /etc/apt/sources.list     && dpkg --add-architecture arm64     && apt-get update -y     && DEBIAN_FRONTEND=noninteractive apt-get install -y g++-13-${GCC_ARCH}-linux-gnu     && ln -s ${GCC_ARCH}-linux-gnu-gcc-13 /usr/bin/${GCC_ARCH}-linux-gnu-gcc     && ln -s ${GCC_ARCH}-linux-gnu-g++-13 /usr/bin/${GCC_ARCH}-linux-gnu-g++;     fi
#14 CANCELED
------
 > [runner  2/15] RUN date:
0.049 exec /bin/sh: exec format error
------
Dockerfile:436
--------------------
 434 |
 435 |     USER root
 436 | >>> RUN date
 437 |     ARG JAVA_VERSION=17
 438 |     ARG ARROW_SOVERSION=1600
--------------------
ERROR: failed to solve: process "/bin/sh -c date" did not complete successfully: exit code: 1

so I guess that part and a lot of the buildx parts of build.sh can be simplified quite a bit when you decide to not support Docker < 23.0 for building the GDAL images.

[rouault]

so I guess that part and a lot of the buildx parts of build.sh can be simplified quite a bit when you decide to not support Docker < 23.0 for building the GDAL images.

indeed . Docker 23 is still a bit new. I see in https://github.com/docker/cli/releases/tag/v23.0.0 it was released in Feb 2023