build with local base image

[jbguerraz]

Hello mates,

We're trying to build an image using a base image stored localy (containerd image store), this base image was built using the same buildkitd instance.
It fails trying to resolve the image domain (which it's expected since there is no registry only local image store).
If we remove the "WORKDIR" instruction, then it builds, if we add it back it fails.
Do you have any idea what's going wrong here ?
Thank you a lot for your support!

$ cat /etc/buildkit/buildkitd.toml 
[worker.oci]                       
  enabled = false
[worker.containerd]
  enabled = true
  address = "/run/containerd/containerd.sock"
  namespace = "k8s.io"

$ sudo ctr namespaces ls
NAME   LABELS 
k8s.io        

$ sudo ctr --namespace k8s.io images ls
REF                                                                                      TYPE                                                 DIGEST                                                                  SIZE    PLATFORMS   LABELS                          
images.local/platform/softwares/alpine:67619d2597bad     application/vnd.docker.distribution.manifest.v2+json sha256:a20246ea5c1d4dc031b2662be65a2b9d958c2785ea53dcc4cbaa094abfff36ba 2.7 MiB linux/amd64 io.cri-containerd.image=managed 
sha256:1e0c8885fe2ce1d3fed35eba1e48c220888ecfe1d70c00623fce2940c4fe1b88                  application/vnd.docker.distribution.manifest.v2+json sha256:a20246ea5c1d4dc031b2662be65a2b9d958c2785ea53dcc4cbaa094abfff36ba 2.7 MiB linux/amd64 io.cri-containerd.image=managed 

$ sudo crictl images
IMAGE                                                            TAG                         IMAGE ID            SIZE
images.local/platform/softwares/alpine  67619d2597bad       1e0c8885fe2ce       2.81MB

$ cat Dockerfile 
FROM images.local/platform/softwares/alpine:67619d2597bad
WORKDIR /tmp
CMD ["echo", "this works"]

$ sudo buildctl build --output type=image,name=images.local/platform/softwares/test:67619d2597,push=false --local context=. --local dockerfile=. --frontend gateway.v0 --opt source=docker/dockerfile
[+] Building 1.2s (7/8)                                                                                                                                                                                                                                                                                                        
 => resolve image config for docker.io/docker/dockerfile:latest                                                                                                                                                                                                                                                           0.9s
 => CACHED docker-image://docker.io/docker/dockerfile@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                                                                                                                                                             0.0s
 => => resolve docker.io/docker/dockerfile@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                                                                                                                                                                        0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                         0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                           0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                                      0.0s
 => => transferring dockerfile: 480B                                                                                                                                                                                                                                                                                      0.0s
 => [internal] load metadata for images.local/platform/softwares/alpine:67619d2597bad                                                                                                                                                                                                0.0s
 => [1/3] FROM images.local/platform/softwares/alpine:67619d2597bad@sha256:a20246ea5c1d4dc031b2662be65a2b9d958c2785ea53dcc4cbaa094abfff36ba                                                                                                                                               0.0s
 => => resolve images.local/platform/softwares/alpine:67619d2597bad@sha256:a20246ea5c1d4dc031b2662be65a2b9d958c2785ea53dcc4cbaa094abfff36ba                                                                                                                                               0.0s
 => ERROR [2/3] WORKDIR /kubelet                                                                                                                                                                                                                                                                                          0.0s
------
 > [2/3] WORKDIR /kubelet:
------
Dockerfile:2
--------------------
   1 |     FROM images.local/platform/softwares/alpine:67619d2597bad
   2 | >>> WORKDIR /tmp
   3 |     CMD ["echo", "this works"]
   4 |   
--------------------
error: failed to solve: failed to compute cache key: failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://images.local/v2/platform/softwares/alpine/blobs/sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49": dial tcp: lookup images.local on 8.8.8.8:53: no such host

[jedevc]

Your error message looks suspicious:

dial tcp: lookup images.local on 8.8.8.8:53: no such host

8.8.8.8 is Google's public DNS server. images.local looks like it's a local domain, and so won't be on a public DNS server - it looks like something somewhere is misconfigured on your local DNS. You'll need to setup your DNS properly on the buildkitd server-side machine as well as on the buildctl client-side machine.

I'm not quite sure why removing WORKDIR makes everything good again. It seems like it might be BuildKit being smart and not pulling images.local/platform/softwares/alpine:67619d2597bad, since there's a cached copy locally somewhere.

[jbguerraz]

There is no registry. The image was not pushed. It exists only on containerd (local) image store.
It doesn't try to resolve when there is nothing between FROM and CMD.

It also fails the same way with a RUN instead of WORKDIR (and I bet with any instruction that creates a layer).
I have the intuition that is related to lazy pulling:

The previous release of BuildKit v0.9 introduced another new feature: lazy image pulling. What this feature means is that whenever BuildKit needs to access a remote image/cache, it will delay the pulling of its layers until there is a task that actually needs to read files from them. For example, when a layer is just used in another image this pulling is not needed and BuildKit can just create a new image referencing the previous layer by its immutable digest.
From https://www.docker.com/blog/image-rebase-and-improved-remote-cache-support-in-new-buildkit/

I then wonder why it would pull an image that is already available in containerd image store ?

[jbguerraz]

probably related info: #2343 (comment)

[jbguerraz]

$ sudo ctr --namespace k8s.io content ls | grep 213ec9aee27d8be0 -B 100 | sed -e 1b -e '$!d'
DIGEST									SIZE	AGE	LABELS
sha256:a20246ea5c1d4dc031b2662be65a2b9d958c2785ea53dcc4cbaa094abfff36ba	527B	6 hours	containerd.io/gc.ref.content.1=sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49,containerd.io/gc.ref.content.0=sha256:1e0c8885fe2ce1d3fed35eba1e48c220888ecfe1d70c00623fce2940c4fe1b88,containerd.io/distribution.source.images.local=platform/softwares/alpine

The blob looks stored as expected in containerd image store.

There is still a confusing thing to me yet, => resolve images.local/platform/softwares/alpine:67619d2597bad@sha256:a20246ea5c1d4dc031b2662be65a2b9d958c2785ea53dcc4cbaa094abfff36ba use the digest sha but Get "https://images.local/v2/platform/softwares/alpine/blobs/sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49" use the the sha used as label containerd.io/gc.ref.content.1; it makes me wonder what is that blob proper ID

[jbguerraz]

Here is a MRE:

$ containerd -v
containerd github.com/containerd/containerd 1.6.6 d0d56c1a4ace8bae8c7c98d28ba98f0537ebe704

$ buildkitd -v
buildkitd github.com/moby/buildkit v0.10.4 a2ba6869363812a210fcc3ded6926757ab780b5f

$ cat /etc/buildkit/buildkitd.toml 
[worker.oci]                       
  enabled = false
[worker.containerd]
  enabled = true
  address = "/run/containerd/containerd.sock"

$ sudo ctr namespace ls                   
NAME     LABELS 
buildkit        

$ sudo ctr --namespace=buildkit content ls  
DIGEST	SIZE	AGE	LABELS

$ cat Dockerfile.parent 
FROM alpine:3.16

$ cat Dockerfile.children 
FROM images.local/mre/parent:latest
WORKDIR /tmp

$ sudo buildctl build --output type=image,name=images.local/mre/parent:latest,store=true,push=false --local context=. --local dockerfile=. --frontend gateway.v0 --opt source=docker/dockerfile --opt filename=Dockerfile.parent  
[+] Building 2.3s (7/7) FINISHED                                                                                                                                                                                     
 => resolve image config for docker.io/docker/dockerfile:latest                                                                                                                                                 1.2s
 => CACHED docker-image://docker.io/docker/dockerfile@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                                                   0.0s
 => => resolve docker.io/docker/dockerfile@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                                                              0.0s
 => [internal] load .dockerignore                                                                                                                                                                               0.0s
 => => transferring context: 2B                                                                                                                                                                                 0.0s
 => [internal] load build definition from Dockerfile.parent                                                                                                                                                     0.0s
 => => transferring dockerfile: 110B                                                                                                                                                                            0.0s
 => [internal] load metadata for docker.io/library/alpine:3.16                                                                                                                                                  0.8s
 => [1/1] FROM docker.io/library/alpine:3.16@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad                                                                                            0.0s
 => => resolve docker.io/library/alpine:3.16@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad                                                                                            0.0s
 => exporting to image                                                                                                                                                                                          0.0s
 => => exporting layers                                                                                                                                                                                         0.0s
 => => exporting manifest sha256:a20246ea5c1d4dc031b2662be65a2b9d958c2785ea53dcc4cbaa094abfff36ba                                                                                                               0.0s
 => => exporting config sha256:1e0c8885fe2ce1d3fed35eba1e48c220888ecfe1d70c00623fce2940c4fe1b88                                                                                                                 0.0s
 => => naming to images.local/mre/parent:latest                                                                                                                                                                 0.0s

$ sudo buildctl build --output type=image,name=images.local/mre/children:latest,store=true,push=false --local context=. --local dockerfile=. --frontend gateway.v0 --opt source=docker/dockerfile --opt filename=Dockerfile.children 
[+] Building 0.7s (7/7) FINISHED                                                                                                                                                                                     
 => resolve image config for docker.io/docker/dockerfile:latest                                                                                                                                                 0.4s
 => CACHED docker-image://docker.io/docker/dockerfile@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                                                   0.0s
 => => resolve docker.io/docker/dockerfile@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                                                              0.0s
 => [internal] load build definition from Dockerfile.children                                                                                                                                                   0.0s
 => => transferring dockerfile: 144B                                                                                                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                                                                               0.0s
 => => transferring context: 2B                                                                                                                                                                                 0.0s
 => [internal] load metadata for images.local/mre/parent:latest                                                                                                                                                 0.0s
 => [1/2] FROM images.local/mre/parent:latest@sha256:a20246ea5c1d4dc031b2662be65a2b9d958c2785ea53dcc4cbaa094abfff36ba                                                                                           0.0s
 => => resolve images.local/mre/parent:latest@sha256:a20246ea5c1d4dc031b2662be65a2b9d958c2785ea53dcc4cbaa094abfff36ba                                                                                           0.0s
 => ERROR [2/2] WORKDIR /tmp                                                                                                                                                                                    0.0s
------
 > [2/2] WORKDIR /tmp:
------
Dockerfile.children:2
--------------------
   1 |     FROM images.local/mre/parent:latest
   2 | >>> WORKDIR /tmp
   3 |     
--------------------
error: failed to solve: failed to compute cache key: failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://images.local/v2/mre/parent/blobs/sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49": dial tcp: lookup images.local on 8.8.8.8:53: no such host

$ sudo ctr --namespace=buildkit content ls 
DIGEST									SIZE	AGE		LABELS
sha256:1304f174557314a7ed9eddb4eab12fed12cb0cd9809e4c28f29af86979a3c870	528B	22 seconds	containerd.io/distribution.source.docker.io=library/alpine
sha256:1d96e60e5270815238e999aed0ae61d22ac6f5e5f976054b24796d0e0158b39c	528B	22 seconds	-
sha256:1e0c8885fe2ce1d3fed35eba1e48c220888ecfe1d70c00623fce2940c4fe1b88	870B	22 seconds	containerd.io/distribution.source.images.local=mre/parent
sha256:1e8a16826fd1c80a63fa6817a9c7284c94e40cded14a9b0d0d3722356efa47bd	2.367kB	23 seconds	containerd.io/distribution.source.docker.io=docker/dockerfile
sha256:9a88b8895602862ed151c64204a1fa465366d6fa4bb7797f09e7d32d9e6763c0	1.47kB	22 seconds	-
sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc	2.004kB	23 seconds	containerd.io/distribution.source.docker.io=docker/dockerfile
sha256:9c6f0724472873bb50a2ae67a9e7adcb57673a183cea8b06eb778dca859181b5	1.47kB	22 seconds	containerd.io/distribution.source.docker.io=library/alpine
sha256:a20246ea5c1d4dc031b2662be65a2b9d958c2785ea53dcc4cbaa094abfff36ba	527B	22 seconds	containerd.io/gc.ref.content.0=sha256:1e0c8885fe2ce1d3fed35eba1e48c220888ecfe1d70c00623fce2940c4fe1b88,containerd.io/distribution.source.images.local=mre/parent,containerd.io/gc.ref.content.1=sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49
sha256:ad87fb03593d1b71f9a1cfc1406c4aafcb253b1dabebf569768d6e6166836f34	528B	23 seconds	containerd.io/distribution.source.docker.io=docker/dockerfile
sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad	1.638kB	22 seconds	containerd.io/distribution.source.docker.io=library/alpine

[jbguerraz]

Looks like I've got it..
Wrapping an image (nothing more than FROM instruction) is the issue, if we add any layer creating instruction in Dockerfile.parent like the WORKDIR one, then it starts to work!

[jbguerraz]

@jedevc @tonistiigi since the case is clear now (single FROM instruction images) maybe we better create an issue from this discussion ? WDYT?

[thaJeztah]

👆 ticket was created in #3164