Fix excessive memory and disk usage when sharing layers

[mtrmac]

Absolutely untested, filing for early review.

createNewLayer can add new entries to s.filenames for uncompressed versions of layers; these entries don't match manifest.LayerInfos, causing the existing logic to assume they are a config and to store them as ImageBigData.

That's both a potentially unlimited disk space waste (recording uncompressed versions of layers unnecessarily), and a memory pressure concern (ImageBigData is created by reading the whole blob into memory).

Instead, precisely track the digest of the config, and never include more than one file in ImageBigData.

Blobs added via PutBlob/TryReusingBlob but not referenced from the manifest are now ignored instead of being recorded inside the image.

This is intended to fix containers/podman#24527 . @nalind is there anything that depends on the current behavior that I’m missing?

Cc: @giuseppe , affects performance measurements.

[rhatdan]

typo

[mtrmac]

Thanks! Fixed.

[nalind]

This is intended to fix containers/podman#24527 . @nalind is there anything that depends on the current behavior that I’m missing?

I can't recall any places where we're using PutBlob() with isConfig set to true on a local storage destination for a blob that isn't mentioned in the manifest. I'd go so far as to say we don't have many locations that directly call PutBlob() for local storage destinations.

[mtrmac]

Podman CI run: containers/podman#24629

[mtrmac]

In manual testing, I have confirmed both the bug hypothesis and that this fixes it.

[rhatdan]

Can you remove Draft flag?

[edsantiago]

Can this safely be merged? The skip F40, Go is too old commit suggests that this won't be able to be merged into podman or buildah, but it's not clear to me if it's "until F40 gets a new Golang update" or "never as long as F40 exists". go 1.22.6 -> .8 suggests the former but I recommend understanding this before merging.

[mtrmac]

Can you remove Draft flag?

I want to see the Podman CI ~passing first.

[mtrmac]

Can this safely be merged? The skip F40, Go is too old commit suggests that this won't be able to be merged into podman or buildah

That’s already the case before this PR, due to #2601 .

, but it's not clear to me if it's "until F40 gets a new Golang update" or "never as long as F40 exists". go 1.22.6 -> .8 suggests the former

https://bodhi.fedoraproject.org/updates/FEDORA-2024-9b94592629 means this should be resolved shortly.

In general, I‘ve been assuming that Go patch releases tend to include vulnerability fixes, and therefore tend to be included in Fedora quickly (… and given the static linking of Go binaries, our updated RPMs would be ~blocked on that Go update anyway). That’s not been the case with Go 1.22.{7,8}.

Separately, there’s an effort to change sigstore so that it doesn’t add the dependency on very newest Golang so quickly, around sigstore/sigstore#1879 .

[mtrmac]

containers/podman#24629 looks reasonable, rebased and ready for review.

[rhatdan]

LGTM
@nalind PTAL

[cgwalters]

So this was a regression from #2067 ? Which was motivated by containers/storage#595 ? Wow, a lot going on there.

Anyways, this code looks superficially sane to me...

[mtrmac]

No, that’s not related at all; #2067 only moves the dataBlobs code. This logic to select what is stored as ImageBigData has existed since the very first versions of the storage transport.

containers/podman#24527 (comment) points at the commit that introduced this bug.

[vrothberg]

LGTM
Merge at will

[cgwalters]

Ahh, OK.

Tangential: just glancing at that code one thing that seems like it'd help is if we had the canonical uncompressed size too in the spec, then some of this could just be mapping between the blob descriptor and and uncompressed descriptor instead of having separate maps for digest and size or so?

[mtrmac]

Tangential: just glancing at that code one thing that seems like it'd help is if we had the canonical uncompressed size too in the spec, then some of this could just be mapping between the blob descriptor and and uncompressed descriptor instead of having separate maps for digest and size or so?

• I don’t see the relationship to this PR
• The code needs to handle all currently-existing images; spec extensions are worth considering but would not simplify the code
• What if the manifest contains a wrong uncompressed size? A lot of the complexity in that file comes from the need to track untrusted / trusted values, and to ensure that a wrong or actively malicious layer in one image does not break using some other image where the layer shares an identifier. (Arguably, that’s 99% a self-imposed problem, when the key we use to deduplicate (uncompressed digest / TOC digest) and the key we use to identify inputs (compressed digests) differ. Maybe, if the key for all were a digest of the full layer metadata, and we gave up on some of the sharing, things could be much simpler to implement. But I’d first like to see a composefs-native c/storage design, supporting arbitrary combinations of independent layers instead of single parent links, before thinking what that would look like.)