This is a fork of the SOCI Snapshotter that includes a few changes:
- Added support for podman, based on the Stargz Snapshotter
- A mechanism for storing image-to-index mappings on the machine's local filesystem and
- A preference for retrieving SOCI artifacts from the machine's local filesystem before the container registry
This fork was forked at 0fa2e8c without the intention of merging upstream, per discussion with the SOCI Snapshotter maintainers.
You can test out the podman support by following the Getting Started instructions over in the SOCI Snapshotter repository to create and push SOCI indices to your container registry. Once you've done that, you can build the code with make, run it with ./out/soci-store /var/lib/soci/soci-store, update your /etc/containers/storage.conf to point to the soci-store:
[storage]
driver = "overlay"
runroot = "/run/containers/storage"
graphroot = "/var/lib/containers/storage"
[storage.options]
additionallayerstores=["/var/lib/soci-store/store:ref"]
And finally, run podman: podman pull --storage-opt=additionallayerstore=/var/lib/soci-store/store:ref imagename:tag
Quick list of things for me to look at / clean up at some point:
- Still requires sudo :-( (fine for our use case for now).
- What's the
diff1file and why is it being read so much? Looks like the stargz-snapshotter doesn't have any special treatment for it either.
Below is the readme from the SOCI Snapshotter repository at 0fa2e8c.
SOCI Snapshotter is a containerd snapshotter plugin. It enables standard OCI images to be lazily loaded without requiring a build-time conversion step. "SOCI" is short for "Seekable OCI", and is pronounced "so-CHEE".
The standard method for launching containers starts with a setup phase during which the container image data is completely downloaded from a remote registry and a filesystem is assembled. The application is not launched until this process is complete. Using a representative suite of images, Harter et al FAST '16 found that image download accounts for 76% of container startup time, but on average only 6.4% of the fetched data is actually needed for the container to start doing useful work.
One approach for addressing this is to eliminate the need to download the entire image before launching the container, and to instead lazily load data on demand, and also prefetch data in the background.
Existing lazy loading snapshotters rely on a build-time conversion step, to produce a new image artifact. This is problematic for container developers who won't or can't modify their CI/CD pipeline, or don't want to manage the cost and complexity of keeping copies of images in two formats. It also creates problems for image signing, since the conversion step invalidates any signatures that were created against the original OCI image.
SOCI addresses these issues by loading from the original, unmodified OCI image. Instead of converting the image, it builds a separate index artifact (the "SOCI index"), which lives in the remote registry, right next to the image itself. At container launch time, SOCI Snapshotter queries the registry for the presence of the SOCI index using the mechanism developed by the OCI Reference Types working group.
Another big consideration that we haven't implmented/integrated into SOCI is to image load order based on your specific workload. See design README for more details.
- Getting Started: walk through SOCI setups and features.
- Build: how to build SOCI from source, test SOCI (and contribute).
- Install: how to install SOCI as a systemd unit.
- Debug: accessing logs/metrics and debugging common errors.
- Glossary: glossary we use in the project.
There a few different lazy loading projects in the containerd snapshotter community. This project began as a fork of the popular Stargz-snapshotter project from commit 743e5e70a7fdec9cd4ab218e1d4782fbbd253803 with the intention of an upstream patch. During development the changes were fundamental enough that the decision was made to create soci-snapshotter as a standalone project. Soci-snapshotter builds on stargz's success and innovative ideas. Long term, this project intends and hopes to join containerd as a non-core project and intends to follow CNCF best practices.