Add client conformance tests

[makkes]

This repository has a great suite of registry conformance tests. What it is missing, though, if I'm not mistaken, are conformance tests for clients. I took a look at some clients and none of them fully conformed to the protocol defined in the distribution spec. A client conformance test suite would help immensely adapting existing clients and building new ones that follow this spec.

[sudo-bmitch]

This is something I'd like to see myself, but I'm not sure how that would work considering the different types of clients (e.g. build tooling, runtimes, vulnerability scanners, mirroring, pull through cache, signing, artifacts like helm charts, tools to monitor for updates, etc). Do you have thoughts on how to build a standard OCI test suite that would handle the variety of clients written in different languages?

[makkes]

What this project could potentially provide is some kind of pluggable framework to run conformance tests that reflect what's defined in the spec, pretty much what is available in the registry conformance tests, grouped by workflow categories. A starter could be to provide this to CLI clients where you would provide the actual client command to perform the workflow actions like "pull manifest", "pull blob", "push blob" etc. with varying inputs. A conforming registry would be run to back the tests.

For libraries this might be a little more involved but probably a framework for Go libraries could be a good start.

[sudo-bmitch]

To take two examples from the many, a vulnerability scanner, and Helm, I'm not sure they could work with a test like that. They don't have the granular interface to interact with an OCI registry at that level. This could be useful for genetic tools, like crane, oras, regctl, and skopeo. But even for an image runtime, it may only pull images and may only accept a reference (which isn't yet defined in OCI) to a manifest. Image build tooling may pull images, but only base images for the build. One tool would need a Dockerfile, another would need a kubernetes manifest. And tools that push content cannot be controlled by the test harness. One tool would only push a helm chart, many container image builders are not reproducible, another tool might be designed to add metadata to an image (SBOM, or soci layer data), or some like estargz convert an existing image's layers.

For me, it's the variety of client tooling that makes this challenge difficult. But if you think that can be solved, I'll be very interested to see what you come up with.

[makkes]

I understand. I tried to find similar cases and one of those alI found is the OIDC RP conformance test suite. It works by providing a server component and a set of runbooks. I could imagine doing something similar for distribution spec. Probably would best be verified by a PoC so I'll look into coming up with something, putting my thoughts into code.

[waynr]

I was just thinking about this yesterday and had two ideas:

• add some kind of conformance middleware to an existing registry
• create a registry conformance proxy that can sit in front of any registry implementation

For either of these approaches the middleware/proxy would record the following information:

• User-Agent request header
• request method
• all request headers relevant to each request method (also taking note of their against that reabsence)
• query parameters
• session ids (would be necessary to infer what type of blob push is being performed)
• request/response latency

To make this a conformance test suite, registry clients in each of their intended workflows could be run against any registry set up like this.

EDIT: i didn't really finish my thought here...

Anyway, the idea would be that by collecting the above information we could create a report for each user agent that grades the client based on how nicely it plays with the registry API and outline any troubling/problematic behavior that only works because of ambiguity in the spec or laxity in key registry implementations.