Endpoint discovery of referrers API

[m5i-work]

The artifact spec covers discovery of referrers API. But there're several corners that need clarification.

1 - Discovery API vs. 404 probe

Client needs to consider service implementations that don't support discovery API. In such case, client has to probe referrers endpoint directly. If the response code is 404, client assumes service doesn't support referrers API.

This brings out a question: can we ignore discovery API and always probe referrers endpoint directly?

One concern of 404 probe:

404 may be ambiguous (say the digest we are querying referrers for doesn't exist).
If client gets an answer from discovery API, it doesn't need to check API support again for subsequent requests. While for a 404 response from referrers API, client still need to probe for every subsequent referrers request.

2 - Standardization of oras extension

For discovery API to work, client needs to know pre-defined endpoints of a module (e.g. referrers API).
Therefore, the mapping between a module and its endpoint should be covered in specification.
The distribution-spec defines a way to add a new extension specification here.

Question: do we have any plan to submit an equivalent _oras.md file to distribution spec?

3 - Extensibility of oras extension

The discovery response in API discovery section is:

    "extensions": [
        {
            "name": "_oras",
            "url": "https://github.com/oras-project/artifacts-spec/blob/main/manifest-referrers-api.md",
            "description": "ORAS referrers listing API",
            "endpoints": [
              "_oras/artifacts/referrers"
            ]
        }
    ]

Note that url and description are specific to referrers API, but they are at the same level of name _oras.

Question: how is the extension structure supposed to be when we have more _oras modules?
Will there be multiple entries with the same name?

4 - Discover endpoint instead of checking existence

This point may go beyond the scope of artifact spec. But let me start off here.

The current usage of discovery API is to check whether a known endpoint is supported. But a more helpful usage is to find endpoint of a known module (in this case, referrers API). This way, backward compatibility would be much easier when we change endpoint of a module next time (we did this).

A possible discovery API response may look like:

    "extensions": [
        {
            "name": "_oras",
            "url": "...",
            "description": "...",
            "endpoints": [
              {
                  "cncf.oras.referrers": "_oras/artifacts/referrers",
                  "...": "...",
              }              
            ]
        }
    ]

[sajayantony]

I would recommend the client check the discovery endpoint and possibly store in the repository state.

The OCI spec does require this endpoint that can be used to determine behavior. 404 or empty referrers is a specific endpoint/api choice and would NOT consider that as the way to determine API existence.

/cc @michaelb990 for more comments.