Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IPIP-337: Delegated Content Routing HTTP API #337

Merged
merged 29 commits into from
Feb 11, 2023

Conversation

guseggert
Copy link
Contributor

No description provided.

@guseggert guseggert force-pushed the feat/delegated-routing-http-api branch 6 times, most recently from b301486 to 586a466 Compare October 18, 2022 15:12
@guseggert guseggert force-pushed the feat/delegated-routing-http-api branch from 586a466 to 1d9ec9c Compare October 18, 2022 15:29
IPIP/0000-delegated-routing-http-api.md Outdated Show resolved Hide resolved
IPIP/0000-delegated-routing-http-api.md Outdated Show resolved Hide resolved
IPIP/0000-delegated-routing-http-api.md Outdated Show resolved Hide resolved
IPIP/0000-delegated-routing-http-api.md Outdated Show resolved Hide resolved
IPIP/0000-delegated-routing-http-api.md Outdated Show resolved Hide resolved
routing/DELEGATED_ROUTING_HTTP.md Outdated Show resolved Hide resolved
routing/DELEGATED_ROUTING_HTTP.md Outdated Show resolved Hide resolved
routing/DELEGATED_ROUTING_HTTP.md Outdated Show resolved Hide resolved
routing/DELEGATED_ROUTING_HTTP.md Outdated Show resolved Hide resolved
routing/DELEGATED_ROUTING_HTTP.md Outdated Show resolved Hide resolved
@ajnavarro
Copy link
Member

👏 👏 🎉

The main thing I'm missing is type specification apart from the JSON example:

  • How CIDs are encoded
  • How PeerIDs are encoded
  • How IPNS IDs are encoded
  • And so on.

Also, I'm missing some limitations on results and payloads.

@guseggert guseggert changed the title feat: Delegated Routing HTTP API IPIP: Delegated Routing HTTP API Oct 19, 2022
@guseggert guseggert force-pushed the feat/delegated-routing-http-api branch from 2a9e695 to a94bfb2 Compare October 19, 2022 18:02
* Add detailed signature procedure for PUT /v1/providers
* Add error codes
* Remove pagination and limit field
* Many other small changes based on feedback
@guseggert guseggert force-pushed the feat/delegated-routing-http-api branch from a94bfb2 to 65d178b Compare October 19, 2022 18:17
@guseggert guseggert force-pushed the feat/delegated-routing-http-api branch from 63e9402 to 4c024dd Compare October 19, 2022 18:25
routing/DELEGATED_ROUTING_HTTP.md Outdated Show resolved Hide resolved
routing/DELEGATED_ROUTING_HTTP.md Outdated Show resolved Hide resolved
Copy link
Member

@lidel lidel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should land GET spec when Kubo 0.18 ships with the initial implementation.

@guseggert will you have time to move PUTs to a separate IPIP created on top of this one?
It will enable us to land GETs and continue work on PUTs in less noisy PR.

routing/DELEGATED_CONTENT_ROUTING_HTTP.md Outdated Show resolved Hide resolved
routing/DELEGATED_CONTENT_ROUTING_HTTP.md Outdated Show resolved Hide resolved
lidel and others added 2 commits January 16, 2023 14:20
Co-authored-by: Antonio Navarro Perez <[email protected]>
Co-authored-by: Masih H. Derkani <[email protected]>
Co-authored-by: Masih H. Derkani <[email protected]>
Reduce the scope of IPIP-337 by temporarily excluding write operation.
The write operations are to be added in a separate PR once the IPIP-337
is merged.

See:
 - #337
masih and others added 3 commits January 24, 2023 14:13
Fix lint issues as reported by GitHub `super-liner`, including:
- empty lines surrounding headers.
- maximum line length.
- unique headers
- natural language.
- tailing space.
Reduce the scope of IPIP-337 by excluding write operations
Copy link
Member

@lidel lidel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This IPIP mostly LGTM, reflects GET-only API used in Kubo 0.18 and cid.contact.
Some things are still not implemented by anybody, asked for clarifications below.

Let's wait with merge until IPFS Implementers Sync this Thursday.
If there are no concerns raised during the sync, and we decide what to do with things that are not implemented yet, we will merge after the Thursday call.

routing/DELEGATED_CONTENT_ROUTING_HTTP.md Outdated Show resolved Hide resolved
routing/DELEGATED_CONTENT_ROUTING_HTTP.md Outdated Show resolved Hide resolved
routing/DELEGATED_CONTENT_ROUTING_HTTP.md Outdated Show resolved Hide resolved
routing/DELEGATED_CONTENT_ROUTING_HTTP.md Outdated Show resolved Hide resolved
routing/DELEGATED_CONTENT_ROUTING_HTTP.md Outdated Show resolved Hide resolved

- Default limit: 100 providers
- Optional query parameters
- `transfer` only return providers who support the passed transfer protocols, expressed as a comma-separated list of transfer protocol names such as `transfer=bitswap,filecoin-graphsync-v1`
Copy link
Member

@lidel lidel Jan 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just want to flag that transfer parameter is a hardcoded list of pre-vetted strings, which is making future changes extremely difficult:

  • it will be impossible to query for providers that expose novel third-party protocols
  • every new protocol needs to be manually added to the spec here, then to libraries, and then rolled out to indexers and clients

Feels like gatekeeping system which makes it impossible to experiment with novel transfer protocols.

Last year I suggested to not invent own strings, avoid transport protocol codes from table.csv.

Since transfer is not implemented yet, I suggest removing it, or adding escape hatch for third-party protocols not specified in this document.

If we want to keep it, need to ensure we are not gatekeeping:

Option A

Ensure clients can talk the same version of bitswap by reusing strings from libp2p identify's protocols list:

$ ipfs id | jq .Protocols
[
  "/ipfs/bitswap",
  "/ipfs/bitswap/1.0.0",
  "/ipfs/bitswap/1.1.0",
  "/ipfs/bitswap/1.2.0",
  "/libp2p/fetch/0.0.1",
  ...
]

We could update spec and say that every value that starts with / should be interpreted as libp2p protocol prefix:

transfer=/ipfs/bitswap would return all peers that speak any version of bitswap.

Option B

Alternative, is to just accept number, and allow people to use codes from reserved private range

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO using the numbers from the global table feels pretty bad for experimentation as well. The existing IPNI implementation doesn't support them and they require some gatekeeping anyway. For the time being I've put up ipni/specs#6.

It seems like we could reasonably integrate this with something like the / proposal.

transfer=/ipfs/bitswap would return all peers that speak any version of bitswap.

Unfortunately, this type of query isn't supported in IPNI today and if it did it'd probably be with some specific semantics (e.g. adding a field to the "BitswapMetadata" section that's a list of protocolIDs) or a similar thing for GraphSync, unless they're going to reserve a new number every time they modify the protocol).

If we started leveraging named-record in addition to (or instead of) numbers in the global table we could just query against those. Basically, this would mean that the query could be fulfilled using custom logic per-number, or just using the named-record wrapper.

Since transfer is not implemented yet, I suggest removing it

Since the general policy is not to merge specs without implementations we should remove and then re-add later with the implementation, right?

Copy link
Member

@masih masih Jan 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

experimentation
require some gatekeeping

IPNI imposes no restriction on the protocol ID in metadata; this can be any number and treated as arbitrary bytes by the indexers.

Unfortunately, this type of query isn't supported in IPNI today

Happy to capture an issue on this if this is needed?

if it did it'd probably be with some specific semantics

It'd probably be varint prefix matching.

If we started leveraging named-record in addition to (or instead of) numbers in the global table we could just query against those

i'm sorry i am struggling to see how using names instead of numbers would make a difference in this case. Surely we can do the same with numbers?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the consensus is to remove this, but the idea was that these are treated by this spec as opaque strings used for filtering, and the servers have no requirement to enforce any particular values (unless they want/need to). But happy to have this conversation another day, since we never implemented this anyway.

routing/DELEGATED_CONTENT_ROUTING_HTTP.md Outdated Show resolved Hide resolved
routing/DELEGATED_CONTENT_ROUTING_HTTP.md Show resolved Hide resolved
routing/DELEGATED_CONTENT_ROUTING_HTTP.md Show resolved Hide resolved
These were not implemented so do not belong in this spec. If we
implement them later, we can reintroduce them at that time.
Copy link
Member

@lidel lidel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As agreed a while ago, this is GETs-only that shipped in Kubo 0.18.
We've ratified it a while ago, but stewards had no bandwidth to clean up and merge until now.

I made small editorial changes, filled some gaps, and removed left-overs from removed things (pagination).
Unanswered questions are moved to #377

Merging to unblock follow-up IPIPs @masih and @guseggert want to submit (streaming, PUTs) and future work (peer records, IPNS records, privacy-preserving double hashing etc).

IPIP/0337-delegated-routing-http-api.md Outdated Show resolved Hide resolved
routing/DELEGATED_CONTENT_ROUTING_HTTP.md Outdated Show resolved Hide resolved
routing/DELEGATED_CONTENT_ROUTING_HTTP.md Outdated Show resolved Hide resolved
routing/DELEGATED_CONTENT_ROUTING_HTTP.md Show resolved Hide resolved
IPIP/0337-delegated-routing-http-api.md Outdated Show resolved Hide resolved
routing/DELEGATED_CONTENT_ROUTING_HTTP.md Outdated Show resolved Hide resolved
routing/DELEGATED_CONTENT_ROUTING_HTTP.md Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Ratified
Archived in project
Development

Successfully merging this pull request may close these issues.

8 participants