Lassie compatible http api for fetching CARs #34

olizilla · 2023-04-12T11:10:57Z

Offer an http api that Lassie / Saturn could use to fetch CARs from us. Tweak our existing CAR responses to match the Lassie spec.

We already support CAR responses, we just need to tweak the existing code to write traversed blocks to the CAR to make them verifiable, and handle sending subsets of the total dag when directed to by the ?car-scope param.

### Tasks
- [ ] #32
- [x] Ensure expected block ordering (see: https://github.com/filecoin-project/lassie/blob/main/docs/CAR.md#block-deduplication)
- [ ] #33
- [ ] https://github.com/web3-storage/w3up/issues/786
- [x] Test against lassie
- [ ] Dedupe repeated blocks in CAR response (see: https://github.com/filecoin-project/lassie/blob/main/docs/CAR.md#block-deduplication)
- [ ] Ensure Identity CIDs are not included as blocks (see: https://github.com/filecoin-project/lassie/blob/main/docs/CAR.md#dag-depth)

The text was updated successfully, but these errors were encountered:

add getPath method as a generator that returns blocks for the targeted dag and all blocks traversed while resolving a cid+path string supports carScope to specify what blocks to return for the resolved dag - 'all': return the entire dag starting at path. (default) - 'block': return the block identified by the path. - 'file': Mimic gateway semantics: Return All blocks for a multi-block file or just enough blocks to enumerate a dir/map but not the dir contents. see: storacha/freeway#33 see: storacha/freeway#34 TODO: - [] find out how to identify the boundaries of a unixfs hamt (unixfs-exported seems to define it as "not having an empty or null Link.Name after the first 2 chars are stripped, which seems risky... what happens if the actual dir listing has 2 char long link names? see: https://github.com/ipfs/js-ipfs-unixfs/blob/e853049bd63d6773442e1540ae49b6a443ca8672/packages/ipfs-unixfs-exporter/src/resolvers/unixfs-v1/content/hamt-sharded-directory.ts#L20-L42 License: MIT Signed-off-by: Oli Evans <[email protected]>

willscott · 2023-04-24T13:41:28Z

Test against lassie
- using this branch Add an HTTP retrieval protocol filecoin-project/lassie#193 we should be able to confirm things work
Announce HTTP endpoint as an extended family member to the indexer
- there's a multicodec to advertise for supporting http transfer that we'll want to have as an 'extended family member' of the eipfs provider's advertisement stream.

add getPath method as a generator that returns blocks for the targeted dag and all blocks traversed while resolving a cid+path string supports carScope to specify what blocks to return for the resolved dag - `'all'`: return the entire dag starting at path. (default) - `'block'`: return the block identified by the path. - `'file'`: Mimic gateway semantics: Return All blocks for a multi-block file or just enough blocks to enumerate a dir/map but not the dir contents. see: storacha/freeway#33 see: storacha/freeway#34 see: ipfs/specs#402 TODO: - [x] find out how to identify the boundaries of a unixfs hamt ...unixfs-exporter seems to define it as "not having an empty or null Link.Name after the first 2 chars are stripped, which seems loose... what happens if the actual dir listing has 2 char long link names? see: https://github.com/ipfs/js-ipfs-unixfs/blob/e853049bd63d6773442e1540ae49b6a443ca8672/packages/ipfs-unixfs-exporter/src/resolvers/unixfs-v1/content/hamt-sharded-directory.ts#L20-L42 License: MIT --------- Signed-off-by: Oli Evans <[email protected]> Co-authored-by: Alan Shaw <[email protected]>

- update dagula to get `getPath` with carScope support https://github.com/web3-storage/dagula/releases/tag/v6.0.0 - update handleCar to extract ?car-scope query and use `dagula.getPath` BREAKING CHANGE: CARs returned for cid+path will now be rooted at the root cid rather than the resovled cid for the end of the path and include all blocks needed to verify the path was traveresed correctly. see: storacha/freeway#33 see: storacha/freeway#34 License: MIT Signed-off-by: Oli Evans <[email protected]>

- update dagula to get `getPath` with carScope support https://github.com/web3-storage/dagula/releases/tag/v6.0.0 - update handleCar to extract ?car-scope query and use `dagula.getPath` BREAKING CHANGE: CARs returned for cid+path will now be rooted at the root cid rather than the resovled cid for the end of the path and include all blocks needed to verify the path was traveresed correctly. see: storacha/freeway#33 see: storacha/freeway#34 License: MIT --------- Signed-off-by: Oli Evans <[email protected]>

update `dagula` and `gateway-lib` to add supoport for car-scope and verifiable paths for cars see: #34 License: MIT Signed-off-by: Oli Evans <[email protected]>

olizilla · 2023-05-03T11:05:03Z

@willscott is "Announce HTTP endpoint as an extended family member to the indexer" required before we can test against lassie?

willscott · 2023-05-03T11:06:31Z

I don't think so!

Lassie as a CLI App allows retrieval against a manually specified provider endpoint, skipping the indexer lookup.

olizilla · 2023-05-03T11:06:43Z

ah, i see, something something lassie fetch --providers...

willscott · 2023-05-03T11:09:39Z

you'll want to be on the http retrieval branch filecoin-project/lassie#204 in order to test

cc @rvagg

olizilla · 2023-05-03T11:25:49Z

Success!? ✨ 🎷 🐩

~/Code/filecoin-project/lassie on rvagg/http  
❯ ./lassie fetch --providers /dns4/freeway.dag.haus/tcp/443/https/p2p/bafzbeibhqavlasjc7dvbiopygwncnrtvjd2xmryk5laib7zyjor6kf3avm bafybeihetumsy24mjzbts7t4vpft2rwput44joxxnzxhfh5woq6z46fe2y
Fetching bafybeihetumsy24mjzbts7t4vpft2rwput44joxxnzxhfh5woq6z46fe2y from [{QmQzqxhK82kAmKvARFZSkUVS6fo9sySaiogAnx5EnZ6ZmC: [/dns4/freeway.dag.haus/tcp/443/https]}]...........
Fetched [bafybeihetumsy24mjzbts7t4vpft2rwput44joxxnzxhfh5woq6z46fe2y] from [QmQzqxhK82kAmKvARFZSkUVS6fo9sySaiogAnx5EnZ6ZmC]:
	Duration: 809.277584ms
	  Blocks: 11
	   Bytes: 3.2 MiB

❯ ipfs-car --list-full bafybeihetumsy24mjzbts7t4vpft2rwput44joxxnzxhfh5woq6z46fe2y.car
bafybeihetumsy24mjzbts7t4vpft2rwput44joxxnzxhfh5woq6z46fe2y bafybeihetumsy24mjzbts7t4vpft2rwput44joxxnzxhfh5woq6z46fe2y
bafkreidpr7zz5yflyocmglpref5vvx4yglo3zmihh3mpiaezecgrggqwiq bafybeihetumsy24mjzbts7t4vpft2rwput44joxxnzxhfh5woq6z46fe2y/johnny-5-goes-camping.jpg
...etc

Minor: you get an error if you try and provide a more succinct http flavour multiaddr sans p2p like /dns4/freeway.dag.haus/tcp/443/https

~/Code/filecoin-project/lassie on rvagg/http  
❯ ./lassie fetch --providers /dns4/freeway.dag.haus/tcp/443/https bafybeihetumsy24mjzbts7t4vpft2rwput44joxxnzxhfh5woq6z46fe2y
2023-05-03T12:24:48.376+0100	FATAL	lassie	lassie/main.go:57	invalid p2p multiaddr

olizilla · 2023-05-03T11:28:51Z

cid+path works

./lassie fetch --providers /dns4/freeway.dag.haus/tcp/443/https/p2p/bafzbeibhqavlasjc7dvbiopygwncnrtvjd2xmryk5laib7zyjor6kf3avm bafybeihetumsy24mjzbts7t4vpft2rwput44joxxnzxhfh5woq6z46fe2y/johnny-5-is-cowboy.jpg
Fetching bafybeihetumsy24mjzbts7t4vpft2rwput44joxxnzxhfh5woq6z46fe2y/johnny-5-is-cowboy.jpg from [{QmQzqxhK82kAmKvARFZSkUVS6fo9sySaiogAnx5EnZ6ZmC: [/dns4/freeway.dag.haus/tcp/443/https]}]..
Fetched [bafybeihetumsy24mjzbts7t4vpft2rwput44joxxnzxhfh5woq6z46fe2y] from [QmQzqxhK82kAmKvARFZSkUVS6fo9sySaiogAnx5EnZ6ZmC]:
	Duration: 403.931916ms
	  Blocks: 2
	   Bytes: 443 KiB

olizilla · 2023-05-03T11:31:25Z

--car-scope file is working: only directory block returned for unixfs dir.

❯ ./lassie fetch --providers /dns4/freeway.dag.haus/tcp/443/https/p2p/bafzbeibhqavlasjc7dvbiopygwncnrtvjd2xmryk5laib7zyjor6kf3avm --car-scope file bafybeihetumsy24mjzbts7t4vpft2rwput44joxxnzxhfh5woq6z46fe2y
Fetching bafybeihetumsy24mjzbts7t4vpft2rwput44joxxnzxhfh5woq6z46fe2y from [{QmQzqxhK82kAmKvARFZSkUVS6fo9sySaiogAnx5EnZ6ZmC: [/dns4/freeway.dag.haus/tcp/443/https]}].
Fetched [bafybeihetumsy24mjzbts7t4vpft2rwput44joxxnzxhfh5woq6z46fe2y] from [QmQzqxhK82kAmKvARFZSkUVS6fo9sySaiogAnx5EnZ6ZmC]:
	Duration: 390.517625ms
	  Blocks: 1
	   Bytes: 681 B

❯ ipfs-car --list-cids bafybeihetumsy24mjzbts7t4vpft2rwput44joxxnzxhfh5woq6z46fe2y.car 
bafybeihetumsy24mjzbts7t4vpft2rwput44joxxnzxhfh5woq6z46fe2y

olizilla · 2023-05-03T11:49:01Z

How to announce what we have over http to the indexers needs some discussion as we only support dag roots via http today storacha/w3up#786

heyjay44 mentioned this issue Apr 12, 2023

Tracking saturn integration 🪐 storacha/w3up#754

Closed

olizilla mentioned this issue Apr 17, 2023

feat: getPath with carScope storacha/dagula#8

Merged

1 task

lidel mentioned this issue Apr 17, 2023

IPIP-402: Partial CAR Support on Trustless Gateways ipfs/specs#402

Merged

olizilla mentioned this issue May 1, 2023

feat!: support ?car-scope and verifiable paths for format=car storacha/gateway-lib#21

Merged

olizilla added a commit that referenced this issue May 2, 2023

feat!: support ?car-scope and verifiable paths for format=car

ec64e53

update `dagula` and `gateway-lib` to add supoport for car-scope and verifiable paths for cars see: #34 License: MIT Signed-off-by: Oli Evans <[email protected]>

olizilla mentioned this issue May 2, 2023

feat!: support ?car-scope and verifiable paths for format=car #35

Merged

olizilla mentioned this issue May 3, 2023

HTTP protocol filecoin-project/lassie#204

Merged

olizilla self-assigned this May 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lassie compatible http api for fetching CARs #34

Lassie compatible http api for fetching CARs #34

olizilla commented Apr 12, 2023 •

edited

Loading

willscott commented Apr 24, 2023 •

edited by alanshaw

Loading

olizilla commented May 3, 2023

willscott commented May 3, 2023

olizilla commented May 3, 2023

willscott commented May 3, 2023

olizilla commented May 3, 2023 •

edited

Loading

olizilla commented May 3, 2023

olizilla commented May 3, 2023

olizilla commented May 3, 2023

Lassie compatible http api for fetching CARs #34

Lassie compatible http api for fetching CARs #34

Comments

olizilla commented Apr 12, 2023 • edited Loading

willscott commented Apr 24, 2023 • edited by alanshaw Loading

olizilla commented May 3, 2023

willscott commented May 3, 2023

olizilla commented May 3, 2023

willscott commented May 3, 2023

olizilla commented May 3, 2023 • edited Loading

olizilla commented May 3, 2023

olizilla commented May 3, 2023

olizilla commented May 3, 2023

olizilla commented Apr 12, 2023 •

edited

Loading

willscott commented Apr 24, 2023 •

edited by alanshaw

Loading

olizilla commented May 3, 2023 •

edited

Loading