fuse pb regression: could not convert protobuf or raw node #9044
Comments
bmwiedemann
added
kind/bug
|
also reproduced with official https://github.com/ipfs/go-ipfs/releases/download/v0.13.0/go-ipfs_v0.13.0_linux-amd64.tar.gz |
aschmahmann
added
P2
The FUSE mount functionality of IPFS was broken by the update to v0.13.0, so disable it. Hopefully it will be fixed soon. See ipfs/kubo#9044.
|
@Jorropo wonder if this might be related to invalid codec mapping in go-cid, see BREAKING CHANGE fix in https://github.com/ipfs/go-cid/releases/tag/v0.2.0 Really concerning that we have no tests for FUSE that would catch this regression – whoever will be fixing this, should also add sharness test for UnixFS DAG that has both dag-pb and raw file items. |
|
By debugging with ipfs 0.12 and 0.17, using the followings command I could find that there is a difference of path taken in kubo/fuse/readonly/readonly_unix.go Line 92 in a6f446a In ipfs 0.12, nd is of type It then goes in here https://github.com/ipfs/kubo/blob/master/fuse/readonly/readonly_unix.go#L95 and get the substrate and run In ipfs 0.17, nd is of type Then it goes in https://github.com/ipfs/kubo/blob/master/fuse/readonly/readonly_unix.go#L95 and ProtoNodeConverter returns an error because nd is not a I'm not familiar with those concepts, so the amount of knowledge to have to debug this issues seems a bit overwhelming to me. But with some guidance, may be I can help. Most of my day to day life relies on using ipfs mount and using ipfs content like any other files. It has been very annying to have to resort to ipfs cat, ipfs get or the gateway to just read a file. Therefore, I am quite motivated to make this work. Of course, during the time I can spare to this task. I guess my next action in this project would be to try to understand what are those PBNode about, right? |
|
PB probably stands for ProtoBuf which is the encoding-method of metadata. E.g. the DAG-PB (Directed Acyclic Graph) contains a list of entries that each have Size, Name, CID In (unrelated) https://github.com/bmwiedemann/ipfs-iso-jigsaw/blob/master/ipfsjigsaw.py I also construct a custom PBNode from python. |
|
Thank you
|
|
Digging into ipfs, ipld, PBNode, ADL etc looks like a pretty big rabbit hole to me. I realized that, in term of functionality, the ipfs gateway is very similar to the fuse mount point: it provides access to the data. So I figure there should be also a lookup-like method in the gateway code. I intuitively think that if I find this code, I can either
Does this idea make sense? |
|
This problem still exists in 0.21.0 |
|
BTW, my ongoing investigation on the subject is here -> https://konubinix.eu/braindump/f9f882e4-0d84-453e-b3aa-5e34e411b90d?title=ipfs_mount_0_12_2_does_not_work_for_several_files It moves into my someday/maybe list, because I have more important matters now, and learning about ipfs, ipld, boxo and stuffs like that needs quite a lot of cognitive load. But because I have the frustration everyday of having to work around this issue, I will definitely continue working on this, so stay tuned :-). Of course, if someone fix it before I do, I would be verrrrry grateful. |
|
still broken in 0.22.0 |
|
I ran a git bisect between 0.12.2 and 0.13.0 and found Specifically reverting this part, fixes fuse: - github.com/ipfs/go-graphsync v0.11.0
+ github.com/ipfs/go-graphsync v0.13.1
- github.com/ipfs/go-unixfsnode v1.1.3
+ github.com/ipfs/go-unixfsnode v1.4.0and I further narrowed it down to 12 commits in |
|
It seems, ipfs/go-unixfsnode#14 by @willscott broke fuse. |
|
Thanks for tracking this down, @bmwiedemann ! It's not obvious to me exactly how the update of unixfsprime from 2 years ago broke fuse. One thing that is immediately clear is that this is an unhelpful error message, in that the actual error is not printed. |
|
with 10102 ontop of 0.22.0 I get |
|
My guess at fixing the immediate regression is ipfs/go-unixfsnode#67 |
|
new version of go-unixfsnode released with that change and a few other fixes |
Fixes ipfs#9044 regression in 0.13.0 FUSE access to some files
Fixes ipfs#9044 regression in 0.13.0 FUSE access to some files
Fixes ipfs#9044 regression in 0.13.0 FUSE access to some files
Fixes #9044 regression in 0.13.0 FUSE access to some files
|
Reopening as Details and repro steps: #10243 (comment) – take a look if you had good intuition on this before 🙏 |
lidel
changed the title
|
See also #10242, which I created just for this new problem. |
|
Triage notes:
We won't have bandwidth to fix this any time soon, |
Checklist
Installation method
built from source
Version
Config
{ "API": { "HTTPHeaders": {} }, "Addresses": { "API": "/ip4/127.0.0.1/tcp/5001", "Announce": [ "/ip4/10.162.191.176/tcp/4001", "/ip4/176.9.1.211/tcp/34176", "/ip4/10.162.191.176/udp/4001/quic", "/ip4/176.9.1.211/udp/34176/quic" ], "Gateway": "/ip4/127.0.0.1/tcp/8080", "NoAnnounce": [], "Swarm": [ "/ip4/0.0.0.0/tcp/4001", "/ip6/2620:113:80c0:8130:ec4:7aff:fe57:1808/tcp/4001", "/ip4/0.0.0.0/udp/4001/quic", "/ip6/2620:113:80c0:8130:ec4:7aff:fe57:1808/udp/4001/quic" ] }, "Bootstrap": [ "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN", "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa", "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb", "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt", "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ", "/ip4/104.236.179.241/tcp/4001/p2p/QmSoLPppuBtQSGwKDZT2M73ULpjvfd3aZ6ha4oFGL1KrGM", "/ip4/128.199.219.111/tcp/4001/p2p/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu", "/ip4/104.236.76.40/tcp/4001/p2p/QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64", "/ip4/178.62.158.247/tcp/4001/p2p/QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd", "/ip6/2604:a880:1:20::203:d001/tcp/4001/p2p/QmSoLPppuBtQSGwKDZT2M73ULpjvfd3aZ6ha4oFGL1KrGM", "/ip6/2400:6180:0:d0::151:6001/tcp/4001/p2p/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu", "/ip6/2604:a880:800:10::4a:5001/tcp/4001/p2p/QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64", "/ip6/2a03:b0c0:0:1010::23:1001/tcp/4001/p2p/QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd", "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ" ], "Datastore": { "BloomFilterSize": 0, "GCPeriod": "1h", "HashOnRead": false, "Spec": { "mounts": [ { "child": { "path": "blocks", "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2", "sync": true, "type": "flatfs" }, "mountpoint": "/blocks", "prefix": "flatfs.datastore", "type": "measure" }, { "child": { "compression": "none", "path": "datastore", "type": "levelds" }, "mountpoint": "/", "prefix": "leveldb.datastore", "type": "measure" } ], "type": "mount" }, "StorageGCWatermark": 90, "StorageMax": "90GB" }, "Discovery": { "MDNS": { "Enabled": true, "Interval": 10 } }, "Experimental": { "FilestoreEnabled": false, "Libp2pStreamMounting": false, "P2pHttpProxy": false, "QUIC": false, "ShardingEnabled": false, "UrlstoreEnabled": true }, "Gateway": { "APICommands": [], "HTTPHeaders": { "Access-Control-Allow-Headers": [ "X-Requested-With", "Range", "User-Agent" ], "Access-Control-Allow-Methods": [ "GET" ], "Access-Control-Allow-Origin": [ "*" ] }, "NoFetch": false, "PathPrefixes": [], "RootRedirect": "", "Writable": false }, "Identity": { "PeerID": "QmbBikAJAnTrry9XEVxc1G3YTVi13M1GHU2fk1m1Tag4Am" }, "Ipns": { "RecordLifetime": "", "RepublishPeriod": "", "ResolveCacheSize": 128 }, "Mounts": { "FuseAllowOther": false, "IPFS": "/ipfs", "IPNS": "/ipns" }, "Pubsub": { "DisableSigning": false, "Router": "", "StrictSignatureVerification": false }, "Reprovider": { "Interval": "12h", "Strategy": "all" }, "Routing": { "Type": "dht" }, "Swarm": { "AddrFilters": null, "ConnMgr": { "GracePeriod": "20s", "HighWater": 900, "LowWater": 600, "Type": "basic" }, "DisableBandwidthMetrics": false, "DisableNatPortMap": false, "DisableRelay": false, "EnableAutoNATService": false, "EnableAutoRelay": false, "EnableRelayHop": false } }Description
I noticed that a number of my files stopped working in the fuse mount after upgrade from go-ipfs 0.12.0 to 0.13.0 e.g.
while
ipfs catstill works fine.It seems to affect all the bafybei (pb) objects, but the bafkrei (raw data) files work fine.
The text was updated successfully, but these errors were encountered: