Using IPFS in the OpenStreetMap infrastructure

[RubenKelevra]

I've written a ticket on the OSM operation tracker, regarding the possible use of IPFS for their mirror servers - and as an archive:

openstreetmap/operations#388

[bertrandfalguiere]

For what it's worth, there is a partial mirror on IPFS mentionnend at https://wiki.openstreetmap.org/wiki/Planet.osm .
It's https://ipfs.io/ipns/ipfs.placebazaar.org

[RubenKelevra]

Yes, I saw that. But it only contains files without any information about the date they have been added. It might be months out of date.

Apart from this, I tried to receive a file from this "mirror"... well after 10 minutes I got 4 nodes of the europe file.

I guess the mirror is long gone and someone is holding parts of the content in his cache.

[bertrandfalguiere]

The OSM team seems pretty hostile. I think the smart move if we want support is to maintain an IPFS mirror of OSM, and show them some numbers down the road, or to become the go-to server for IPFS users.

[RangerMauve]

Relevant: https://peermaps.org/

[RubenKelevra]

@bertrandfalguiere well I wouldn't say hostile, but not amused about the 'demand' for additional work. They are pretty stressed right now, because of high server loads because of corona induced outages and several replacements going on additionally.

I'm always worried about my tone, to native speakers ... and this time once more I get the feeling my tone was wrong. :/

I don't want to push them to do anything, but I already reserved enough space to add 1-2 redundancies in 2 European countries.

Doing this for ourselves means rsync to IPFS work again, I just wrote a script which does this, and it's not as simple as it might sound like - except if you do just full replacements every time something changes. Delta updates, on the other hand, are pretty well, shitty.

It's much easier to not parse rsync logs and do the work just while generating the files anyway.

And we would have to rely on the md5 files, which I really don't like.

[RubenKelevra]

Relevant: https://peermaps.org/

This uses the dat protocol, which reads like a very basic copy of IPFS 🤔

[RangerMauve]

Dat and IPFS are pretty different under the hood. Neither is really a copy of the other other than some basic similarities liker having p2p networking and using merkle DAGs.

The thing about peermaps is that there's a bunch of non-dat libraries in there for distributing the actual OSM data cross multiple peers and being able to process that into maps which is hard to do with the raw data unless you have some sort of superpeer.

[RubenKelevra]

The thing about peermaps is that there's a bunch of non-dat libraries in there for distributing the actual OSM data cross multiple peers and being able to process that into maps which is hard to do with the raw data unless you have some sort of superpeer.

Yes true. Are you part of the project? Maybe it's interesting to take a look how this would perform in an IPFS cluster?

[bertrandfalguiere]

There's a grant in the pipe to tackle this: ipfs/devgrants#41

[okdistribute]

Hi 👋 I'm from the peermaps project. We will be pushing the files to dat, ipfs, and also hopefully filecoin.