Using Reed-Solomon codes to transfer data between the client - server

[LazarenkoA]

Hi, I tried to create a test example using this repository, and everything works fine. I was wondering if Solomon codes can be applied to the task of data exchange between two systems, but I can't figure out how to use this package for such a task.

Here's what I have in mind: there's a client and a server, where the client sends chunks (a large file in parts) to the server. I want the server to check the Reed-Solomon codes after receiving all the chunks and restore integrity if some chunks were lost during transmission.

The only idea I could come up with is splitting the process into two steps on the client side: step 1 – encoding the entire file, step 2 – sending the file, and then decoding it on the server. But what if I don't want to encode the entire file? That is, I have a transport layer that receives []byte from another application layer. I wouldn't want to move the Reed-Solomon encoding outside of the transport layer.

[klauspost]

Applying the correct RS to transport is tricky. If you have unreliable transport I assume you are using something like UDP.

I have seen approaches where each UDP package is encoded as a shard, with say 40000 data packets and 20000 RS packets are sent in groups. Exact numbers are not important, but total below 65536. Since the UDP packets are fairly small, that can be done with a single Encode/Decode. Each packet then also contains the index and a checksum (xxhash3 for example).

If you are worried about long-stretch errors, you can use EncodeIdx on say 16 groups and send one or two parity shards in case you lose more than 20K UDP blocks in a group.

Doing long shard encodes can also be done, but typically require files to be on disk, which is cumbersome, annoying and slow in a transport. So don't think in files, but do RS on data that that can fit in memory.