Add scattered Get and Put requests

[nemanja-boric-sociomantic]

Splitting records apart would be a lot more appealing with a scatter-put and gather-get operation, so that we could perform simple joins on the primary key.

Ie, get records from channel X and Y both with key K, sending them as a single packet. This will allow the applications to avoid do manual joins, which are very painful.

[don-clugston-sociomantic]

For Neo, I think this could initially be done on the client side, without changing the server at all. Merging the gets and puts into a single packet would be a later optimization.

[gavin-norman-sociomantic]

For the Get side of the equation, I don't think it'd even need to be a separate request. The assignment method could just accept a list of one or more channel names.

I'm not sure how much sense it makes for Put, though. Surely the record written to each channel will be different? The only gain would be that the key would only need to be transmitted once, but I don't think that's a big improvement, when compared with the significantly more complicated API a request like this would require.

[don-clugston-sociomantic]

The only reason for wanting a scatter put, is because of notifications. If one put succeeds but the other times out, what do you do? You're not finished until they're both puts are finished, and you don't even know which one will finish first. So the logic gets a little bit complicated. But that logic is the same for every app that does a scatter push, so I think it makes sense for it to be in a library.

More interesting is that if both subrecords are actually stored on the same node, though, and are sent as a single packet, then it'd be pretty close to an ACID operation, which it'd be a very nice feature. Right now, splitting a record into smaller subrecord introduces new ways that database integrity can fail. So it's a downside of splitting records up.

[gavin-norman-sociomantic]

Good point. The API would be more complicated than single Put requests, but the logic around the operation would be simpler, and the atomicity would certainly be an advantage.

[gavin-norman-sociomantic]

The more I think about this idea, the more I like it. neo 4 ever.

[joseph-wakeling-sociomantic]

For the gather-get, how would you handle the case of some subset of the requested records being missing? e.g. if I request the records for key k from channels A, B and C, but a record for key k does not exist in channel C, what is the expected/wanted behaviour?

[nemanja-boric-sociomantic]

I would probably pass the value for the missing record to a notifier as an empty array, or not pass it back at all (with the slight preference to the former)

[joseph-wakeling-sociomantic]

Yes, agree. It should be up to the receiver to decide how to handle that case.