nsqd: enable compression/decompression pipeline from producer to consumer

[andyxning]

NSQ currently supports snappy compression and decompression for TCP-protocol based producer and consumer. But it has some shortcomings.

• the memory and disk capacity nsqd used can not be decreased since all messages are stored just like what nsqd receives, i.e., the original messages.
• the in network bandwidth can not be decreased since the message are not compressed before it is send.
• they are just available for TCP protocol based producer and consumer.

I have made a proposal for this: https://docs.google.com/document/d/107yDH6pN8-b_i22AfvPkzdDU2yneOC-1lcUllpUmN4w/edit#heading=h.tla1oogr3wac

/cc @mreiferson @ploxiln

[andyxning]

@ploxiln This will rely on your re-design about the nsq storage backend format.

[mreiferson]

Hi @andyxning, thanks for this proposal.

I'm trying to understand the motivation here, is it to reduce on-disk and in-memory footprint? Have you considered compressing the data in your producer or consumer instead? Remember, to nsqd the messages are just bytes...

[ploxiln]

the in network bandwidth can not be decreased since the message are not compressed before it is send

The network bandwidth is reduced, that's the only thing that snappy or deflate compression of the tcp-protocol does.

the memory and disk capacity nsqd used can not be decreased

this is true, the messages are not compressed in memory or on disk, only "over the wire"

they are just available for TCP protocol based producer and consumer

TCP protocol consumer is the only kind of consumer. For the producer, the HTTP /pub request could use Content-Encoding: gzip ... I'm not 100% sure if that works with the nsqd http endpoint today, but if not, it would be an easy stand-alone change to make it work.

So what remains is some way for messages to be compressed in memory and disk queues. As @mreiferson says the producer and consumer can be made to use any message format, including compressed, so that's an easy thing a user can do today. I suppose it would be a nice additional feature to optionally have nsqd "take care of it" for the common case of text messages ... and it would be nice to avoid double-compressing messages over the wire (without giving up compression of control frames). I haven't read through and digested the proposal yet ...

[andyxning]

@mreiferson @ploxiln PTAL.

[mreiferson]

let's continue this conversation in #1149