nsqd: refactor storage engine #1170
Conversation
andyxning
force-pushed
the
refactor_nsqd_storage_engine
branch
8 times, most recently
from
7fbab14
to
a593551
Compare
|
/cc @mreiferson @ploxiln This PR is ready for review. PTAL. |
andyxning
changed the title
andyxning
force-pushed
the
refactor_nsqd_storage_engine
branch
from
a593551
to
29bd197
Compare
andyxning
force-pushed
the
refactor_nsqd_storage_engine
branch
from
29bd197
to
5c219dc
Compare
|
what's not great about |
@ploxiln I think it deserves adding the these dependencies since msgpack will make it faster and denser than json or plain bytes. This will store more messages on the same disk and write to disk more messages. Actually, when the functionality grows, the project will get more and more complex such as logic and dependency. Kubernetes is an example. |
|
/cc @ploxiln @mreiferson |
|
/cc @jehiah |
1 similar comment
|
/cc @jehiah @mreiferson @ploxiln Anyone can help to take a look at this? It has waited for more than 24 days. :( |
|
There are msgpack libraries which do not have dependencies. While msgpack is more efficient than json, especially for numbers and byte buffers, this is still using strings for keys, instead of a single byte. So, I'm still not a fan of this approach. |
Using single byte for keys will force us to manage parsing keys manually and it is obviously error-prone. This is even make the parsing logic more complicated when a new field added. By using json or msgpack we offload this parsing logic to json or msgpack and they have done quite well in marshal and unmarshal in normal cases and abnormal cases such as some field loss or newly added. We need a tradeoff between maintainability and storage efficiency. In this case, i prefer maintainability. Btw, @ploxiln thanks for reviewing this. Can you reach @mreiferson @jehiah personally for informing them help to review this PR. This PR has been made for more than 30d. :( |
|
@mreiferson recently started at a new job/employer, and is waiting to get official authorization to contribute to NSQ |
|
OK, let's be more patient. :) |
|
@mreiferson Ping. |
|
@ploxiln @mreiferson We should work on this PR for now. :) |
|
We need to push this PR forward. @ploxiln Do you know who is also actively working on nsq. We can discuss this on slack or mailing list. Anyway, we need to support this ASAP. This has been delayed by more than one month. :( |
|
@ploxiln @jehiah @mreiferson Can you please take a look at this. Internally, i have upgraded a cluster and everything works fine. |
|
But you're running a fork of nsq 0.3.8 right? So it doesn't really benefit your organization much if this is merged? (I'm curious why 0.3.8 - which of the interfaces removed in 1.0 are you dependent on?) |
Nothing. We just run 0.3.8 well and nothing new feature in versions after 0.3.8 we need. |
Yes. But i think with this PR nsqd can support more complicated features easily, such as the compression feature. |
mreiferson
changed the title
|
Hi, is there any plan to release the new version of |
|
Hi @ploxiln @jehiah @mreiferson |
Fix #1139
Fix #1137