nsqd: fix the issue of deferred data persistence

[qaqhy]

Fix the issue of deferred data persistence loss(Note: Incompatible with old data storage methods)
This is a new data structure storage method, perhaps the method with flag bits below is easier to expand

For disk persistence we can perhaps try to do forward compatible iterative updates using flag bits

The later updates refer to the mode of PR 1137, which is Forward compatibility with the old data

[mreiferson]

Thanks for submitting this @qaqhy. I've got to refresh my memory on where #1170 left off to be able to provide feedback here.

[qaqhy]

Thanks for submitting this @qaqhy. I've got to refresh my memory on where #1170 left off to be able to provide feedback here.

In this commit, I merged the latest code and fixed the data race issue

[qaqhy]

@mreiferson Hello, do you think this restart retention delay time plan is feasible? Or do you have any better suggestions?

[mreiferson]

Hi, thanks for submitting this.

I looked back over #1170. While it had some issues in terms of the specific implementation, I think overall it has a more generic forward compatible design, rather than specifically modifying the on-disk format to support just the issue of deferred persistence.

In other words, if we're going to change the on-disk format, since this will be the first time we've basically ever done that, we should establish a new format that allows us to properly version and expand capabilities while being able to better preserve forward and backward compatibility.

[qaqhy]

Hi, thanks for submitting this.

I looked back over #1170. While it had some issues in terms of the specific implementation, I think overall it has a more generic forward compatible design, rather than specifically modifying the on-disk format to support just the issue of deferred persistence.

In other words, if we're going to change the on-disk format, since this will be the first time we've basically ever done that, we should establish a new format that allows us to properly version and expand capabilities while being able to better preserve forward and backward compatibility.

Another issue is that when we read tasks from a topic and are unable to consume them and throw them back into the queue, it slows down the loading speed of tasks from the disk into memory. We only read the disk when there are no tasks in memory. Do we need to change it to automatically load tasks from the disk into memory when the memory queue is not full.