0.0.5-rc1

Release notes

The primary feature made available with this release is the "indexed-segmented-log"

laminarmq specific enhancements to the segmented_log data structure

While the conventional segmented_log data structure is quite performant for a commit_log implementation, it still requires the following properties to hold true for the record being appended:

• We have the entire record in memory
• We know the record bytes' length and record bytes' checksum before the record is appended

It's not possible to know this information when the record bytes are read from an asynchronous stream of bytes. Without the enhancements, we would have to concatenate intermediate byte buffers to a vector before appending to the segment. This would not only incur more allocations but also slow down our system.

Hence, to accommodate this use case, we introduced an intermediate indexing layer to our design.

Fig: Data organisation for persisting the segmented_log data structure on a *nix file system.

In the new design, instead of referring to records with a raw offset, we refer to them with indices. The index in each segment translates the record indices to the raw file position in the segment store file.

Now, the store append operation accepts an asynchronous stream of bytes instead of a contiguously laid-out slice of bytes. We use this operation to write the record bytes, and at the time of writing the record bytes, we calculate the record bytes' length and checksum. Once we are done writing the record bytes to the store, we write its corresponding record_header (containing the checksum and length), position and index as an index_record in the segment index.

This provides two quality-of-life enhancements:

• Allow asynchronous streaming writes, without having to concatenate intermediate byte buffers
• Records are accessed much more easily with easy-to-use indices

Now, to prevent a malicious user from overloading our storage capacity and memory with a maliciously crafted request which infinitely loops over some data and sends it to our server, we have provided an optional append_threshold parameter to all append operations. When provided, it prevents streaming append writes to write more bytes than the provided append_threshold.

At the segment level, this requires us to keep a segment overflow capacity. All segment append operations now use segment_capacity - segment.size + segment_overflow_capacity as the append_threshold value. A good segment_overflow_capacity value could be segment_capacity / 2.

The new indexed segmented log implementation is present under the storage::commit_log::segmented_log module.

Why is this a release candidate?

We intend to move from the old segmented log implementation to the new indexed version going forward. However, the new implementation is not API-compatible with the old implementation. Hence other modules which depend on the segmented log such as the server::* modules, might need to be re-rewritten. Hence, I am making this release candidate available before this drastic change as a safety measure.

The next release will focus on integration with the server modules, impls for different async runtimes and better documentation coverage.

What's Changed

• Provide necessary abstractions for the REST API server by @arindas in #7
• Return spawned task from HyperExecutor::serve() to better handle task cleanup. by @arindas in #8
• Adds streaming append capability. by @arindas in #11
• Add development updates from #11 by @arindas in #12
• doc: fix doc badge urls, updates README by @arindas in #13
• Adds doc updates from #13 by @arindas in #14

Full Changelog: 0.0.4...0.0.5-rc1