Consider using a database for storing session data

[DmitryAstafyev]

Context

Chipmunk allows for opening both text and binary files. In the case of a text file, a session file is not created; the opened file itself is used as the session file. When a binary file is opened, Chipmunk decodes the contents of the file and creates a text representation in a new session file. Thus, each decoded message in the original binary file corresponds to one line in the created session file.

This concept is transparent and convenient in terms of searching within the created session file. However, this approach is not flexible and limits the ability to work with session data, especially when the data is structured as a set of fields (columns). In such cases, even basic column sorting becomes impossible.

For working with binary data (files, streams, etc.), it is proposed to use a database as a storage solution for session data. Transitioning to the use of databases is proposed to be carried out in several stages:

Stage 1: Selecting Candidates

• Evaluate and shortlist the current technologies (databases) and arrive at a minimum of two "candidates."

• Criteria for evaluating candidates:

  1. Declared performance (especially search queries).
  2. Support for search queries using regular expressions.
  3. Need for additional software installation. If a candidate requires additional installation of programs, drivers, etc., it should be deprioritized. Ideally, Chipmunk should remain a portable solution (not requiring installation).
  4. The database should be well-represented in the Rust ecosystem, with well-maintained crates and an active repository. A strong community would be a great bonus.

Stage 2: Evaluating Candidates

At this stage, it is necessary to compare the search speed between the current solution and the use of a database. For this, it will most likely be necessary to create a simple session example where the test file (e.g., DLT) is opened, decoded, and searched for its data.

[AmmarAbouZor]

I can provide information about SQLite database engine, since I've worked with it in different Desktop and terminal applications with different programming language (Rust, C++ and C#).

• It's really lightweight and easy to install in portable apps. With Rust we can use the crate sqlx_sqlite which bundles the database driver and compile it with the crate itself, therefore we wouldn't need to add any installation steps.
• It's easy to configure to work with one file only without creating any extra files, which open the possibility for users to share their logs as database files or save them for later.
• Regex is supported but we need to provide and register our own regex function which would be an advantage since we can provide a really high performant regex function written in rust.
• I think the performance will be good enough for our use-case, because SQLite short-comes compared with the other heavy databases are with handling queries in parallel, which is a problem that we don't have since the database will be running on a local machine. The other disadvantage is the size of the data, which I don't think it would make a problem here because the size of logs is still relatively small to the other use-cases where databases are usually used.
• Well supported by the crate SQLx and its sub-crate sqlx_sqlite which is normally to go-to crate for rust databases without ORMs

Here is an example from the creating and dealing with a SQLite database in a terminal app I've created tui-journal