SQLite database scalability

[ghost]

Hello,
What is the current strategy for scaling the currently available SQLite database?
Does it employ a partitioning scheme at this moment?

Thanks in advance

[hendrikvanantwerpen]

Hi @augmented-fog! The current SQLite-based database implementation is more of a proof-of-concept, really. It is functional, but not optimized, and performs rather poorly, in my experience.

[hendrikvanantwerpen]

For JanusGraph, I analyzed the stack graph logic from this library, and re-implemented it as a gremlin query.

That sounds very interesting. What does this mapping look like?

[hendrikvanantwerpen]

The cool thing about the JanusGraph approach was that there was no need to do "partial paths". You could simply throw the entire graph to it.

That sounds pretty good. Were these graphs for individual files, a whole repository or source project?

[ghost]

Well, first of all, I was not clever enough to compute paths in advance as it is done in this library.

The implementation was a Gremlin traversal, which would start at a reference node, and it would follow edges based on constraints which are essentially the same constraints that this library implements. To achieve this, Traverser#sack was used to store the state of the traversal (i.e.: a Groovy Stack object), and then the choose step would be used to handle different cases such as comparing the top of the stack with the symbol in the node based on node type, etc. The resulting query was really long and complex.

I used this to store a whole project. The indexing time was several times slower than your SQLite solution.
But I guess if you used a fancier JanusGraph backend and bulk insertion, you could accelerate this further.

Another cool thing is that some gremlin clients have visualizers. So you can run an arbitrary query on the graph and get a graphical representation of the results.

[hendrikvanantwerpen]

Interesting! I am not familiar with Gremlin or JanusGraph. Is that query something you could share?

[hendrikvanantwerpen]

🙇🏻