Probably the fastest node database in the world, blazing simple, uses nodejs n-api interface, designed to work in node clusters. It is also very low level database, so inappropriate handling might lead to data corruption. Read documentation carefully before you use it.
Works on 64 bit linux platforms. Other platforms not tested.
Some examples of usage:
- logging data
- storing sessions, large numger of documents
- building custom database data structures (trees, chain lists, ...)
- microservices
- services
- storing training data of ai, ann, ... storing trained results...
- building more advanced databases with ultradb as their backend
- everywhere where you need document ID without additional querying for it (you receive document ID always as a result of adding document to database)
It:
- is small - arround 300kB
- is written in C for node js intentionally - it is not a wrapper, or port of another database
- is multicore processors friendly - scales up with node processes, multiple node instances/forks can work simultaneously on same database file
- is pretty simple and straight forward - not intended to hide any kind of abstraction from you
- supports text, integers, floats and data buffers
- supports cursors that allows to iterate through documents starting from newest and allows paging
- is VERY FAST, try it on SSD drive, compare with other databases, tell me the results - I'm sure you'll be pleased when you'll compare it with other databases for node js
- is Binary JSON (BSON) and other binary formats friendly
If performance and/or costs of your infrastructure is for you a key factor you should consider use of Ultra DB. It is simple to use library, but requires little more attention from developer and more extensive testing, since there is not much control over what you do with database (for example there is no guarantee that database will trigger an error when trying to update document with non existing ID within a range of IDs in database, more over it might corrupt database). But in return you get a database, that runs within node process (no extra process, no extra overheads for TCP/IP stack, no waisteless copying data, no parsing queries, no switching to additional db tasks, no callbacks and promisses overheads ... and so on). With very little effort you can make pretty advanced services serving many thousands requests a second on a single cheap VPS machine.
Feel free to drop a line of code, write tests, prepare documentation, tutorials, guidelines, come up with ideas and so on!
It is unsafe to allow access to documents by their id from any kind of public api (unless all documents are fixed size and you control modulo of document ID). In case of public API additional step of document id validation should be taken (combining document ID and random key saved in the document as public document ID, accessing via another fixed size documents lookup database, hashing id or hasing id combined with kind of checksum ... and so on). Current API profides set of document ID checksum functions (8 - 32 bit) and unique key for checksums for each database file - this will be helpfull to prevent accessing non existing documents.
If document id's are used internally only and you have full control of how are they used, then you don't have to worry about above.
If you need full ACID database you have to design data and recovery mechanisms yourself. But if you only add new records and read them, you're safe, you might consider this database kinda of ACID in that case.
If you're going to use db in nodejs cluster make sure you use transactions where it needs to be synchronized across processes.
Since transactions are blocking other processes that want to use particular database file/files it is a good idea to split data to databases for different document types each.
Newly added documents are hidden till their content is filled into a database.
64 bit integers currently are actually limited to 53 bits. This is due javascript limitation for numbers, not database itself.
Before you start, check examples located in "tests" directory on github repo: https://github.com/DanielMazurkiewicz/ultradb/tree/master/tests
<header> <document1 data> <document1 length> ... <documentN data> <documentN length>
Document ID points indirectly to address of most significant byte of document length. Most significant byte contains additional 3 bit information:
- most significant bit if set to 1 means that document is hidden
- second and third most significant bit encodes document length field size:
- 00 : 1 byte
- 01 : 2 bytes
- 10 : 4 bytes
- 11 : 8 bytes
Since Document ID is indirect address it is technically possible to remove phisically documents from the beginning of database while keeping same Document IDs.
const UltraDB = require('ultradb');
const testDb = UltraDB('./myDatabase.udb');
This will open database.
const test1Db = UltraDB('./myDatabase.udb, 2048');
This will open database with specified page size, default is 16384. Page size has an impact on performance, the bigger the better performance. The best if it is multiplication of expected average document size.
testDb.close();
Database doesn't require to be closed so in most of cases you can skip this, however, if you're going to open dynamically a lot of databases it is good practice to close the ones you're not using any more.
const documentId = testDb.addUtf8z("My frirst ultradb document");
This will add new document to database, and if successfull, it will return its id;
const documentContent = testDb.getUtf8z(documentId);
This will retrieve document from database.
If document is hidden it will return "null"
If document doesn't exist it will return "undefined"
Note that "documentId" parameter is examined only briefly, so providing appropriate value is on your responsibility. Providing inappropriate value might lead to strange behaviour.
Note that it is your responsibility to use appropriate method to read appropriate data type, database doesn't encode document type.
const testDb = UltraDB('./dbTestCursor.udb').default('Utf8z');
Adds shorten names for methods containing given default string. For above example:
const documentId = testDb.add('some document');
will be equivalent to
const documentId = testDb.addUtf8z('some document');
Method returns current database instance
testDb.isEmpty();
Returns true if database is empty
const documentId = testDb.last();
It returns "undefined" if there is no documents in database. This method skips all hidden documents. If you need to get latest document, including hidden ones use:
const documentId = testDb.lastOfAll();
const previousDocumentId = testDb.previous(documentId);
It returns "undefined" if there is no previous document to given one This method skips all hidden documents. If you need to get previous document, including hidden ones use:
const previousDocumentId = testDb.previousOfAll(documentId);
Example: Iterate through all documents
for (let id = testDb.last(); id != undefined; id = testDb.previous(id)) {
console.log(testDb.get(id));
}
Note that "documentId" parameter is examined only briefly, so providing appropriate value is on your responsibility. Providing inappropriate value might lead to strange behaviour.
testDb.roll();
Writes new documents from begining of database file (with a guarantee of assigning non existing yet new document id's for new documents). It replaces oldest added documents in database.
testDb.hide(documentId);
Note that "documentId" parameter is examined only briefly, so providing appropriate value is on your responsibility. Providing inappropriate value might lead to strange behaviour or database corruption.
testDb.unhide(documentId);
Note that "documentId" parameter is examined only briefly, so providing appropriate value is on your responsibility. Providing inappropriate value might lead to strange behaviour or database corruption.
testDb.start();
//code that needs to be run exclusively on database so other processes have to wait
testDb.stop();
Method "start" informs that transaction has started, and method "stop" informs that transaction has ended.
Note that simple adding new documents to database and reading them is process safe, so there is no need to use transactions. Same if your node application is single process accessing particular database.
const testDb1 = UltraDb('./testdb1.udb');
const testDb2 = UltraDb('./testdb2.udb');
const testDb3 = UltraDb('./testdb3.udb');
const multiTransaction123 = UltraDB.prepareMultiTransaction(testDb1, testDb2, testDb3);
// ...
UltraDB.start(multiTransaction123);
//code that needs to be run exclusively on given set of databases
UltraDB.stop(multiTransaction123);
Using this methods prevents cross transaction lock while transaction on multiple databases.
(methods with skipped description simply have same functionality as previously described, but on different size of data)
docAddHidden(documentSize)
creates empty hidden document and returns documentInfo reference
creates empty document and returns documentInfo reference
returns documentInfo reference for given documentId
docGetHiddenAndVisible(documentId)
returns documentInfo reference for given documentId for both, visible and hidden documents
returns id of document based on given documentInfo reference
creates key which is a Buffer type containing instructions for encrypting/decrypting documentID, bits defines expected maximum size of documentID (6-64 bits), rounds defines ciphering complexity - the higher number the harder to crack (0 gives the weakest ciphering), randomData - at least 64 bytes of random buffer data has to be provided in order to be able to generate key - the more rounds the more data should be provided (approximatly each round should have additional 24 bytes of random data)
encrypts given documentId with given key, returns Number which is a ciphered representation of documentId
decrypts given encryptedDocumentID with given key, returns Number which is decrypted documentID
returns true if database is empty
rolls database, new documents will replace oldest documents in database
allows to set top level document if someone need a document with some kind of settings, or keeping any kind of structures where entry document is required
retrieves id of current root document
validates 32 bit checksum of document ID, provided indirectly by documentInfo, at given position
sets 32 bit checksum for document provided by documentInfo reference, checksum is set at given position, but it is recommended for checksum not to be last element in the document.
adds new document with given buffer, returns id of created document, if length is -1 adds entire buffer, otherwise adds begining number of bytes from buffer and if length is bigger than buffer length, rest of space is left empty
adds new document with given buffer, document is fixed length, remaining space after buffer is empty, writes only as many bytes as provided in "length" property (if -1 writes all of them, if bigger then length of buffer, then writes as much as available and rest remains empty), returns id of created document
returns to buffer "length" number of bytes of document with given id, if length is -1 then reads all of document content
sets/replaces buffer of document with given id, length points how many bytes to write (if -1 writes all of them, if bigger then length of buffer, then writes as much as available and rest remains unchanged)
creates document with buffer and leaves begining of document empty, returns newly created document id, length points how many bytes to write (if -1 writes all of them, if bigger then length of buffer, then writes as much as available and rest remains empty)
creates document with buffer placed in the middle of document, document is fixed size, returns newly created document id, length points how many bytes to write (if -1 writes all of them, if bigger then length of buffer, then writes as much as available and rest remains empty)
returns buffer of document with given id at given position, if length is -1 then reads all content from given offset position
sets/replaces buffer placed in the middle of document, length points how many bytes to write (if -1 writes all of them, if bigger then length of buffer, then writes as much as available and rest remains unchanged)
returns buffer located at given position in document based on given documentInfo (see "Document API"), if length is -1 then reads all content from given offset position
sets/replaces buffer located at given position in document based on given documentInfo, length points how many bytes to write (if -1 writes all of them, if bigger then length of buffer, then writes as much as available and rest remains unchanged)
adds new document with given text, returns id of created document
adds new document with given text, checks if it fits given maximum length fot text in utf8 format, document is fixed length, remaining space after text is empty. returns id of created document
returns text of document with given id
sets/replaces text of document with given id
creates document with text and leaves begining of document empty, returns newly created document id
creates document with text placed in the middle of document, document is fixed size, returns newly created document id
returns text of document with given id at given position
sets/replaces text placed in the middle of document
returns text located at given position in document based on given documentInfo (see "Document API")
sets/replaces text located at given position in document based on given documentInfo (see "Document API"), validates if it fits maximum length
adds new document with given number, returns id of created document
adds new document with given number, document is fixed length, remaining space after number is empty. returns id of created document
returns number of document with given id
sets/replaces number of document with given id
creates document with number and leaves begining of document empty, returns newly created document id
creates document with number placed in the middle of document, document is fixed size, returns newly created document id
returns number of document with given id at given position
sets/replaces number placed in the middle of document
returns number located at given position in document based on given documentInfo (see "Document API")
sets/replaces number located at given position in document based on given documentInfo (see "Document API")
pkg install python2
pkg install make
pkg install clang
Currently not working on Android due to Androids lack of support for sys/shm. :(