Batch and bulk processing

db/batch and db/bcp showcase batch primitives.

db/bcp is faster, but not 'order of magnitude faster';
  nor operates with the transaction log disabled or - acutely - curtailed.
So it's just faster batch processing.
  That's what's possible with - mariadb - innodb.

cf great basis for bcp primitives:

1. Db2 not logged initially
     lets you disable the transaction log while keeping all other - SQL - facilities
2. Sybase bcp cli (esp fast bcp)

Ideally, we'd have both, in the same db.
And maybe even:
3. the possibility to - alter database files directly

I pass batch params to procs through Memory tables.

For variadic fields (eg varchar(n)) that means an n * k allocation (eg 4n for utf8mb4).
In some cases, I try to amortize this inefficiency, limitation by using myisam 'Ramdisk' tables.

• I also thought about building a BTree on a bigger proc param (eg longtext); and have udfs manipulate it; but besides being very involved, also has it's own inneficiencies, eg.

chunk and async primary purpose is supporting batch processing.

db

db/conn is my driver wrapper with logging, retry, reconnect and other shenanigans.

db/exc highlight is DmlSemSqlCode (cf).

db/proc is the pl side of stored procs.

hash

I like truncating md5 to a 64 bit unsigned int.
  I want uniqueness and 64 bits is often enough.

• blake3 is amazing and even better,
    but I'm yet to write the udf and adapt my systems to it.

As in 1 and 2.

bp

boilerplate

I write my mains as:

import { main, BB } from "../bp.mjs"

const ProgName = progName(import.meta.url)

main({ ProgName, db: { BB } }, async ({ bbc, L, log }) => {
  // ...
})

bp gives me the requested database connections and loggers; and handles a lot of repetitive stuff, overall.
And say I want a larger sort_buffer_size, then - instead of importing BB:

const BB = {
  initCmd: "set sort_buffer_size=10*1024*1024"
}

The 'elusive' withL basically prepends strings to the loggers corresponding to the returned objects:

const [bbc_, log_, L_] = withL(bbc, ["prepended"], L)

It's a way to give logs some context, eg for Promise.all.

ras

Resilience and serviceability

I cherish cockatiel and, by extension, Polly.

db

There's a world of difference between programs that retry database operations and those that don't.

got

I might extend sindresorhus/got like this.