two_billion_rows.md

Two Billion Rows

An example of a regular-table data model which generates data on-the-fly to simulate a 2,000,000,000 row <table>.

HTML

You'll need a <regular-table> for this example, which you can create in HTML:

<regular-table></regular-table>

Virtual Data Model

To simulate a really big data set, this model will take advantage of the regular-table Virtual Data Model to generate data only for the window currently visible on screen. This is how really big:

const NUM_ROWS = 2000000000;
const NUM_COLUMNS = 1000;

The dataListener function for this virtual data set is simple, and returns the static dimensions directly:

export function dataListener(num_rows, num_columns) {
    return (x0, y0, x1, y1) => ({
        num_rows,
        num_columns,
        row_headers: range(y0, y1, group_header.bind(null, "Row")),
        column_headers: range(x0, x1, group_header.bind(null, "Column")),
        data: range(x0, x1, (x) => range(y0, y1, (y) => formatter.format(x + y))),
    });
}

It makes copious use of the range() function, which generates a sequence from [x0 .. x1], mapped by the function argument f().

function range(x0, x1, f) {
    return Array.from(Array(x1 - x0).keys()).map((x) => f(x + x0));
}

Generated row and column headers, as well as header groups for every group of 10, are also done on demand via group_header(), this time using the clamp() function.

function group_header(name, i) {
    const group = clamp(i, 10);
    return [`Group ${group}`, `${name} ${formatter.format(i)}`];
}

clamp() formats a number x to it's nearest y

const clamp = (x, y) => formatter.format(Math.floor(x / y) * y);

regular-table Initialization

With these, all that's left is to register the dataListener and draw the <table>.

export function init() {
    const table = document.getElementsByTagName("regular-table")[0];
    const dl = dataListener(NUM_ROWS, NUM_COLUMNS);
    table.setDataListener(dl);
    table.draw();
}

We'll initialize this table within a <script> tag, so that this ".js" output from literally of this Markdown file will not initialize, allowing dataListener() to be re-used as a data model in other examples.

<script type="module">
    import { init } from "/dist/examples/two_billion_rows.js";
    window.addEventListener("load", () => init());
</script>

Styling

We want to distinguish the cells from the headers:

td {
    color: #1078d1;
}

And some special styling to separate the row_headers from the column_headers:

tbody th:last-of-type,
thead tr:nth-child(2) th:nth-child(2),
thead tr:first-child th:first-child {
    border-right: 1px solid #ddd;
}

Appendix (Utilities)

A formatter for Numbers:

const formatter = new Intl.NumberFormat("en-us");

Appendix (Dependencies)

<script src="/dist/esm/regular-table.js"></script>
<link rel="stylesheet" href="/dist/css/material.css" />

license: apache-2.0