Feature request: support for pmtiles / shape files for scalable interactive maps?

[jaanli]

This project is amazing, thank you @mbostock and team!!

Just wondering how deep into geospatial folks plan to venture.

For example, I see that some geospatial features are supported: https://observablehq.com/framework/examples/eia/

Are there plans to support:

(1) millions of markers?

This is feasible with libraries like lonboard (https://github.com/developmentseed/lonboard), Mosaic (https://uwdata.github.io/mosaic/).

Example of visualizing a few million individual responsss to the 2022 US census survey: https://github.com/jaanli/lonboard/blob/example-american-community-survey/examples/american-community-survey.ipynb

gif: https://s13.gifyu.com/images/SCGH2.gif

(2) Is there support planned for Protomaps (Protomaps.com) or other open vectorized tiles? This is needed to scale to global visualizations with support for interaction.

(Tools like tippecanoe can be used to convert between parquet, geoparquet, pmtiles format for Protomaps, and dbt can be used to automate this. I see Observable Framework as an amazing ecosystem on top of this!)

Happy to collaborate on this if anyone’s interested in making a pull request. The most direct path here is to contribute an Observable Framework example that uses a Mosaic plot (can be imported via ESM), and is linked to a pmtiles/geoparquet dataset - that should scale to millions of points in the browser.

Exciting times ahead :) please let me know if this aligns or there are already plans for this!!

[Fil]

yeah, all of it :)

The magic of Observable Framework is that it is vanilla javascript: as a user you don't need to wait for us to implement anything explicitly. You can just start using it: most of the JavaScript universe can be imported direct from npm, and used with no limitation in a page.

For example, the Mosaic + vgplot "10M flights" demo works out of the box:

I haven't published it, but here is the full code for that page in my project:

# Mosaic Cross-Filter Flights 10M

An example using [Mosaic vgplot](https://uwdata.github.io/mosaic/vgplot/) to interactively cross-filter 10 million flight records.
You may need to wait a few seconds for the dataset to load.

<div style="display: flex; flex-wrap: wrap">

${makePlot("delay")}

${makePlot("time")}

${makePlot("distance")}

</div>

_Try selecting delayed flights. How much more likely are they to leave later in the day?_

```js
// load flights data from external parquet file
await vg.coordinator().exec(`CREATE TABLE IF NOT EXISTS flights10m AS
    SELECT
      GREATEST(-60, LEAST(ARR_DELAY, 180))::DOUBLE AS delay,
      DISTANCE AS distance,
      DEP_TIME AS time
    FROM 'https://uwdata.github.io/mosaic-datasets/data/flights-10m.parquet'`);

// create a selection with crossfilter resolution
const brush = vg.Selection.crossfilter();

// helper method to generate a binned plot filtered by brush
// a plot contains a rectY mark for a histogram, as well as
// an intervalX interactor to populate the brush selection
const makePlot = (column) =>
  vg.plot(
    vg.rectY(
      vg.from("flights10m", { filterBy: brush }), // data set and filter selection
      { x: vg.bin(column), y: vg.count(), fill: "steelblue", inset: 0.5 }
    ),
    vg.intervalX({ as: brush }), // create an interval selection brush
    vg.xDomain(vg.Fixed), // don't change the x-axis domain across updates
    vg.marginLeft(75),
    vg.width(470),
    vg.height(160)
  );

// generate dashboard with three linked histograms
//  display(vg.vconcat(makePlot("delay"), makePlot("time"), makePlot("distance")));
```

```js
// import vgplot and configure Mosaic to use DuckDB-WASM
const vg = await import("npm:@uwdata/vgplot");
{
  const wasm = await vg.wasmConnector();
  await vg.coordinator().databaseConnector(wasm);
}
```

I've also been able to use a recent duckdb-wasm with the SPATIAL extension—in that case, one difficulty is to align all the versions between duckdb-wasm, parquet, and arrow, because they're all moving very fast and breaking things.

[jaanli]

Woha, you're amazing @Fil !!! Thank you so much and cannot wait to try this :D hope to contribute some examples!

[bdon]

Here's an example of an Observable Framework app loading PMTiles basemaps + overlays using both Leaflet and MapLibre GL JS:

https://bdon.github.io/observable-framework-maps/example-map#interaction%3A-maplibre-gl-js

[Fil]

this is awesome!

[Fil]

Here's another example of using maplibregl with pmtiles
https://observablehq.observablehq.cloud/pangea/party/maplibre-pmtiles

(now I'm thinking I should host my own copy of the world)