Experiences using Mosaic in a React application (+ Observable/Observer pattern)

[willium]

Mosaic is a very exciting step towards a paradigm for interactively linking selections, coordinating and optimizing querying, and pushing data transformations into DuckDB. I've really enjoyed playing with it so far!

As we've explored integrating Mosaic into our production environment, we've encountered some challenges that I thought might be worth sharing. Of course, I understand that the project is open source and is steered by various goals across research and industry applications.

I'm sharing these discussion points in case they're helpful for future steering. Thank you for all the hard work you've put into this project — really excited to see where it goes and hopefully contribute to its ongoing development!

To reference how we're wrapping Mosaic, I've uploaded a few of the relevant files from our codebase to a gist here.

State Management Integration:

Using Mosaic within existing state management frameworks (e.g., React, Svelte) can be quite cumbersome. For example, the singleton nature of the Coordinator is difficult to negotiate in applications where global state is not prevalent. Similarly, Mosaic Clients' reliance on internal state necessitates complex subclassing and hook injections, which can lead to really tricky states and hard-to-reconcile render lifecycles. It's not straightforward how one might delay creating a new client until the underlying table exists in DuckDB, for instance.

DOM Manipulation:

Mosaic Clients assume self-contained DOM manipulation, which doesn't align well with frameworks that handle their own DOM layout and manipulation, like React. While similar challenges occur when using D3 with hooks, it's more straightforward to use React's refs with D3 than with Mosaic.

Observable Pattern Considerations:

Mosaic's Coordinator and Client paradigm is quite similar to Observables and Observers. We're curious if there's been any consideration of adopting more standardized APIs, like those found in RxJS? Such patterns could potentially simplify state management and change propagation within Mosaic.

Core Feature Accessibility:

Mosaic's standout features, like M4 optimization and data cube indexing, are super powerful and exciting. However, they seem tightly coupled with internal aspects of vgplot and the Mosaic coordinator. We'd love to use these optimizations with other visualization libraries (e.g., Vega-Lite) and have more control over indexing strategies.

Adaptability to Evolving Data Models:

We're particularly excited about the evolution of MotherDuck's hybrid execution model. We're curious about how Mosaic, especially its data cube indexing, might adapt to such dynamic or 'unstable' data sources. Any insights into Mosaic's plans for handling these scenarios would be valuable.

TypeScript Integration:

Mosaic not being written in TypeScript presents some challenges. While the JSDoc type decorations are great, it's been an adventure chasing the API with declaration files. This is especially challenging with components like vgplot where 'unknown'/'any' typing allows for flexible API composition strategies, but at the cost of type safety. Additionally, being implemented in TypeScript would make upstream contributions much easier for us to manage. I'm curious to understand the reasoning behind this approach and any plans for fuller TypeScript integration!

[jaanli]

Incredibly helpful and we’ve had to deal with some of the same issues @onefact.

[jheer]

Thanks @willium, appreciate the thoughts here!

For state management, please note that you do not have to use the Coordinator as a singleton. While the singleton (via the coordinator() method) is provided for convenience for basic apps, but you are free to create and manage your own Coordinator. For example, vgplot supports use of an API context which can isolate a Mosaic app, including the coordinator used. In general, you can create your own Coordinator and add clients to it. Clients have their _coordinator member variable set upon registration -- clients should always use that if they need to access the coordinator; they should not call the singleton accessor. Of course, this may not address all your state management concerns, but I wanted to share this in case it wasn't clear.

For DOM manipulation, clients don't really assume much of anything, though I agree that our current examples do seem to embody that assumption. The main role of a client is to request and receive data (they don't have to render anything at all), and the rest is up to you. A client does not need to be a UI component itself; for example, it could be a sub-part of an encompassing UI component and a UI component could potentially contain multiple Mosaic clients (this is loosely how a Plot in vgplot works, with multiple "marks" as single-client sub-components). If you have thoughts on how client functionality might be better incorporated (ideally in a general, as opposed to toolkit-specific, way) please share!

For observable patterns, I'd be happy to adhere to a standard for reactive variables if/when it exists, assuming it will actually improve interoperability. I don't really want to hitch our wagon to one proprietary take vs. another, and possibly add more bloat/dependency in the process. In addition, there are some subtleties in how Mosaic Params/Selections handle listeners that return Promises -- this supports async updates (including throttling/debouncing) and would need to be properly handled in any alternative reactive model.

For core feature accessibility, I think in many cases (e.g., M4) this may largely boil down to making some internal methods available as external exports. That said, those methods may be streamlined towards the current use cases. Happy to consider PRs that do some of this exporting work. In contrast, the data cube indexer depends heavily on multiple core Mosaic abstractions -- at this point it's not clear to me how to make that "standalone".

We're also tracking MotherDuck, etc. More generally, we have some thoughts on making Mosaic more amenable to dynamic data sources (along the lines of previous issues you've raised), so please look for more in the coming weeks.

As for TypeScript, I'm supportive of porting over, especially as I am becoming more TS proficient. But it will take a lot of time, and I can't say when I will have that time for what is basically volunteer effort here. I think a first step is to rewrite thesql helpers, and then a second step is to rewrite the core package. I expect this may cause some architectural (or at least, method overloading/chaining, changes) along the way.

[willium]

Awesome. Let me know how I can be helpful!