Initializing web worker with sqlite database

[thachnb85]

Hello,

I'm trying to initialize the sqlite database with web worker connection. Following the code here: https://github.com/simolus3/moor/tree/develop/extras/flutter_web_worker_example

My current implementation without the web worker is to use InMemoryWebStorage, when I construct the database I can pass the whole downloaded db into:

Future<Database> constructRemoteDb({bool logStatements = false}) async {
  String url = "http://127.0.0.1:4567/db/testdb";
  http.Client client = new http.Client();
  var req = await client.get(Uri.parse(url));
  var dataBytes = req.bodyBytes;
  return Database(WebDatabase.withStorage(InMemoryWebStorage(inputData: dataBytes), logStatements: logStatements));
}

Is this possible to pass the dataBytes when initializing? or any better way to send the dataBytes to the worker to init the database?

  final database = MyDatabase(Platform.createDatabaseConnection('sample'));

Thank you in advance!

[simolus3]

Hi!

So what I'd probably do here is

1. Open the underlying storage implementation
2. Check if the database exists already, and
3. Write the asset if it doesn't

In code, this could be something like

final storage = DriftWebStorage.indexedDbIfSupported('my_database');
await storage.open();
final existingData = await storage.restore();
if (existingData == null) {
  final dataBytes = await downloadDatabaseAsInYourExample();
  await storage.store(dataBytes);
}
await storage.close();

// And then, finally...
return WebDatabase.withStorage(DriftWebStorage.indexedDbIfSupported('my_database'));

You can wrap all of this in a LazyDatabase so that it gets executed once when a database is needed. You don't need a web worker for this (the main advantages of a shared worker are stream-query synchronization across tabs and doing less work on the rendering thread, if you don't need them you don't need to use web workers for drift). But you can put this in a web worker, if needed (just pass inWebWorker: true to indexedDbIfSupported if you want to run that code in a worker.

[simolus3]

You could do

QueryExecutor createDatabaseConnection(String databaseName) {
  return LazyDatabase(() async {
  final storage = DriftWebStorage.indexedDbIfSupported('my_database');
  await storage.open();
  final existingData = await storage.restore();
  if (existingData == null) {
    final dataBytes = await downloadDatabaseAsInYourExample();
    await storage.store(dataBytes);
  } 

    return _connectToWorker(databaseName).executor;
  });
}

But I would probably put this logic in worker.dart instead.

[thachnb85]

Thank you, this works great!

[thachnb85]

@simolus3 I also found that this method doesn't work on my Chrome, browser just stopped there and do nothing

      final existingData = await storage.restore();

Is there any way to debug?

Thank you,

[simolus3]

Do you get any exception in the console or is it just nothing?

[thachnb85]

Hi @simolus3

I tried to move the downloading file task to worker, but have some issues, I tested different ways and checked that the indexdb is created and loaded with downloaded file content properly, but the database seems cannot query from it.

If I download file from the main app, and save to indexDB, then the app works properly, for example this code run on main work properly:

    await downloadDB();
    db = Database(Platform.createDatabaseConnection('testdb'));

The downloadDB() method:

Future<void> downloadDB() async {
    final storage = DriftWebStorage.indexedDb('testdb', migrateFromLocalStorage: false, inWebWorker: true);
    await storage.open();
    String url = "http://127.0.0.1:4567/db/testdb";
    http.Client client = http.Client();
    var req = await client.get(Uri.parse(url));
    var bytes = req.bodyBytes;
    await storage.store(bytes);
  }

Is there any possibility that downloading file takes time at worker side so it's not fast enough to "serve" when the main app call _connectToWorker?

Here is the worker.dart code which is not working

// ignore: avoid_web_libraries_in_flutter
import 'dart:html';

import 'package:drift/drift.dart';
import 'package:drift/web.dart';
import 'package:drift/remote.dart';
import 'package:http/http.dart' as http;

void main() async {
  final self = SharedWorkerGlobalScope.instance;
  self.importScripts('sql-wasm.js');

  final storage = DriftWebStorage.indexedDb('testdb', migrateFromLocalStorage: false, inWebWorker: true);
  await storage.open();
  String url = "http://127.0.0.1:4567/db/testdb";
  http.Client client = http.Client();
  var req = await client.get(Uri.parse(url));

  var bytes = req.bodyBytes;
  await storage.store(bytes);

  final db = WebDatabase.withStorage(storage);
  final server = DriftServer(DatabaseConnection.fromExecutor(db));
  self.onConnect.listen((event) {
    final msg = event as MessageEvent;
    server.serve(msg.ports.first.channel());
  });
}

Thank you!