Skip to content

Commit

Permalink
[Chore] Monorepo and Drift (#50)
Browse files Browse the repository at this point in the history
* move sqlite_async to packages

* monorepo updates

* move drift package

* fix actions

* more test fixes

* Clean up pubspec and README

* Clean up readme

* Add detailed usage instructions

* Remove older packages links

* Fix drift repository url

* Fix pana analyze packages

* Revert pana script

* Fix melos pana analyze

---------

Co-authored-by: Mugi Khan <[email protected]>
  • Loading branch information
stevensJourney and mugikhan authored Jul 8, 2024
1 parent ea80324 commit 67acc56
Show file tree
Hide file tree
Showing 106 changed files with 1,877 additions and 172 deletions.
8 changes: 4 additions & 4 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,11 @@ jobs:

- uses: dart-lang/setup-dart@v1

- name: Install dependencies
run: dart pub get
- name: Install Melos
run: dart pub global activate melos

- name: Compile WebWorker
run: dart compile js -o assets/db_worker.js -O0 lib/src/web/worker/worker.dart
- name: Install dependencies
run: melos prepare

- name: Set tag name
id: tag
Expand Down
25 changes: 13 additions & 12 deletions .github/workflows/test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,20 @@ jobs:
- uses: actions/checkout@v3
- uses: dart-lang/setup-dart@v1

- name: Install Melos
run: dart pub global activate melos
- name: Install dependencies
run: dart pub get
run: melos prepare
- name: Check formatting
run: dart format --output=none --set-exit-if-changed .
run: melos format:check:packages
- name: Lint
run: dart analyze
run: melos analyze:packages
- name: Publish dry-run
run: dart pub publish --dry-run
run: melos publish --dry-run --yes
- name: Check publish score
run: |
dart pub global activate pana
dart pub global run pana --no-warning --exit-code-threshold 0
melos analyze:packages:pana
test:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -51,18 +53,17 @@ jobs:
with:
sdk: ${{ matrix.dart_sdk }}

- name: Install Melos
run: dart pub global activate melos

- name: Install dependencies
run: dart pub get
run: melos prepare

- name: Install SQLite
run: |
./scripts/install_sqlite.sh ${{ matrix.sqlite_version }} ${{ matrix.sqlite_url }}
mkdir -p assets && curl -LJ https://github.com/simolus3/sqlite3.dart/releases/download/sqlite3-2.4.3/sqlite3.wasm -o assets/sqlite3.wasm
- name: Compile WebWorker
run: dart compile js -o assets/db_worker.js -O0 lib/src/web/worker/worker.dart
- name: Run Tests
run: |
export LD_LIBRARY_PATH=./sqlite-autoconf-${{ matrix.sqlite_version }}/.libs
dart test -p vm,chrome
export LD_LIBRARY_PATH=$(pwd)/sqlite-autoconf-${{ matrix.sqlite_version }}/.libs
melos test
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ assets
test-db
sqlite-autoconf-*
doc
*.iml

build
8 changes: 0 additions & 8 deletions DEVELOPING.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,5 @@
# Developing Instructions

## Testing

Running tests for the `web` platform requires some preparation to be executed. The `sqlite3.wasm` and `db_worker.js` files need to be available in the Git ignored `./assets` folder.

See the [test action](./.github/workflows/test.yaml) for the latest steps.

On your local machine run the commands from the `Install SQLite`, `Compile WebWorker` and `Run Tests` steps.

## Releases

Web worker files are compiled and uploaded to draft Github releases whenever tags matching `v*` are pushed. These tags are created when versioning. Releases should be manually finalized and published when releasing new package versions.
99 changes: 7 additions & 92 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,100 +2,15 @@

High-performance asynchronous interface for SQLite on Dart & Flutter.

[SQLite](https://www.sqlite.org/) is small, fast, has a lot of built-in functionality, and works
great as an in-app database. However, SQLite is designed for many different use cases, and requires
some configuration for optimal performance as an in-app database.

The [sqlite3](https://pub.dev/packages/sqlite3) Dart bindings are great for direct synchronous access
to a SQLite database, but leaves the configuration up to the developer.

This library wraps the bindings and configures the database with a good set of defaults, with
all database calls being asynchronous to avoid blocking the UI, while still providing direct SQL
query access.

## Features

- All operations are asynchronous by default - does not block the main isolate.
- Watch a query to automatically re-run on changes to the underlying data.
- Concurrent transactions supported by default - one write transaction and many multiple read transactions.
- Uses WAL mode for fast writes and running read transactions concurrently with a write transaction.
- Direct synchronous access in an isolate is supported for performance-sensitive use cases.
- Automatically convert query args to JSON where applicable, making JSON1 operations simple.
- Direct SQL queries - no wrapper classes or code generation required.

See this [blog post](https://www.powersync.co/blog/sqlite-optimizations-for-ultra-high-performance),
explaining why these features are important for using SQLite.

## Installation

```sh
dart pub add sqlite_async
```

For flutter applications, additionally add `sqlite3_flutter_libs` to include the native SQLite
library.

For other platforms, see the [sqlite3 package docs](https://pub.dev/packages/sqlite3#supported-platforms).

Web is currently not supported.
| package | build | pub | likes | popularity | pub points |
|----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------| ------- | ------- |
| sqlite_async | [![build](https://github.com/powersync-ja/sqlite_async.dart/actions/workflows/test.yaml/badge.svg?branch=main)](https://github.com/powersync-ja/sqlite_async.dart/actions?query=workflow%3Atest) | [![pub package](https://img.shields.io/pub/v/sqlite_async.svg)](https://pub.dev/packages/sqlite_async) | [![likes](https://img.shields.io/pub/likes/powersync?logo=dart)](https://pub.dev/packages/sqlite_async/score) | [![popularity](https://img.shields.io/pub/popularity/sqlite_async?logo=dart)](https://pub.dev/packages/sqlite_async/score) | [![pub points](https://img.shields.io/pub/points/sqlite_async?logo=dart)](https://pub.dev/packages/sqlite_async/score)
| drift_sqlite_async | [![build](https://github.com/powersync-ja/sqlite_async.dart/actions/workflows/test.yaml/badge.svg?branch=main)](https://github.com/powersync-ja/sqlite_async/actions?query=workflow%3Atest) | [![pub package](https://img.shields.io/pub/v/drift_sqlite_async.svg)](https://pub.dev/packages/drift_sqlite_async) | [![likes](https://img.shields.io/pub/likes/drift_sqlite_async?logo=dart)](https://pub.dev/packages/drift_sqlite_async/score) | [![popularity](https://img.shields.io/pub/popularity/drift_sqlite_async?logo=dart)](https://pub.dev/packages/drift_sqlite_async/score) | [![pub points](https://img.shields.io/pub/points/drift_sqlite_async?logo=dart)](https://pub.dev/packages/drift_sqlite_async/score)

## Getting Started

```dart
import 'package:sqlite_async/sqlite_async.dart';
final migrations = SqliteMigrations()
..add(SqliteMigration(1, (tx) async {
await tx.execute(
'CREATE TABLE test_data(id INTEGER PRIMARY KEY AUTOINCREMENT, data TEXT)');
}));
void main() async {
final db = SqliteDatabase(path: 'test.db');
await migrations.migrate(db);
// Use execute() or executeBatch() for INSERT/UPDATE/DELETE statements
await db.executeBatch('INSERT INTO test_data(data) values(?)', [
['Test1'],
['Test2']
]);
// Use getAll(), get() or getOptional() for SELECT statements
var results = await db.getAll('SELECT * FROM test_data');
print('Results: $results');
// Combine multiple statements into a single write transaction for:
// 1. Atomic persistence (all updates are either applied or rolled back).
// 2. Improved throughput.
await db.writeTransaction((tx) async {
await tx.execute('INSERT INTO test_data(data) values(?)', ['Test3']);
await tx.execute('INSERT INTO test_data(data) values(?)', ['Test4']);
});
await db.close();
}
```

# Web

Note: Web support is currently in Beta.

Web support requires Sqlite3 WASM and web worker Javascript files to be accessible via configurable URIs.

Default URIs are shown in the example below. URIs only need to be specified if they differ from default values.

The compiled web worker files can be found in our Github [releases](https://github.com/powersync-ja/sqlite_async.dart/releases)
The `sqlite3.wasm` asset can be found [here](https://github.com/simolus3/sqlite3.dart/releases)

Setup

```Dart
import 'package:sqlite_async/sqlite_async.dart';
This monorepo uses [melos](https://melos.invertase.dev/) to handle command and package management.

final db = SqliteDatabase(
path: 'test.db',
options: SqliteOptions(
webSqliteOptions: WebSqliteOptions(
wasmUri: 'sqlite3.wasm', workerUri: 'db_worker.js')));
To configure the monorepo for development run `melos prepare` after cloning.

```
For detailed usage, check out the inner [sqlite_async](https://github.com/powersync-ja/sqlite_async.dart/tree/main/packages/sqlite_async) and [drift_sqlite_async](https://github.com/powersync-ja/sqlite_async.dart/tree/main/packages/drift_sqlite_async) packages.
46 changes: 46 additions & 0 deletions melos.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
name: sqlite_async_monorepo

packages:
- packages/**

scripts:
prepare: melos bootstrap && melos prepare:compile:webworker && melos prepare:sqlite:wasm

prepare:compile:webworker:
description: Compile Javascript web worker distributable
run: dart compile js -o assets/db_worker.js -O0 packages/sqlite_async/lib/src/web/worker/worker.dart

prepare:sqlite:wasm:
description: Download SQLite3 WASM binary
run: dart run ./scripts/sqlite3_wasm_download.dart

format:
description: Format Dart code.
run: dart format .

format:check:packages:
description: Check formatting of Dart code in packages.
run: dart format --output none --set-exit-if-changed packages

analyze:packages:
description: Analyze Dart code in packages.
run: dart analyze packages --fatal-infos

analyze:packages:pana:
description: Analyze Dart packages with Pana
exec: dart pub global run pana --no-warning --exit-code-threshold 0
packageFilters:
noPrivate: true

test:
description: Run tests in a specific package.
run: dart test -p chrome,vm
exec:
concurrency: 1
packageFilters:
dirExists:
- test
# This tells Melos tests to ignore env variables passed to tests from `melos run test`
# as they could change the behaviour of how tests filter packages.
env:
MELOS_TEST: true
4 changes: 4 additions & 0 deletions packages/drift_sqlite_async/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@

## 0.1.0-alpha.1

Initial release.
21 changes: 21 additions & 0 deletions packages/drift_sqlite_async/LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2024 Journey Mobile, Inc.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
62 changes: 62 additions & 0 deletions packages/drift_sqlite_async/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# drift_sqlite_async

`drift_sqlite_async` allows using drift on an sqlite_async database - the APIs from both can be seamlessly used together in the same application.

Supported functionality:
1. All queries including select, insert, update, delete.
2. Transactions and nested transactions.
3. Table updates are propagated between sqlite_async and Drift - watching queries works using either API.
4. Select queries can run concurrently with writes and other select statements.


## Usage

Use `SqliteAsyncDriftConnection` to create a DatabaseConnection / QueryExecutor for Drift from the sqlite_async SqliteDatabase:

```dart
@DriftDatabase(tables: [TodoItems])
class AppDatabase extends _$AppDatabase {
AppDatabase(SqliteConnection db) : super(SqliteAsyncDriftConnection(db));
@override
int get schemaVersion => 1;
}
Future<void> main() async {
// The sqlite_async db
final db = SqliteDatabase(path: 'example.db');
// The Drift db
final appdb = AppDatabase(db);
}
```

A full example is in the `examples/` folder.

For details on table definitions and using the database, see the [Drift documentation](https://drift.simonbinder.eu/).

## Transactions and concurrency

sqlite_async uses WAL mode and multiple read connections by default, and this
is also exposed when using the database with Drift.

Drift's transactions use sqlite_async's `writeTransaction`. The same locks are used
for both, preventing conflicts.

Read-only transactions are not currently supported in Drift.

Drift's nested transactions are supported, implemented using SAVEPOINT.

Select statements in Drift use read operations (`getAll()`) in sqlite_async,
and can run concurrently with writes.

## Update notifications

sqlite_async uses SQLite's update_hook to detect changes for watching queries,
and will automatically pick up changes made using Drift. This also includes any updates from custom queries in Drift.

Changes from sqlite_async are automatically propagated to Drift when using SqliteAsyncDriftConnection.
These events are only sent while no write transaction is active.

Within Drift's transactions, Drift's own update notifications will still apply for watching queries within that transaction.

Note: There is a possibility of events being duplicated. This should not have a significant impact on most applications.
6 changes: 6 additions & 0 deletions packages/drift_sqlite_async/build.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
targets:
$default:
builders:
drift_dev:
options:
fatal_warnings: true
49 changes: 49 additions & 0 deletions packages/drift_sqlite_async/example/main.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
import 'package:drift/drift.dart';
import 'package:drift_sqlite_async/drift_sqlite_async.dart';
import 'package:sqlite_async/sqlite_async.dart';

part 'main.g.dart';

class TodoItems extends Table {
@override
String get tableName => 'todos';

IntColumn get id => integer().autoIncrement()();
TextColumn get description => text()();
}

@DriftDatabase(tables: [TodoItems])
class AppDatabase extends _$AppDatabase {
AppDatabase(SqliteConnection db) : super(SqliteAsyncDriftConnection(db));

@override
int get schemaVersion => 1;
}

Future<void> main() async {
final db = SqliteDatabase(path: 'example.db');

// Example where the schema is managed manually
await db.execute(
'CREATE TABLE IF NOT EXISTS todos(id integer primary key, description text)');

final appdb = AppDatabase(db);

// Watch a query on the Drift database
appdb.select(appdb.todoItems).watch().listen((todos) {
print('Todos: $todos');
});

// Insert using the Drift database
await appdb
.into(appdb.todoItems)
.insert(TodoItemsCompanion.insert(description: 'Test Drift'));

// Insert using the sqlite_async database
await db.execute('INSERT INTO todos(description) VALUES(?)', ['Test Direct']);

await Future.delayed(const Duration(milliseconds: 100));

await appdb.close();
await db.close();
}
Loading

0 comments on commit 67acc56

Please sign in to comment.