Snowplow BigQuery Loader

This project contains applications used to load Snowplow enriched data into Google BigQuery.

Quickstart

Assuming git and SBT installed:

$ git clone https://github.com/snowplow-incubator/snowplow-bigquery-loader
$ cd snowplow-bigquery-loader
$ sbt "project loader" test
$ sbt "project streamloader" test
$ sbt "project mutator" test
$ sbt "project repeater" test

Benchmarks

This project comes with sbt-jmh.

To run a specific benchmark test:

$ sbt 'project benchmark' '+jmh:run -i 20 -wi 10 -f2 -t3 .*TransformAtomic.*'

Or, to run all benchmark tests (once more are added):

$ sbt 'project benchmark' '+jmh:run -i 20 -wi 10 -f2 -t3'

The number of warm-ups and iterations is what the sbt-jmh project recommends but they can be lowered for faster runs.

To see all sbt-jmh options: jmh:run -h.

Add new benchmarks to this module.

Building fatjars

You can build the jar files for Mutator, Repeater and Streamloader with sbt like so:

$ sbt clean 'project mutator' assembly
$ sbt clean 'project repeater' assembly
$ sbt clean 'project streamloader' assembly

Find out more

Technical Docs	Setup Guide	Contributing

Copyright and License

Snowplow BigQuery Loader is copyright 2018-2021 Snowplow Analytics Ltd.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.