A tool for automating the process of running and comparing Go benchmarks across code changes.
$ benchdiff --help
usage: benchdiff [--old <commit>] [--new <commit>] <pkgs>...
benchdiff automates the process of running and comparing Go microbenchmarks
across code changes.
benchdiff runs all microbenchmarks in the specified packages against the old and
new commit. It then passes the benchmark output through benchstat to compute
statistics about the results.
By default, benchdiff outputs these results in a textual format. However, if the
--sheets flag is passed then it will upload the result to a Google Sheets
spreadsheet. To access this, users must have a Google service account. For
information, see https://cloud.google.com/iam/docs/service-accounts.
The Google service account must meet the following conditions:
1. The Google Sheets API must be enabled for the account's project
2. The Google Drive API must be enabled for the account's project
When the --sheets flag is passed, benchdiff will search for a credentials file
containing the service account key using the GOOGLE_APPLICATION_CREDENTIALS
environment variable. See https://cloud.google.com/docs/authentication/production.
Options:
-n, --new <commit> measure the difference between this commit and old (default HEAD)
-o, --old <commit> measure the difference between this commit and new (default new~)
-r, --run <regexp> run only benchmarks matching regexp
-c, --count <n> run tests and benchmarks n times (default 10)
-t, --threshold <n> exit with code 0 if all regressions are below threshold, else 1
--post-checkout an optional command to run after checking out each branch to
configure the git repo so that 'go build' succeeds
--csv output the results in a csv format
--html output the results in an HTML table
--sheets output the results to a new Google Sheets document
--help display this help
Example invocations:
$ benchdiff --sheets ./pkg/...
$ benchdiff --old=master~ --new=master --threshold=0.2 ./pkg/kv ./pkg/storage/...
$ benchdiff --new=d1fbdb2 --run=Datum --count=2 --csv ./pkg/sql/...
$ benchdiff --new=6299bd4 --sheets --post-checkout='dev generate go' ./pkg/workload/...
Using text output:
$ benchdiff --new=6299bd4 ./pkg/workload/...
test binaries already exist for 'efcf66c'; skipping build
test binaries already exist for '6299bd4'; skipping build
running benchmarks:
pkg=1/7 iter=2/2 cockroachdb/cockroach/pkg/workload -
pkg=2/7 iter=2/2 cockroachdb/cockroach/pkg/workload/bank /
pkg=3/7 iter=2/2 cockroachdb/cockroach/pkg/workload/faker |
pkg=4/7 iter=2/2 cockroachdb/cockroach/pkg/workload/movr |
pkg=5/7 iter=2/2 cockroachdb/cockroach/pkg/workload/tpcc /
pkg=6/7 iter=2/2 cockroachdb/cockroach/pkg/workload/workloadsql \
pkg=7/7 iter=2/2 cockroachdb/cockroach/pkg/workload/ycsb |
name old time/op new time/op delta
InitialData/tpcc/warehouses=1-8 304ms ± 4% 195ms ± 1% -35.61% (p=0.008 n=5+5)
InitialData/bank/rows=1000-8 281µs ± 3% 282µs ± 2% ~ (p=0.548 n=5+5)
CSVRowsReader-8 17.2µs ± 1% 17.5µs ± 0% +1.80% (p=0.016 n=5+4)
WriteCSVRows-8 14.8µs ± 3% 15.6µs ± 5% +5.28% (p=0.032 n=5+5)
name old speed new speed delta
InitialData/tpcc/warehouses=1-8 363MB/s ± 4% 563MB/s ± 1% +55.23% (p=0.008 n=5+5)
CSVRowsReader-8 98.1MB/s ± 1% 101.0MB/s ± 0% +2.96% (p=0.016 n=5+4)
InitialData/bank/rows=1000-8 413MB/s ± 3% 411MB/s ± 2% ~ (p=0.548 n=5+5)
WriteCSVRows-8 113MB/s ± 3% 113MB/s ± 5% ~ (p=1.000 n=5+5)
name old alloc/op new alloc/op delta
InitialData/tpcc/warehouses=1-8 128kB ± 0% 81kB ± 0% -36.19% (p=0.008 n=5+5)
InitialData/bank/rows=1000-8 19.1kB ± 0% 19.1kB ± 0% ~ (all equal)
WriteCSVRows-8 5.70kB ± 0% 5.70kB ± 0% ~ (all equal)
CSVRowsReader-8 7.38kB ± 0% 7.38kB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
InitialData/tpcc/warehouses=1-8 587 ± 0% 583 ± 0% -0.61% (p=0.008 n=5+5)
InitialData/bank/rows=1000-8 1.02k ± 0% 1.02k ± 0% ~ (all equal)
WriteCSVRows-8 50.0 ± 0% 50.0 ± 0% ~ (all equal)
CSVRowsReader-8 55.0 ± 0% 55.0 ± 0% ~ (all equal)
Using Google Sheets output:
$ export GOOGLE_APPLICATION_CREDENTIALS=/Users/nathan/.service-account-creds.json
$ benchdiff --new=6299bd4 --sheets ./pkg/workload/...
test binaries already exist for 'efcf66c'; skipping build
test binaries already exist for '6299bd4'; skipping build
running benchmarks:
pkg=1/7 iter=2/2 cockroachdb/cockroach/pkg/workload -
pkg=2/7 iter=2/2 cockroachdb/cockroach/pkg/workload/bank /
pkg=3/7 iter=2/2 cockroachdb/cockroach/pkg/workload/faker |
pkg=4/7 iter=2/2 cockroachdb/cockroach/pkg/workload/movr |
pkg=5/7 iter=2/2 cockroachdb/cockroach/pkg/workload/tpcc /
pkg=6/7 iter=2/2 cockroachdb/cockroach/pkg/workload/workloadsql \
pkg=7/7 iter=2/2 cockroachdb/cockroach/pkg/workload/ycsb |
generated sheet: https://docs.google.com/spreadsheets/d/...