benchmark all dbs #465

baabeetaa · 2022-01-29T07:56:16Z

Update to run benchmarks for all dbs (memdb, goleveldb, rocksdb, badgerdb) except boltdb.
Update script to run the benchmarks with docker quickly. So its ok to run on local or dev machine.

Run the command below to install leveldb and rocksdb from source then run the benchmarks

replace (optional):

baabeetaa with your repo username and
fix-bencharks with your branch.

docker run --rm -it ubuntu:16.04 /bin/bash -c \
"apt-get update && apt-get install -y curl && \
sh <(curl -s https://raw.githubusercontent.com/baabeetaa/iavl/fix-bencharks/benchmarks/setup/INSTALL_ROOT.sh) && \
sh <(curl -s https://raw.githubusercontent.com/baabeetaa/iavl/fix-bencharks/benchmarks/setup/RUN_BENCHMARKS.sh) fix-bencharks baabeetaa && \
cat ~/iavl/results.txt"

output:

➜  ~ docker run --rm -it ubuntu:16.04 /bin/bash -c \
"apt-get update && apt-get install -y curl && \
sh <(curl -s https://raw.githubusercontent.com/baabeetaa/iavl/fix-bencharks/benchmarks/setup/INSTALL_ROOT.sh) && \
sh <(curl -s https://raw.githubusercontent.com/baabeetaa/iavl/fix-bencharks/benchmarks/setup/RUN_BENCHMARKS.sh) fix-bencharks baabeetaa && \
cat ~/iavl/results.txt"
Get:1 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
...
The following additional packages will be installed:
  binutils bzip2 bzip2-doc cpp cpp-5 dpkg-dev fakeroot g++ g++-5 gcc gcc-5 git-man ifupdown iproute2 isc-dhcp-client isc-dhcp-common less
...
  libxdmcp6 libxext6 libxmuu1 libxtables11 libzstd-dev libzstd0 linux-libc-dev make manpages manpages-dev netbase openssh-client patch perl
  perl-modules-5.22 rename rsync screen wget xauth xz-utils zlib1g-dev
0 upgraded, 86 newly installed, 0 to remove and 3 not upgraded.
Need to get 55.9 MB of archives.
...
Cloning into 'leveldb'...
remote: Enumerating objects: 3430, done.
remote: Counting objects: 100% (148/148), done.
remote: Compressing objects: 100% (85/85), done.
remote: Total 3430 (delta 66), reused 109 (delta 61), pack-reused 3282
...
g++ -I. -I./include -fno-builtin-memcmp -pthread -DOS_LINUX -DLEVELDB_PLATFORM_POSIX -DSNAPPY -O2 -DNDEBUG        -c port/port_posix.cc -o port/port_posix.o
rm -f libleveldb.a
ar -rs libleveldb.a db/builder.o db/c.o db/db_impl.o db/db_iter.o db/dbformat.o db/filename.o db/log_reader.o db/log_writer.o db/memtable.o db/repair.o db/table_cache.o db/version_edit.o db/version_set.o db/write_batch.o table/block.o table/block_builder.o table/filter_block.o table/format.o table/iterator.o table/merger.o table/table.o table/table_builder.o table/two_level_iterator.o util/arena.o util/bloom.o util/cache.o util/coding.o util/comparator.o util/crc32c.o util/env.o util/env_posix.o util/filter_policy.o util/hash.o util/histogram.o util/logging.o util/options.o util/status.o port/port_posix.o
ar: creating libleveldb.a
ln -fs libleveldb.so.1.7 libleveldb.so
ln -fs libleveldb.so.1.7 libleveldb.so.1
Cloning into 'rocksdb'...
remote: Enumerating objects: 106222, done.
remote: Counting objects: 100% (433/433), done.
remote: Compressing objects: 100% (238/238), done.
remote: Total 106222 (delta 205), reused 301 (delta 139), pack-reused 105789
Receiving objects: 100% (106222/106222), 175.38 MiB | 3.70 MiB/s, done.
Resolving deltas: 100% (80605/80605), done.
Checking connectivity... done.
Note: checking out 'v6.15.5'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at abd4b1f... bump version and update HISTORY.md for 6.15.5
$DEBUG_LEVEL is 0
  GEN      util/build_version.cc
$DEBUG_LEVEL is 0
  GEN      util/build_version.cc
echo 'prefix=/usr/local' > rocksdb.pc
echo 'exec_prefix=${prefix}' >> rocksdb.pc
  CC       cache/cache.o
  CC       cache/clock_cache.o
  CC       cache/lru_cache.o
echo 'includedir=${prefix}/include' >> rocksdb.pc
echo 'libdir=/usr/local/lib' >> rocksdb.pc
echo '' >> rocksdb.pc
echo 'Name: rocksdb' >> rocksdb.pc
echo 'Description: An embeddable persistent key-value store for fast storage' >> rocksdb.pc
echo Version: 6.15.5 >> rocksdb.pc
echo 'Libs: -L${libdir}  -ldl -Wl,-rpath -Wl,'$ORIGIN' -lrocksdb' >> rocksdb.pc
echo 'Libs.private: -lpthread -lrt -ldl -lsnappy -lgflags -lz -lbz2 -llz4 -lzstd' >> rocksdb.pc
echo 'Cflags: -I${includedir} -std=c++11  -DROCKSDB_PLATFORM_POSIX -DROCKSDB_LIB_IO_POSIX  -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=1 -DZLIB -DBZIP2 -DLZ4 -DZSTD -DROCKSDB_MALLOC_USABLE_SIZE -DROCKSDB_PTHREAD_ADAPTIVE_MUTEX -DROCKSDB_BACKTRACE -DROCKSDB_RANGESYNC_PRESENT -DROCKSDB_SCHED_GETCPU_PRESENT -DROCKSDB_AUXV_GETAUXVAL_PRESENT -march=native   -DHAVE_SSE42  -DHAVE_PCLMUL  -DHAVE_AVX2  -DHAVE_BMI  -DHAVE_LZCNT -DHAVE_UINT128_EXTENSION -DROCKSDB_SUPPORT_THREAD_LOCAL  -isystem third-party/gtest-1.8.1/fused-src -isystem ./third-party/folly' >> rocksdb.pc
  CC       cache/sharded_cache.o
  CC       db/arena_wrapped_db_iter.o
...
  CC       third-party/folly/folly/synchronization/WaitOptions.o
install -d /usr/local/lib
install -d /usr/local/lib/pkgconfig
for header_dir in `find "include/rocksdb" -type d`; do \
	install -d //usr/local/$header_dir; \
done
for header in `find "include/rocksdb" -type f -name *.h`; do \
	install -C -m 644 $header //usr/local/$header; \
done
install -C -m 644 rocksdb.pc /usr/local/lib/pkgconfig/rocksdb.pc
  CCLD     librocksdb.so.6.15.5
install -d /usr/local/lib
install -C -m 755 librocksdb.so.6.15.5 /usr/local/lib
ln -fs librocksdb.so.6.15.5 /usr/local/lib/librocksdb.so.6.15
ln -fs librocksdb.so.6.15.5 /usr/local/lib/librocksdb.so.6
ln -fs librocksdb.so.6.15.5 /usr/local/lib/librocksdb.so
--2022-01-29 07:44:05--  https://go.dev/dl/go1.17.6.linux-amd64.tar.gz
Resolving go.dev (go.dev)... 216.239.34.21, 216.239.36.21, 216.239.38.21, ...
Connecting to go.dev (go.dev)|216.239.34.21|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://dl.google.com/go/go1.17.6.linux-amd64.tar.gz [following]
--2022-01-29 07:44:05--  https://dl.google.com/go/go1.17.6.linux-amd64.tar.gz
Resolving dl.google.com (dl.google.com)... 172.217.25.14, 2404:6800:4005:802::200e
Connecting to dl.google.com (dl.google.com)|172.217.25.14|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 134830580 (129M) [application/x-gzip]
Saving to: 'go1.17.6.linux-amd64.tar.gz'

go1.17.6.linux-amd64.tar.gz          100%[=====================================================================>] 128.58M  6.95MB/s    in 19s

2022-01-29 07:44:24 (6.89 MB/s) - 'go1.17.6.linux-amd64.tar.gz' saved [134830580/134830580]

Cloning into 'iavl'...
remote: Enumerating objects: 6003, done.
remote: Counting objects: 100% (434/434), done.
remote: Compressing objects: 100% (238/238), done.
remote: Total 6003 (delta 261), reused 323 (delta 195), pack-reused 5569
Receiving objects: 100% (6003/6003), 9.32 MiB | 3.32 MiB/s, done.
Resolving deltas: 100% (3896/3896), done.
Checking connectivity... done.
Branch fix-bencharks set up to track remote branch fix-bencharks from origin.
Switched to a new branch 'fix-bencharks'
go: downloading golang.org/x/crypto v0.0.0-20210921155107-089bfa567519
go: downloading github.com/stretchr/testify v1.7.0
...
go: downloading github.com/DataDog/zstd v1.4.1
cd benchmarks && \
	go test -ldflags "-X github.com/cosmos/iavl.Version=0.17.2-13-g3342d1e -X github.com/cosmos/iavl.Commit=3342d1e9c40f3cb2258f65a3192643b18a4b639d -X github.com/cosmos/iavl.Branch=fix-bencharks" -bench=RandomBytes . && \
	go test -ldflags "-X github.com/cosmos/iavl.Version=0.17.2-13-g3342d1e -X github.com/cosmos/iavl.Commit=3342d1e9c40f3cb2258f65a3192643b18a4b639d -X github.com/cosmos/iavl.Branch=fix-bencharks" -tags cleveldb,rocksdb,boltdb,badgerdb -bench=Small . && \
	go test -ldflags "-X github.com/cosmos/iavl.Version=0.17.2-13-g3342d1e -X github.com/cosmos/iavl.Commit=3342d1e9c40f3cb2258f65a3192643b18a4b639d -X github.com/cosmos/iavl.Branch=fix-bencharks" -tags cleveldb,rocksdb,boltdb,badgerdb -bench=Medium . && \
	go test -ldflags "-X github.com/cosmos/iavl.Version=0.17.2-13-g3342d1e -X github.com/cosmos/iavl.Commit=3342d1e9c40f3cb2258f65a3192643b18a4b639d -X github.com/cosmos/iavl.Branch=fix-bencharks" -bench=BenchmarkMemKeySizes .
iavl: 0.17.2-13-g3342d1e
git commit: 3342d1e9c40f3cb2258f65a3192643b18a4b639d
git branch: fix-bencharks
go version go1.17.6 linux/amd64

goos: linux
goarch: amd64
pkg: github.com/cosmos/iavl/benchmarks
cpu: Intel(R) Core(TM) i5-10400 CPU @ 2.90GHz
BenchmarkRandomBytes/random-4-12         	39355621	        29.71 ns/op
BenchmarkRandomBytes/random-16-12        	25840755	        44.98 ns/op
BenchmarkRandomBytes/random-32-12        	18686253	        60.69 ns/op
BenchmarkRandomBytes/random-100-12       	 9206001	       127.0 ns/op
BenchmarkRandomBytes/random-1000-12      	 1000000	      1011 ns/op
PASS
ok  	github.com/cosmos/iavl/benchmarks	5.962s
iavl: 0.17.2-13-g3342d1e
git commit: 3342d1e9c40f3cb2258f65a3192643b18a4b639d
git branch: fix-bencharks
go version go1.17.6 linux/amd64

Init Tree took 0.76 MB
goos: linux
goarch: amd64
pkg: github.com/cosmos/iavl/benchmarks
cpu: Intel(R) Core(TM) i5-10400 CPU @ 2.90GHz
BenchmarkSmall/memdb-1000-100-4-10/query-miss-12         	  798007	      1550 ns/op	     390 B/op	       8 allocs/op
BenchmarkSmall/memdb-1000-100-4-10/query-hits-12         	  624960	      1992 ns/op	     561 B/op	      11 allocs/op
BenchmarkSmall/memdb-1000-100-4-10/update-12             	   48502	     26555 ns/op	   11689 B/op	     174 allocs/op
BenchmarkSmall/memdb-1000-100-4-10/block-12              	     298	   6019845 ns/op	 2285795 B/op	   35064 allocs/op
Init Tree took 0.47 MB
BenchmarkSmall/goleveldb-1000-100-4-10/query-miss-12     	  607468	      2077 ns/op	     544 B/op	      12 allocs/op
BenchmarkSmall/goleveldb-1000-100-4-10/query-hits-12     	  470210	      2595 ns/op	     769 B/op	      17 allocs/op
BenchmarkSmall/goleveldb-1000-100-4-10/update-12         	   34101	     43059 ns/op	   21065 B/op	     177 allocs/op
BenchmarkSmall/goleveldb-1000-100-4-10/block-12          	     205	   8917088 ns/op	 3991160 B/op	   40946 allocs/op
Init Tree took 0.38 MB
BenchmarkSmall/cleveldb-1000-100-4-10/query-miss-12      	  531919	      2266 ns/op	     452 B/op	      13 allocs/op
BenchmarkSmall/cleveldb-1000-100-4-10/query-hits-12      	  426804	      2907 ns/op	     648 B/op	      18 allocs/op
BenchmarkSmall/cleveldb-1000-100-4-10/update-12          	   24032	     47533 ns/op	   10917 B/op	     318 allocs/op
BenchmarkSmall/cleveldb-1000-100-4-10/block-12           	     169	  14094519 ns/op	 1959739 B/op	   59489 allocs/op
Init Tree took 0.41 MB
BenchmarkSmall/boltdb-1000-100-4-10/query-miss-12        	  576866	      2126 ns/op	     886 B/op	      16 allocs/op
BenchmarkSmall/boltdb-1000-100-4-10/query-hits-12        	  438992	      2833 ns/op	    1280 B/op	      22 allocs/op
BenchmarkSmall/boltdb-1000-100-4-10/update-12            	    5035	    392903 ns/op	   33860 B/op	     237 allocs/op
BenchmarkSmall/boltdb-1000-100-4-10/block-12             	      24	  53887873 ns/op	 4413098 B/op	   31188 allocs/op
Init Tree took 0.37 MB
BenchmarkSmall/rocksdb-1000-100-4-10/query-miss-12       	  472177	      2556 ns/op	     478 B/op	      14 allocs/op
BenchmarkSmall/rocksdb-1000-100-4-10/query-hits-12       	  345380	      3359 ns/op	     661 B/op	      18 allocs/op
BenchmarkSmall/rocksdb-1000-100-4-10/update-12           	   25657	     60410 ns/op	   11596 B/op	     346 allocs/op
BenchmarkSmall/rocksdb-1000-100-4-10/block-12            	     124	  12247287 ns/op	 2029298 B/op	   62391 allocs/op
Init Tree took 0.41 MB
BenchmarkSmall/badgerdb-1000-100-4-10/query-miss-12      	  501952	      2493 ns/op	     948 B/op	      16 allocs/op
BenchmarkSmall/badgerdb-1000-100-4-10/query-hits-12      	  369922	      3242 ns/op	    1366 B/op	      22 allocs/op
BenchmarkSmall/badgerdb-1000-100-4-10/update-12          	   23092	     73531 ns/op	   26973 B/op	     453 allocs/op
BenchmarkSmall/badgerdb-1000-100-4-10/block-12           	     100	  14903450 ns/op	 6487498 B/op	   67268 allocs/op
PASS
ok  	github.com/cosmos/iavl/benchmarks	65.417s
iavl: 0.17.2-13-g3342d1e
git commit: 3342d1e9c40f3cb2258f65a3192643b18a4b639d
git branch: fix-bencharks
go version go1.17.6 linux/amd64

Init Tree took 78.75 MB
goos: linux
goarch: amd64
pkg: github.com/cosmos/iavl/benchmarks
cpu: Intel(R) Core(TM) i5-10400 CPU @ 2.90GHz
BenchmarkMedium/memdb-100000-100-16-40/query-miss-12         	  166834	      6852 ns/op	     440 B/op	       8 allocs/op
BenchmarkMedium/memdb-100000-100-16-40/query-hits-12         	  149720	      8513 ns/op	     604 B/op	      11 allocs/op
BenchmarkMedium/memdb-100000-100-16-40/update-12             	   12628	    101200 ns/op	   27997 B/op	     340 allocs/op
BenchmarkMedium/memdb-100000-100-16-40/block-12              	     100	  10827332 ns/op	 2912395 B/op	   38498 allocs/op
Init Tree took 46.69 MB
BenchmarkMedium/goleveldb-100000-100-16-40/query-miss-12     	   83956	     12610 ns/op	    1342 B/op	      23 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-12     	   71010	     16115 ns/op	    1886 B/op	      32 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/update-12         	   10000	    125394 ns/op	   39754 B/op	     382 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/block-12          	      96	  15526349 ns/op	 4625226 B/op	   45819 allocs/op
Init Tree took 38.44 MB
BenchmarkMedium/cleveldb-100000-100-16-40/query-miss-12      	  133000	      8770 ns/op	     524 B/op	      13 allocs/op
BenchmarkMedium/cleveldb-100000-100-16-40/query-hits-12      	  102427	     11033 ns/op	     728 B/op	      18 allocs/op
BenchmarkMedium/cleveldb-100000-100-16-40/update-12          	   10000	    188645 ns/op	   21875 B/op	     621 allocs/op
BenchmarkMedium/cleveldb-100000-100-16-40/block-12           	      91	  15797866 ns/op	 2409674 B/op	   70357 allocs/op
Init Tree took 38.60 MB
BenchmarkMedium/rocksdb-100000-100-16-40/query-miss-12       	  126205	      8096 ns/op	     529 B/op	      13 allocs/op
BenchmarkMedium/rocksdb-100000-100-16-40/query-hits-12       	  112999	     10248 ns/op	     739 B/op	      18 allocs/op
BenchmarkMedium/rocksdb-100000-100-16-40/update-12           	   10000	    132750 ns/op	   23359 B/op	     681 allocs/op
BenchmarkMedium/rocksdb-100000-100-16-40/block-12            	      79	  16220789 ns/op	 2569183 B/op	   76915 allocs/op
Init Tree took 32.16 MB
BenchmarkMedium/badgerdb-100000-100-16-40/query-miss-12      	  148632	      7944 ns/op	    1087 B/op	      17 allocs/op
BenchmarkMedium/badgerdb-100000-100-16-40/query-hits-12      	  128002	      9703 ns/op	    1538 B/op	      23 allocs/op
BenchmarkMedium/badgerdb-100000-100-16-40/update-12          	   10000	    205566 ns/op	   76454 B/op	     826 allocs/op
BenchmarkMedium/badgerdb-100000-100-16-40/block-12           	      82	  17980372 ns/op	 5774758 B/op	   94061 allocs/op
PASS
ok  	github.com/cosmos/iavl/benchmarks	49.823s
PASS
ok  	github.com/cosmos/iavl/benchmarks	0.010s
➜  ~

…lder first - run benchmark for all dbs in BenchmarkSmall, BenchmarkMedium and BenchmarkLarge. Note: boltdb is still disabled as its too slow - dont know why

…) and Makefile to test all dbs

tac0turtle · 2022-01-29T10:29:50Z

I don't think this is really necessary. On chain people are claiming rocks is faster but these benchmarks are proving to be slower.

Happy to merge if Robert thinks its worth it.

robert-zaremba

Thanks. I'm OK to merge it - it's good to have more data.

robert-zaremba · 2022-01-29T11:19:26Z

benchmarks/setup/INSTALL_ROOT.sh

+cd ~/
+mkdir go
+wget https://go.dev/dl/go1.17.6.linux-amd64.tar.gz
+tar -C /usr/local -xzf go1.17.6.linux-amd64.tar.gz


why not using a repository?

Because Ubuntu ships decrepit versions of Go :)

robert-zaremba · 2022-01-29T11:46:44Z

but these benchmarks are proving to be slower.

The benchmark shows that rocksdb is the fastest or on the same level and has the least allocations (not counting memdb).

faddat · 2022-01-29T15:06:15Z

.....and this is why I'm not a fan of synthetic benchmarks, they can lie.

But I'm agreed that having more data would be useful.

I'm going to make a new Dockerfile that has all of the databases ready to rock in it, and make sure that this can work in github actions.

tac0turtle · 2022-01-30T10:43:09Z

.....and this is why I'm not a fan of synthetic benchmarks, they can lie.

But I'm agreed that having more data would be useful.

I'm going to make a new Dockerfile that has all of the databases ready to rock in it, and make sure that this can work in github actions.

oh we have this already actually. we use it in tm-db and cosmos-sdk. It may just need updating.

sorry I read the benchmarks inverted, yes it is way faster. I need to start reading on how to optimise rocks.

faddat · 2022-01-31T17:24:52Z

We're working on the rocks optomizations, too, here's some info on how:

osmosis-labs/osmosis#695

(and specifically, the link at the end, is what tuan and I are working towards being able to use)

https://rocksdb.org/blog/2018/08/01/rocksdb-tuning-advisor.html

tac0turtle · 2022-02-01T09:49:42Z

lets fix linting then merge this.

baabeetaa added 4 commits January 28, 2022 03:15

- fix runBenchmarks for rocksdb, cleveldb and badgerdb by creating fo…

1abc738

…lder first - run benchmark for all dbs in BenchmarkSmall, BenchmarkMedium and BenchmarkLarge. Note: boltdb is still disabled as its too slow - dont know why

update benchmark setup scripts (INSTALL_ROOT.sh and RUN_BENCHMARKS.sh…

6316b01

…) and Makefile to test all dbs

update doc with command to run the benchmarks quickly using docker

6f887ee

update doc...

3342d1e

baabeetaa requested a review from robert-zaremba as a code owner January 29, 2022 07:56

robert-zaremba approved these changes Jan 29, 2022

View reviewed changes

fix lint

481fe0f

tac0turtle merged commit 200e44f into cosmos:master Feb 4, 2022

p0mvn mentioned this pull request Feb 16, 2022

Fix cleveldb, boltdb, rocksdb and badgerdb tests in CI #469

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

benchmark all dbs #465

benchmark all dbs #465

baabeetaa commented Jan 29, 2022

tac0turtle commented Jan 29, 2022

robert-zaremba left a comment

robert-zaremba Jan 29, 2022

faddat Jan 29, 2022

robert-zaremba commented Jan 29, 2022

faddat commented Jan 29, 2022

tac0turtle commented Jan 30, 2022

faddat commented Jan 31, 2022

tac0turtle commented Feb 1, 2022

benchmark all dbs #465

benchmark all dbs #465

Conversation

baabeetaa commented Jan 29, 2022

tac0turtle commented Jan 29, 2022

robert-zaremba left a comment

Choose a reason for hiding this comment

robert-zaremba Jan 29, 2022

Choose a reason for hiding this comment

faddat Jan 29, 2022

Choose a reason for hiding this comment

robert-zaremba commented Jan 29, 2022

faddat commented Jan 29, 2022

tac0turtle commented Jan 30, 2022

faddat commented Jan 31, 2022

tac0turtle commented Feb 1, 2022