-
Notifications
You must be signed in to change notification settings - Fork 590
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Database transition #695
Comments
PS: I haven't assigned this issue to myself because I think that this will take a concerted effort over time and that it should not be done suddenly, but instead gradually, validator by validator, perhaps speeding up over time. |
Comparison videos |
Steps to doing a switch from leveldb to rocksdb imo:
|
NB: rumor has it that @gainsalex from juno has made cw+state sync happy |
PS: we forked the go interface of rocksdb for the upcoming store/v2 release: https://github.com/cosmos/gorocksdb |
@ValarDragon cosmos/cosmos-sdk#10791 As discussed, mentioned issue here |
Ah thanks! That explains why I didn't see it, was looking on Tendermint |
Since we're adding wasm, this looks like a good candidate |
and Both deal with this. the Terra team mentioned diminishing returns on using RocksDB, so last week I began to experiment with PebbleDB. Since it is still possible to choose RocksDB, especially for maturity reasons, we've also updated tm-db to use grocksdb, which will make cosmos/gorocksdb unnecessary, and allow us to support RocksDB v7.x. #1234 also touches on these matters, fixing both state sync scripts. That said, I can't get them to work, specifically because Osmosis no longer crashes when it has applied the last snapshot. |
@ValarDragon like if configured to use rocks, but actually have goleveldb? |
Brief synopsis of new findings, based on some experimentation with pebbledb and using rocksdb v7: long term it is likely that pebble is better and pebble reads rocks, so there is an upgrade path that does not involve everyone updating their images. there is some kind of bug and it is in one of these places
If the bug isn't in pebbledb, it also means that we should improve the tests in tmdb / iavl. the implementation passes all tests. The work on PebbleDB is here: but regarding this bug, I could not find it, Khanh could not find it, Vuong could not find it, and Tuan could not find it. I thought it might be related to fast iavl so I tried all this on gaia and was able to reproduce the same kind of issue The rocksdb release notes make for interesting reading: |
Here's how to do it: install ipfs: you need to download a 1 tb file git clone https://github.com/osmosis-labs/osmosis --branch v6.x-forceprune
go install -ldflags '-w -s -X github.com/cosmos/cosmos-sdk/types.DBBackend=rocksdb' -tags rocksdb ./...
ipfs get QmfEuEx4r6sTZmp1odCMrytV4TWvyrn4ZxYHHVSgt1P38M -o osmo-v6-rocks.squashfs
osmosisd init
unsquashfs osmo-v6-rocks.squashfs data -d ~/.osmosisd/data
osmosisd start --db_backend rocksdb |
What exactly is the observed bug? |
ah, the observed bug in pebbledb is here: osmosis-labs/tm-db#4 (comment) Often manifests looking like a type error and this is in fact the reason for the "speedrun" setup I'm making. I had similarish errors with badger and gaia, I should add. But I really don't think we will have a hard and fast and clear answer to this until we are able to make changes to -- for example, tm-db and /or iavl and then run through the sequence, maybe pruning along the way. If I had to choose between pebble and rocks-- assuming that the upstream pebble is not the source of the problem (and indeed I bet that the bug is in our implementation) then the choice would easily be pebble. I assume that when it's referring to "traversals" that may have something to do with the iterator? |
@faddat do you think https://github.com/linxGnu/grocksdb is better than gorocksdb? What's the advantage of pebble against baddger? If I remember we disqualified pebble because of their limited (storage wise) checkpoint functionality: https://pkg.go.dev/github.com/cockroachdb/pebble#DB.Checkpoint . Did you benchmark pebble against grocksdb or gorocksdb? Maybe it will be worth to add it to cosmos-sdk/db, what do you think? |
OK, I've created cosmos/cosmos-sdk#11683 and it seams grocksdb has more support, is active and includes optimizations. |
Hello, I wanted to provide an update on this. Since some recent update, when using anything other than golveldb, osmosis will apply the 2nd last state sync snapshot, then complain of the app version mismatch, and crash. I think we need a lasting solution on the app version mismatch, weather from upstream, a fork, or here. |
Lets figure out how its set in a DB-agnostic way? Then perhaps we do it in the next upgrade, so we don't have any concerns around it being state machine breaking? |
Sounds great |
Summary
Goleveldb is much slower than Rocksdb, but rocksdb requires cgo and therefore build gymnastics, too. The best path forward is PebbleDB, which is a forwards compatible, pure go implementation of Rocksdb.
Problem Definition
Proposal
I propose that we disqualify all cgo based databases, for two reasons:
This would mean that we drop support for rocksdb and cleveldb altogether.
For Admin Use
The text was updated successfully, but these errors were encountered: