docs(store/v2): update store v2 docs (#19502)

Co-authored-by: cool-developer <[email protected]>

store/README.md

@@ -0,0 +1,31 @@
+# Store
+
+The `store` package contains the implementation of store/v2, which is the SDK's
+abstraction around managing historical and committed state. See [ADR-065](../docs/architecture/adr-065-store-v2.md)
+and [Store v2 Design](https://docs.google.com/document/d/1l6uXIjTPHOOWM5N4sUUmUfCZvePoa5SNfIEtmgvgQSU/edit#heading=h.nz8dqy6wa4g1) for a high-level overview of the design and rationale.
+
+## Migration
+
+<!-- TODO -->
+
+## Pruning
+
+The `root.Store` is NOT responsible for pruning. Rather, pruning is the responsibility
+of the underlying SS and SC layers. This means pruning can be implementation specific,
+such as being synchronous or asynchronous.
+
+## Usage
+
+The `store` package contains a `root.Store` type which is intended to act as an
+abstraction layer around it's two primary constituent components - state storage (SS)
+and state commitment (SC). It acts as the main entry point into storage for an
+application to use in server/v2. Through `root.Store`, an application can query
+and iterate over both current and historical data, commit new state, perform state
+sync, and fetch commitment proofs.
+
+A `root.Store` is intended to be initialized with already constructed SS and SC
+backends (see relevant package documentation for instantiation details). Note,
+from the perspective of `root.Store`, there is no notion of multi or single tree/store,
+rather these are implementation details of SS and SC. For SS, we utilize store keys
+to namespace raw key/value pairs. For SC, we utilize an abstraction, `commitment.CommitStore`,
+to map store keys to a commitment trees.

store/commitment/README.md

@@ -1,3 +1,47 @@
 # State Commitment (SC)
 
-TODO
+The `commitment` package contains the state commitment (SC) implementation.
+Specifically, it contains an IAVL v1 implementation of SC and the necessary types
+and abstractions to support other SC backends, as well as supporting general integration
+into store/v2, specifically the `RootStore` type.
+
+A foremost design goal is that SC backends should be easily swappable, i.e. not
+necessarily IAVL. To this end, the scope of SC has been reduced, it must only:
+
+* Provide a stateful root app hash for height h resulting from applying a batch
+  of key-value set/deletes to height h-1.
+* Fulfill (though not necessarily provide) historical proofs for all heights < `h`.
+* Provide an API for snapshot create/restore to fulfill state sync requests.
+
+Notably, SC is not required to provide key iteration or value retrieval for either
+queries or state machine execution, this now being the responsibility of state
+storage.
+
+An SC implementation may choose not to provide historical proofs past height `h - n`
+(`n` can be 0) due to the time and space constraints, but since store/v2 defines
+an API for historical proofs there should be at least one configuration of a
+given SC backend which supports this.
+
+## Benchmarks
+
+See this [section](https://docs.google.com/document/d/1l6uXIjTPHOOWM5N4sUUmUfCZvePoa5SNfIEtmgvgQSU/edit#heading=h.7l0i621y5vgm) for specifics on SC benchmarks on various implementations.
+
+## Pruning
+
+<!-- TODO -->
+
+## State Sync
+
+State commitment (SC) does not have a direct notion of state sync. Rather,
+`snapshots.Manager` is responsible for creating and restoring snapshots of the
+entire state. The `snapshots.Manager` has a `CommitSnapshotter` field which is
+fulfilled by the `CommitStore` type, specifically it implements the `Snapshot`
+and `Restore` methods.
+
+## Usage
+
+Similar to the `storage` package, the `commitment` package is designed to be used
+in a broader store implementation, i.e. it fulfills the role of the SC backend.
+Specifically, it provides a `CommitStore` type which accepts a `store.RawDB` and
+a mapping from store key, a string meant to represent a single module, to a `Tree`,
+which reflects the commitment structure.

store/storage/README.md

@@ -1,3 +1,110 @@
 # State Storage (SS)
 
-TODO
+The `storage` package contains the state storage (SS) implementation. Specifically,
+it contains RocksDB, PebbleDB, and SQLite (Btree) backend implementations of the
+`VersionedDatabase` interface.
+
+The goal of SS is to provide a modular storage backend, i.e. multiple implementations,
+to facilitate storing versioned raw key/value pairs in a fast embedded database,
+although an embedded database is not required, i.e. you could use a replicated
+RDBMS system.
+
+The responsibility and functions of SS include the following:
+
+* Provide fast and efficient queries for versioned raw key/value pairs
+* Provide versioned CRUD operations
+* Provide versioned batching functionality
+* Provide versioned iteration (forward and reverse) functionality
+* Provide pruning functionality
+
+All of the functionality provided by an SS backend should work under a versioned
+scheme, i.e. a user should be able to get, store, and iterate over keys for the
+latest and historical versions efficiently.
+
+## Backends
+
+### RocksDB
+
+The RocksDB implementation is a CGO-based SS implementation. It fully supports
+the `VersionedDatabase` API and is arguably the most efficient implementation. It
+also supports versioning out-of-the-box using User-defined Timestamps in
+ColumnFamilies (CF). However, it requires the CGO dependency which can complicate
+an app’s build process.
+
+### PebbleDB
+
+The PebbleDB implementation is a native Go SS implementation that is primarily an
+alternative to RocksDB. Since it does not support CF, results in the fact that we
+need to implement versioning (MVCC) ourselves. This comes with added implementation
+complexity and potential performance overhead. However, it is a pure Go implementation
+and does not require CGO.
+
+### SQLite (Btree)
+
+The SQLite implementation is another CGO-based SS implementation. It fully supports
+the `VersionedDatabase` API. The implementation is relatively straightforward and
+easy to understand as it’s entirely SQL-based. However, benchmarks show that this
+options is least performant, even for reads. This SS backend has a lot of promise,
+but needs more benchmarking and potential SQL optimizations, like dedicated tables
+for certain aspects of state, e.g. latest state, to be extremely performant.
+
+## Benchmarks
+
+Benchmarks for basic operations on all supported native SS implementations can
+be found in `store/storage/storage_bench_test.go`.
+
+At the time of writing, the following benchmarks were performed:
+
+```shell
+name                                              time/op
+Get/backend_rocksdb_versiondb_opts-10             7.41µs ± 0%
+Get/backend_pebbledb_default_opts-10              6.17µs ± 0%
+Get/backend_btree_sqlite-10                       29.1µs ± 0%
+ApplyChangeset/backend_pebbledb_default_opts-10   5.73ms ± 0%
+ApplyChangeset/backend_btree_sqlite-10            56.9ms ± 0%
+ApplyChangeset/backend_rocksdb_versiondb_opts-10  4.07ms ± 0%
+Iterate/backend_pebbledb_default_opts-10           1.04s ± 0%
+Iterate/backend_btree_sqlite-10                    1.59s ± 0%
+Iterate/backend_rocksdb_versiondb_opts-10          778ms ± 0%
+```
+
+## Pruning
+
+Pruning is an implementation and responsibility of the underlying SS backend.
+Specifically, the `StorageStore` accepts `store.PruneOptions` which defines the
+pruning configuration. During `ApplyChangeset`, the `StorageStore` will check if
+pruning should occur based on the current height being committed. If so, it will
+delegate a `Prune` call on the underlying SS backend, which can be defined specific
+to the implementation, e.g. asynchronous or synchronous.
+
+
+## State Sync
+
+State storage (SS) does not have a direct notion of state sync. Rather, `snapshots.Manager`
+is responsible for creating and restoring snapshots of the entire state. The
+`snapshots.Manager` has a `StorageSnapshotter` field which is fulfilled by the
+`StorageStore` type, specifically it implements the `Restore` method. The `Restore`
+method reads off of a provided channel and writes key/value pairs directly to a
+batch object which is committed to the underlying SS engine.
+
+## Non-Consensus Data
+
+<!-- TODO -->
+
+## Usage
+
+An SS backend is meant to be used within a broader store implementation, as it
+only stores data for direct and historical query purposes. We define a `Database`
+interface in the `storage` package which is mean to be represent a `VersionedDatabase`
+with only the necessary methods. The `StorageStore` interface is meant to wrap or
+accept this `Database` type, e.g. RocksDB.
+
+The `StorageStore` interface is an abstraction or wrapper around the backing SS
+engine can be seen as the the main entry point to using SS.
+
+Higher up the stack, there should exist a `root.Store` implementation. The `root.Store`
+is meant to encapsulate both an SS backend and an SC backend. The SS backend is
+defined by this `StorageStore` implementation.
+
+In short, initialize your SS engine of choice and then provide that to `NewStorageStore`
+which will further be provided to `root.Store` as the SS backend.

store/storage/storage_bench_test.go

@@ -32,6 +32,10 @@ var (
 		},
 		"pebbledb_default_opts": func(dataDir string) (store.VersionedDatabase, error) {
 			db, err := pebbledb.New(dataDir)
+			if err == nil && db != nil {
+				db.SetSync(false)
+			}
+
 			return storage.NewStorageStore(db, nil, log.NewNopLogger()), err
 		},
 		"btree_sqlite": func(dataDir string) (store.VersionedDatabase, error) {