Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: add docs (backport #575) #577

Open
wants to merge 1 commit into
base: release/v2.x.x
Choose a base branch
from
Open

docs: add docs (backport #575) #577

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
208 changes: 208 additions & 0 deletions docs/0-integrate-the-sdk.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,208 @@
# Integrate the Block SDK

The Block SDK is **open-source software** licensed under MIT. It is free to use, and has existing plug-and-play Lanes that work immediately!

Visit the GitHub repo [here](https://github.com/skip-mev/block-sdk).

We strive to be responsive to questions and issues within 1-2 weeks - please open a GitHub issue or join us on [**discord**](skip.build/discord). Note, we are not currently providing hands-on support for new integrations.


## ⚙️ Architecture [15 mins]

This is a high-level overview of the architecture, please reference [this page](2-how-it-works.md) or the [`Block-SDK` repo](https://github.com/skip-mev/block-sdk) for detailed and up to date info. For those eager to code, feel free to skip this and start down the page at **Set Up**!


### How Were Blocks Constructed pre-Block-SDK?

There are 3 relevant stages of consensus (these are all ABCI++ methods)

- **PrepareProposal**
- In this step, the consensus-engine (CometBFT, etc.) gives the application all of the transactions it has seen thus far.
- The app looks over these, performs some app-specific logic, and then gives them back to the consensus-engine. The consensus-engine then creates and broadcasts a proposal containing the transactions sent back from the app.
- **ProcessProposal**
- In this step, all validators check that the transactions in the proposal are valid, and that the proposal (as a whole) satisfies validity conditions determined by the application
- If the proposal fails, validators will not vote on the block, and the network will be forced to another round of consensus
- if the proposal passes, valdiators vote on the block, and the block will become canonical (barring unforeseen events)

### **Application Mempools**

In `v0.47.0` of the cosmos-sdk, **app-side mempools** were added to the SDK. With app-side mempools, validators no longer need to rely on the consensus-engine to keep track of and order all available transactions. Now applications can define their own mempool implementations, that

1. Store all pending (not finalized in a block) transactions
2. Order the set of pending transactions

#### **How does block-building change?**

Now in **PrepareProposal** instead of getting transactions from the consensus-engine, validators can pull transactions from their application-state aware mempools, and prioritize those transactions instead of the consensus-engine's transactions.

**Why is this better?**

- Mempools that are not app-state aware will not have the ability to make state-aware ordering rules. Like

1. All staker transactions are placed at the top of the block
2. All IBC `LightClientUpdate` messages are placed at the top of the block
3. Anything you can think of!!

- The consensus engine's mempool is generally in-efficient.
- The consensus-engine's mempool does not know when to remove transactions from its own mempool
- The consensus-engine spends most of its time re-broadcasting transactions between peers, hogging network bandwidth

## Block-SDK!!

The `Block-SDK` defines its own custom implementation of an **app-side mempool**, a `LaneMempool`. The `LaneMempool` is composed of `Lanes`, and handles transaction ingress, ordering, and cleaning.

**transaction ingress**

- The `LanedMempool` constructor defines an ordering of lanes. When a transaction is received by the app, it iterates through all lanes in order and inserts the transaction into the first `Lane` that it belongs in.
**ordering**
- Each `Lane` of the `LanedMempool` maintains its own ordering of transactions. When the `LanedMempool` routes a transaction to its corresponding `Lane` the `Lane` then inserts the transaction at its designated position with respect to all other transactions in the lane

### PrepareProposal

When the application is instructed to `PrepareProposal` it iterates through its `Lane`s in order, and calls each `Lane`'s `PrepareLane` method. The `Lane.PrepareLane` method collects transactions from a `Lane` and appends those transactions to the set of transactions from previous `Lane`'s `PrepareLane` calls. In other words, each block-proposal is now a collection of the transactions from the `LanedMempool`'s constituent lanes.

### ProcessProposal

When the application receives a proposal, and calls `ProcessProposal`, the app delegates the validation to the `LaneMempool.ProcessLanes` method. Remember, the proposal is composed of transactions from the sub-lanes of the `LaneMempool`, as such, the `LaneMempool` can route each `Lane`'s contribution to the Proposal to that `Lane` for validation. The proposal passes iff all `Lane`'s contributions are valid.

#### ⚠️ NOTE ⚠️

A block constructed from a `LaneMempool`'s `PrepareLanes` method must always pass that `LaneMempool`'s `ProcessLanes` method, otherwise, the chain will fail to produce blocks!! These functions are consensus critical, so practice caution when implementing them!!


## 📖 Set Up [20 mins]

To get set up, we're going to implement the `Default Lane`, which is the **most general and least restrictive** that accepts all transactions. This will cause **no changes** to your chain functionality, but will prepare you to add `lanes` with more functionality afterwards!

The default lane mirrors how CometBFT creates proposals today.

- It does a basic check to ensure that the transaction is valid.
- Orders the transactions based on tx fee amount (highest to lowest).
- The `PrepareLane` handler will reap transactions from the lane up to the `MaxBlockSpace` limit
- The `ProcessLane` handler will ensure that the transactions are ordered based on their fee amount and pass the same checks done in `PrepareLane`.

<!-- TODO: create script -->

# 🏗️ Default Lane Setup

## 📦 Dependencies

The Block SDK is built on top of the Cosmos SDK. The Block SDK is currently
compatible with Cosmos SDK versions greater than or equal to `v0.47.0`.

### Release Compatibility Matrix

| Block SDK Version | Cosmos SDK |
| :---------------: | :--------: |
| `v1.x.x` | `v0.47.x` |
| `v2.x.x` | `v0.50.x` |

## 📥 Adding the Block SDK to Your Project

```bash
$ go get github.com/skip-mev/block-sdk
```

## 📚 Usage

1. First determine the set of lanes that you want to use in your application. This guide only sets up the `default lane`

```golang
import (
"github.com/skip-mev/block-sdk/abci"
"github.com/skip-mev/block-sdk/block/base"
defaultlane "github.com/skip-mev/block-sdk/lanes/base"
)

// 1. Create the lanes.
//
// NOTE: The lanes are ordered by priority. The first lane is the highest priority
// lane and the last lane is the lowest priority lane. Top of block lane allows
// transactions to bid for inclusion at the top of the next block.
//
// For more information on how to utilize the LaneConfig please
// visit the README in docs.skip.money/chains/lanes/build-your-own-lane#-lane-config.
//
// Default lane accepts all transactions.

func NewApp() {
...
defaultConfig := base.LaneConfig{
Logger: app.Logger(),
TxEncoder: app.txConfig.TxEncoder(),
TxDecoder: app.txConfig.TxDecoder(),
MaxBlockSpace: math.LegacyZeroDec(),
MaxTxs: 0,
}
defaultLane := defaultlane.NewDefaultLane(defaultConfig)
// TODO(you): Add more Lanes!!!
```

2. In your base application, you will need to create a `LanedMempool` composed
of the `lanes` you want to use.

```golang
// 2. Set up the relative priority of lanes
lanes := []block.Lane{
defaultLane,
}
mempool := block.NewLanedMempool(app.Logger(), true, lanes...)
app.App.SetMempool(mempool)
```

3. Next, order the lanes by priority. The first lane is the highest priority lane
and the last lane is the lowest priority lane. **It is recommended that the last
lane is the default lane.**

```golang
// 3. Set up the ante handler.
anteDecorators := []sdk.AnteDecorator{
ante.NewSetUpContextDecorator(),
...
utils.NewIgnoreDecorator(
ante.NewDeductFeeDecorator(
options.BaseOptions.AccountKeeper,
options.BaseOptions.BankKeeper,
options.BaseOptions.FeegrantKeeper,
options.BaseOptions.TxFeeChecker,
),
options.FreeLane,
),
...
}

anteHandler := sdk.ChainAnteDecorators(anteDecorators...)

// Set the lane ante handlers on the lanes.
//
// NOTE: This step is very important. Without the antehandlers, lanes will not
// be able to verify transactions.
for _, lane := range lanes {
lane.SetAnteHandler(anteHandler)
}
app.App.SetAnteHandler(anteHandler)
```

4. You will also need to create a `PrepareProposalHandler` and a
`ProcessProposalHandler` that will be responsible for preparing and processing
proposals respectively. Configure the order of the lanes in the
`PrepareProposalHandler` and `ProcessProposalHandler` to match the order of the
lanes in the `LanedMempool`.

```golang
// 4. Set the abci handlers on base app
// Create the LanedMempool's ProposalHandler
proposalHandler := abci.NewProposalHandler(
app.Logger(),
app.TxConfig().TxDecoder(),
mempool,
)

// set the Prepare / ProcessProposal Handlers on the app to be the `LanedMempool`'s
app.App.SetPrepareProposal(proposalHandler.PrepareProposalHandler())
app.App.SetProcessProposal(proposalHandler.ProcessProposalHandler())
```

### 💅 Next step: implement other `lanes`

See the [Mev Lane](lanes/existing-lanes/1-mev.md) and select the `lanes` you want, or [Build Your Own](lanes/1-build-your-own-lane.md).
77 changes: 77 additions & 0 deletions docs/1-overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Overview

### 🤔What is the Block SDK?

**the Block SDK is a toolkit for building customized blocks**

The Block SDK is a set of Cosmos SDK and ABCI++ primitives that allow chains to fully customize blocks to specific use cases. It turns your chain's blocks into a **`highway`** consisting of individual **`lanes`** with their own special functionality.


Skip has built out a number of plug-and-play `lanes` on the SDK that your protocol can use, including in-protocol MEV recapture and Oracles! Additionally, the Block SDK can be extended to add **your own custom `lanes`** to configure your blocks to exactly fit your application needs.

🚦 **Blocks are like highways**

Let's say you're the designer of a 4 lane highway. You'd want a paid lane, for fast drivers who'd like to be separated from other lanes. You'd like a lane for large vehicles, you can configure this lane to be wider, require more space between vehicles, etc. The other two lanes are for the rest of traffic. The beauty here, is that as the owner of the highway, you get to decide what vehicles (transactions) you'll allow, and how they can behave (ordering)!!


#### If you've been here before

##### [Integrate Block-SDK](0-integrate-the-sdk.md)

##### [Building your own Lane](lanes/1-build-your-own-lane.md)

##### [Searcher docs for MEV Lane](3-searcher-docs.md)

### ❌ Problems: Blocks are not Customizable

Most Cosmos chains today utilize traditional block construction - which is too limited.

- Traditional block building is susceptible to MEV-related issues, such as front-running and sandwich attacks, since proposers have monopolistic rights on ordering and no verification of good behavior. MEV that is created cannot be redistributed to the protocol.
- Traditional block building uses a one-size-fits-all approach, which can result in inefficient transaction processing for specific applications or use cases and sub-optimal fee markets.
- Transactions tailored for specific applications may need custom prioritization, ordering or validation rules that the mempool is otherwise unaware of because transactions within a block are currently in-differentiable when a blockchain might want them to be.

### ✅ Solution: The Block SDK

You can think of the Block SDK as a **transaction `highway` system**, where each
`lane` on the highway serves a specific purpose and has its own set of rules and
traffic flow.

In the Block SDK, each `lane` has its own set of rules and transaction flow management systems.

- A `lane` is what we might traditionally consider to be a standard mempool
where transaction **_validation_**, **_ordering_** and **_prioritization_** for
contained transactions are shared.
- `lanes` implement a **standard interface** that allows each individual `lane` to
propose and validate a portion of a block.
- `lanes` are ordered with each other, configurable by developers. All `lanes`
together define the desired block structure of a chain.

### ✨ Block SDK Use Cases

A block with separate `lanes` can be used for:

1. **MEV mitigation**: a top of block lane could be designed to create an in-protocol top-of-block [auction](lanes/existing-lanes/1-mev.md) to recapture MEV in a transparent and governable way.
2. **Free/reduced fee txs**: transactions with certain properties (e.g. from trusted accounts or performing encouraged actions) could leverage a free lane to facilitate _good_ behavior.
3. **Dedicated oracle space** Oracles could be included before other kinds of transactions to ensure that price updates occur first, and are not able to be sandwiched or manipulated.
4. **Orderflow auctions**: an OFA lane could be constructed such that order flow providers can have their submitted transactions bundled with specific backrunners, to guarantee MEV rewards are attributed back to users
5. **Enhanced and customizable privacy**: privacy-enhancing features could be introduced, such as threshold encrypted lanes, to protect user data and maintain privacy for specific use cases.
6. **Fee market improvements**: one or many fee markets - such as EIP-1559 - could be easily adopted for different lanes (potentially custom for certain dApps). Each smart contract/exchange could have its own fee market or auction for transaction ordering.
7. **Congestion management**: segmentation of transactions to lanes can help mitigate network congestion by capping usage of certain applications and tailoring fee markets.

### 🎆 Chains Currently Using the Block-SDK

#### Mainnets

| Chain Name | Chain-ID | Block-SDK Version |
| ----------- | --------------- | ----------------- |
| Juno | `juno-1` | `v1.0.2` |
| Persistence | `persistence-1` | `v1.0.2` |
| Initia | `NA` | `v1.0.2` |
| Prism | `NA` | `v1.0.2` |
| Terra | `phoenix-1` | `v1.0.2` |

#### Testnets

| Chain Name | Chain-ID | Block-SDK Version |
| ---------- | -------- | ----------------- |
| Juno | `uni-6` | `v1.0.2` |
58 changes: 58 additions & 0 deletions docs/2-how-it-works.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# How it Works

<!-- TODO: add images to this -->

### Summary

With the Block SDK, blocks are broken up into smaller partial blocks called `lanes`.

- Each `lane` has its own custom block building logic and stores distinct types of transactions.
- Each lane can only consume a portion of the block as defined on the `lane`'s configuration (`MaxBlockSpace`).
- When a block proposal is requested, a block will **fill** with transactions from each `lane`, iteratively, in the order in which the `lanes` are defined in the application.
- When a block proposal is processed, each `lane` will **verify** its portion of the block, iteratively, in the order in which the `lanes` are defined in the application.
- **Transactions in blocks MUST respect the ordering of lanes.**

### 🔁 Background: Transaction Lifecycle

Knowledge of the general transaction lifecycle is important to understand how `lanes` work.

- A transaction begins when it is signed and broadcasted to a node on a chain.
- It will be then be verified by the application on the node.
- If it is valid, it will be inserted into the node's `mempool`, which is a storage area for transactions before inclusion in a block.
- If the node happens to be a `validator`, and is proposing a block, the application will call `PrepareProposal` to create a new block proposal.
- The proposer will look at what transactions they have in their mempool, iteratively select transactions until the block is full, and share the proposal with other validators.
- When a different validator receives a proposal, the validator will verify its contents via `ProcessProposal` before signing it.
- If the proposal is valid, the validator will sign the proposal and broadcast their vote to the network.
- If the block is invalid, the validator will reject the proposal.
- Once a proposal is accepted by the network, it is committed as a block and the transactions that were included are removed from every validator's mempool.

### 🛣️ Lane Lifecycle

`Lanes` introduce new steps in the transaction lifecycle outlined above.

A `LanedMempool` is composed of several distinct `lanes` that store their own transactions. The `LanedMempool` will insert the transaction into all `lanes` that accept it

- After the base application accepts a transaction, the transaction will be checked to see if it can go into any `lanes`, as defined by the lane's `MatchHandler`.
- `Lane`'s can be configured to only accept transactions that match a certain criteria. For example, a `lane` could be configured to only accept transactions that are staking related (such as a free-transaction lane).
- When a new block is proposed, the `PrepareProposalHandler` of the application will iteratively call `PrepareLane` on each `lane` (in the order in which they are defined in the application). The `PrepareLane` method is similar to `PrepareProposal`.
- Calling `PrepareLane` on a `lane` will trigger the lane to reap transactions from its mempool and add them to the proposal (if they respect the verification rules of the `lane`).
- When proposals are verified in `ProcessProposal` by other validators, the `ProcessProposalHandler` defined in `abci/abci.go` will call `ProcessLane` on each `lane` in the same order as they were called in the `PrepareProposalHandler`.
- Each subsequent call to `ProcessLane` will filter out transactions that belong to previous lanes. **A given lane's ProcessLane will only verify transactions that belong to that lane.**

**Scenario**

Let's say we have a `LanedMempool` composed of two lanes: `LaneA` and `LaneB`.

`LaneA` is defined first in the `LanedMempool` and `LaneB` is defined second.

`LaneA` contains transactions Tx1 and Tx2 and `LaneB` contains transactions
Tx3 and Tx4.


When a new block needs to be proposed, the `PrepareProposalHandler` will call `PrepareLane` on `LaneA` first and `LaneB` second.

When `PrepareLane` is called on `LaneA`, `LaneA` will reap transactions from its mempool and add them to the proposal. The same applies for `LaneB`. Say `LaneA` reaps transactions Tx1 and Tx2 and `LaneB` reaps transactions Tx3 and Tx4. This gives us a proposal composed of the following:

- `Tx1`, `Tx2`, `Tx3`, `Tx4`

When the `ProcessProposalHandler` is called, it will call `ProcessLane` on `LaneA` with the proposal composed of Tx1, Tx2, Tx3, and Tx4. `LaneA` will then verify Tx1 and Tx2 and return the remaining transactions - Tx3 and Tx4. The `ProcessProposalHandler` will then call `ProcessLane` on `LaneB` with the remaining transactions - Tx3 and Tx4. `LaneB` will then verify Tx3 and Tx4 and return no remaining transactions.
Loading
Loading