indexer-alt: protocol_configs and feature_flags pipelines #20149
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
3 Skipped Deployments
|
amnn
temporarily deployed
to
sui-typescript-aws-kms-test-env
GitHub Actions
Inactive
amnn
force-pushed
the
amnn/idx-chain-id
branch
from
846cc2a
to
d0bde05
Compare
amnn
temporarily deployed
to
sui-typescript-aws-kms-test-env
GitHub Actions
Inactive
## Description Pipeline for writing out the set of latest packages. This is used to do type resolution during reads, and also to answer queries about package versions. Originally, this was called `kv_packages`, but I realised that because of system packages, it is not append-only, so I renamed it to match the other summary tables. ## Test plan Ran the indexer with the new pipeline on the first 1.2M checkpoints. This includes multiple system package upgrades and the first user package publish. Inspect the contents of the table at that point: ``` sui$ cargo run -p sui-indexer-alt --release -- \ --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \ indexer --remote-store-url https://checkpoints.mainnet.sui.io \ --last-checkpoint 1200000 --pipeline sum_packages ``` ``` sui_indexer_alt=# SELECT package_id, original_id, package_version FROM sum_packages; package_id | original_id | package_version --------------------------------------------------------------------+--------------------------------------------------------------------+----------------- \x0000000000000000000000000000000000000000000000000000000000000001 | \x0000000000000000000000000000000000000000000000000000000000000001 | 1 \x0000000000000000000000000000000000000000000000000000000000000002 | \x0000000000000000000000000000000000000000000000000000000000000002 | 2 \x0000000000000000000000000000000000000000000000000000000000000003 | \x0000000000000000000000000000000000000000000000000000000000000003 | 3 \x000000000000000000000000000000000000000000000000000000000000dee9 | \x000000000000000000000000000000000000000000000000000000000000dee9 | 2 \x39ac04c24dbedf422abb8d582973ee733dbbab07a597fc98300666abe7982034 | \x39ac04c24dbedf422abb8d582973ee733dbbab07a597fc98300666abe7982034 | 1 ```
## Description Pipeline for indexing Display VersionUpdated events, so readers can render Display for Objects. ## Test plan Run the indexer on the first ~1.6M checkpoints (the first display events happen around checkpoint 1.5M), and then inspect the database: ``` sui$ cargo run -p sui-indexer-alt --release -- \ --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \ indexer --remote-store-url https://checkpoints.mainnet.sui.io \ --consistent-range 3600 ``` ``` sui_indexer_alt=# SELECT object_type, display_id FROM sum_displays LIMIT 10; object_type | display_id ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------- \x72f9c76421170b5a797432ba9e1b3b2e2b7cf6faa26eb955396c773af2479e1e0967616d655f383139320847616d653831393200 | \xb06e4607642f6c08e196f8bd84b653c784864a6badca2066e63db42e846fdd4c \x72ad8f46d4fc3bbc24df15f6c62a98044c2649a9a31895a957254b2cf16e0cb905646d656e7305446d656e7300 | \x831cd05a3de12ae3dcea4382e9f56eadded8efd83c742036acd789fc41f1a03f \x72ad8f46d4fc3bbc24df15f6c62a98044c2649a9a31895a957254b2cf16e0cb905646d656e7309446d656e734d65746100 | \x3e8acb139222c9e335980f8b427270e92d276152140545bfde46c700472a2e11 \xee496a0cc04d06a345982ba6697c90c619020de9e274408c7819f787ff66e1a1087375696672656e73075375694672656e0107ee496a0cc04d06a345982ba6697c90c619020de9e274408c7819f787ff66e1a10463617079044361707900 | \x3d53effd472191435e7735ff9ed0b3083ef10dee4793db51df92038cea48b139 \x5b45da03d42b064f5e051741b6fed3b29eb817c7923b83b92f37a1d2abf4fbab036e6674034e667400 | \x23453afb61ba4e0c9f040839c0d620c2cab774491656c086e10871aae8f8b3e9 \xbb1531504c9c3235d3cd637ed9573cbe18461255b4175a1cb1e1b07b8aa8e11b0873686f77636173650853686f776361736500 | \xa306db0c6d3f5cbb8169143c22da5b61caf3576f1b3f3e77537aef8d4ca4b515 \xbb1531504c9c3235d3cd637ed9573cbe18461255b4175a1cb1e1b07b8aa8e11b0473756961054d6564616c00 | \x70e764f567b976d8e654064c14a7fbd94f0994a229fb0a7e119d80b4f1e4b760 \xbb1531504c9c3235d3cd637ed9573cbe18461255b4175a1cb1e1b07b8aa8e11b04737569610d506572736f6e616c4d6564616c00 | \xb438c4cbda18b0890de8ca2a1813be70fc3b64fedf171ea7c658d43605a11ece \xbb1531504c9c3235d3cd637ed9573cbe18461255b4175a1cb1e1b07b8aa8e11b08746573745f6e66740b53756961546573744e465400 | \xa37afca51a40bc90e30991631515f2ab8b9ecef90e51fbf4af12aa0ea5413250 \xf1e7b4093872df432b2adb50533a1fa6f9e4b20856217657242fe7d9d4ac42e80b6574686f735f73717561640a4574686f7353717561640107f1e7b4093872df432b2adb50533a1fa6f9e4b20856217657242fe7d9d4ac42e80b6574686f735f73717561640b4574686f7353717561643100 | \x7c330d61f99af7983e8714b4e63437d93813316ac54e8f83103b2ab2a021997e (10 rows) ```
## Description Make it possible to access the ingestion client on the ingestion service and indexer. This will be used to bootstrap the indexer which may need to wait for the genesis checkpoint to initialise certain tables and state before indexing proper starts. ## Test plan CI
## Description Factor out the logic used by the ingestion broadcaster to retry fetching a checkpoint if it is not found. This change is behaviour preserving but the new `wait_for` function will be re-used in some bootstrapping logic that waits for the existence of the genesis checkpoint. ## Test plan CI
## Description ...to be used in a bootstrapping script.
## Description Add explicit `NOT NULL`s to the key columns inside the event filter tables. This is not a behavioural change as these columns are already `NOT NULL` because they are part of the primary key. As per an earlier PR comment, relying on the implicit `NOT NULL` behaviour was confusing when reading migrations (and other migration scripts use the explicit pattern). ## Test plan CI
## Description Allow `graceful_shutdown` to accept tasks that return something other than `()`. ## Test plan CI
## Description Index the chain identifier and initial protocol version in a `kv_genesis` table. This takes the place of chain identifier indexing and will also be used by indexers for protocol configs and feature flags, which need to know the initial protocol version to index and the chain identifier. ## Test plan Run the indexer twice, and note from the tracing messages that the first time, the indexer is bootstrapped, and the second time it reads from the bootstrapped table.
## Description Change the pipeline/handler/processor interface to accept a value representing the handler/processor. This allows the processor to include configuration. This will be used by upcoming pipelines for processing protocol configs and feature flags. ## Test plan CI
## Description Add a pipeline for indexing protocol configs. ## Test plan Index the first ~1M checkpoints, which should pick up the first protocol version upgrade (from 4 to 5 for epoch 10) and then query `kv_protocol_configs` results after that: ``` sui$ cargo run -p sui-indexer-alt -- \ --database-url postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt \ --remote-store-url https://checkpoints.mainnet.sui.io \ --last-checkpoint 500000 ``` ``` sui_indexer_alt=# SELECT * FROM kv_protocol_configs; ```
## Description Like `kv_protocol_configs`, but for feature flags. ## Test plan Run the indexer over the first 700K checkpoints -- this will include two protocol config upgrades, then check the database for feature flags. ``` sui$ cargo run -p sui-indexer-alt --release -- \ --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \ indexer --remote-store-url https://checkpoints.mainnet.sui.io \ --last-checkpoint 700000 ``` ``` sui_indexer_alt=# SELECT * FROM kv_feature_flags ORDER BY protocol_version DESC, flag_name; protocol_version | flag_name | flag_value ------------------+-------------------------------------------------------------+------------ 6 | accept_zklogin_in_multisig | f 6 | advance_epoch_start_time_in_safe_mode | t 6 | advance_to_highest_supported_protocol_version | f 6 | allow_receiving_object_id | f 6 | authority_capabilities_v2 | f [...] ```
amnn
force-pushed
the
amnn/idx-chain-id
branch
from
d0bde05
to
719db00
Compare
amnn
temporarily deployed
to
sui-typescript-aws-kms-test-env
GitHub Actions
Inactive
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Indexing protocol configs and feature flags. To enable this, the framework was extended so that pipelines were added to the indexer by value, and that value was passed on to the processor to use as a form of configuration.
This is used to allow the protocol config and feature flag pipelines to be initialised with the row from the
kv_genesistable (which provides the chain identifier and initial protocol version).The PR is split over 3 commits, the first performs the necessary refactor, and the next two add the pipelines.
Test plan
Index the first ~700K checkpoints, which should pick up the first protocol version upgrade (from 4 to 5 for epoch 10) and then query the new tables:
Stack
Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.