indexer-alt: protocol_configs and feature_flags pipelines #20149

## Description Pipeline for writing out the set of latest packages. This is used to do type resolution during reads, and also to answer queries about package versions. Originally, this was called `kv_packages`, but I realised that because of system packages, it is not append-only, so I renamed it to match the other summary tables. ## Test plan Ran the indexer with the new pipeline on the first 1.2M checkpoints. This includes multiple system package upgrades and the first user package publish. Inspect the contents of the table at that point: ``` sui$ cargo run -p sui-indexer-alt --release -- \ --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \ indexer --remote-store-url https://checkpoints.mainnet.sui.io \ --last-checkpoint 1200000 --pipeline sum_packages ``` ``` sui_indexer_alt=# SELECT package_id, original_id, package_version FROM sum_packages; package_id | original_id | package_version --------------------------------------------------------------------+--------------------------------------------------------------------+----------------- \x0000000000000000000000000000000000000000000000000000000000000001 | \x0000000000000000000000000000000000000000000000000000000000000001 | 1 \x0000000000000000000000000000000000000000000000000000000000000002 | \x0000000000000000000000000000000000000000000000000000000000000002 | 2 \x0000000000000000000000000000000000000000000000000000000000000003 | \x0000000000000000000000000000000000000000000000000000000000000003 | 3 \x000000000000000000000000000000000000000000000000000000000000dee9 | \x000000000000000000000000000000000000000000000000000000000000dee9 | 2 \x39ac04c24dbedf422abb8d582973ee733dbbab07a597fc98300666abe7982034 | \x39ac04c24dbedf422abb8d582973ee733dbbab07a597fc98300666abe7982034 | 1 ```

## Description Pipeline for indexing Display VersionUpdated events, so readers can render Display for Objects. ## Test plan Run the indexer on the first ~1.6M checkpoints (the first display events happen around checkpoint 1.5M), and then inspect the database: ``` sui$ cargo run -p sui-indexer-alt --release -- \ --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \ indexer --remote-store-url https://checkpoints.mainnet.sui.io \ --consistent-range 3600 ``` ``` sui_indexer_alt=# SELECT object_type, display_id FROM sum_displays LIMIT 10; object_type | display_id ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------- \x72f9c76421170b5a797432ba9e1b3b2e2b7cf6faa26eb955396c773af2479e1e0967616d655f383139320847616d653831393200 | \xb06e4607642f6c08e196f8bd84b653c784864a6badca2066e63db42e846fdd4c \x72ad8f46d4fc3bbc24df15f6c62a98044c2649a9a31895a957254b2cf16e0cb905646d656e7305446d656e7300 | \x831cd05a3de12ae3dcea4382e9f56eadded8efd83c742036acd789fc41f1a03f \x72ad8f46d4fc3bbc24df15f6c62a98044c2649a9a31895a957254b2cf16e0cb905646d656e7309446d656e734d65746100 | \x3e8acb139222c9e335980f8b427270e92d276152140545bfde46c700472a2e11 \xee496a0cc04d06a345982ba6697c90c619020de9e274408c7819f787ff66e1a1087375696672656e73075375694672656e0107ee496a0cc04d06a345982ba6697c90c619020de9e274408c7819f787ff66e1a10463617079044361707900 | \x3d53effd472191435e7735ff9ed0b3083ef10dee4793db51df92038cea48b139 \x5b45da03d42b064f5e051741b6fed3b29eb817c7923b83b92f37a1d2abf4fbab036e6674034e667400 | \x23453afb61ba4e0c9f040839c0d620c2cab774491656c086e10871aae8f8b3e9 \xbb1531504c9c3235d3cd637ed9573cbe18461255b4175a1cb1e1b07b8aa8e11b0873686f77636173650853686f776361736500 | \xa306db0c6d3f5cbb8169143c22da5b61caf3576f1b3f3e77537aef8d4ca4b515 \xbb1531504c9c3235d3cd637ed9573cbe18461255b4175a1cb1e1b07b8aa8e11b0473756961054d6564616c00 | \x70e764f567b976d8e654064c14a7fbd94f0994a229fb0a7e119d80b4f1e4b760 \xbb1531504c9c3235d3cd637ed9573cbe18461255b4175a1cb1e1b07b8aa8e11b04737569610d506572736f6e616c4d6564616c00 | \xb438c4cbda18b0890de8ca2a1813be70fc3b64fedf171ea7c658d43605a11ece \xbb1531504c9c3235d3cd637ed9573cbe18461255b4175a1cb1e1b07b8aa8e11b08746573745f6e66740b53756961546573744e465400 | \xa37afca51a40bc90e30991631515f2ab8b9ecef90e51fbf4af12aa0ea5413250 \xf1e7b4093872df432b2adb50533a1fa6f9e4b20856217657242fe7d9d4ac42e80b6574686f735f73717561640a4574686f7353717561640107f1e7b4093872df432b2adb50533a1fa6f9e4b20856217657242fe7d9d4ac42e80b6574686f735f73717561640b4574686f7353717561643100 | \x7c330d61f99af7983e8714b4e63437d93813316ac54e8f83103b2ab2a021997e (10 rows) ```

## Description Make it possible to access the ingestion client on the ingestion service and indexer. This will be used to bootstrap the indexer which may need to wait for the genesis checkpoint to initialise certain tables and state before indexing proper starts. ## Test plan CI

## Description Factor out the logic used by the ingestion broadcaster to retry fetching a checkpoint if it is not found. This change is behaviour preserving but the new `wait_for` function will be re-used in some bootstrapping logic that waits for the existence of the genesis checkpoint. ## Test plan CI

## Description ...to be used in a bootstrapping script.

## Description Add explicit `NOT NULL`s to the key columns inside the event filter tables. This is not a behavioural change as these columns are already `NOT NULL` because they are part of the primary key. As per an earlier PR comment, relying on the implicit `NOT NULL` behaviour was confusing when reading migrations (and other migration scripts use the explicit pattern). ## Test plan CI

## Description Allow `graceful_shutdown` to accept tasks that return something other than `()`. ## Test plan CI

## Description Index the chain identifier and initial protocol version in a `kv_genesis` table. This takes the place of chain identifier indexing and will also be used by indexers for protocol configs and feature flags, which need to know the initial protocol version to index and the chain identifier. ## Test plan Run the indexer twice, and note from the tracing messages that the first time, the indexer is bootstrapped, and the second time it reads from the bootstrapped table.

## Description Change the pipeline/handler/processor interface to accept a value representing the handler/processor. This allows the processor to include configuration. This will be used by upcoming pipelines for processing protocol configs and feature flags. ## Test plan CI

## Description Add a pipeline for indexing protocol configs. ## Test plan Index the first ~1M checkpoints, which should pick up the first protocol version upgrade (from 4 to 5 for epoch 10) and then query `kv_protocol_configs` results after that: ``` sui$ cargo run -p sui-indexer-alt -- \ --database-url postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt \ --remote-store-url https://checkpoints.mainnet.sui.io \ --last-checkpoint 500000 ``` ``` sui_indexer_alt=# SELECT * FROM kv_protocol_configs; ```

## Description Like `kv_protocol_configs`, but for feature flags. ## Test plan Run the indexer over the first 700K checkpoints -- this will include two protocol config upgrades, then check the database for feature flags. ``` sui$ cargo run -p sui-indexer-alt --release -- \ --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \ indexer --remote-store-url https://checkpoints.mainnet.sui.io \ --last-checkpoint 700000 ``` ``` sui_indexer_alt=# SELECT * FROM kv_feature_flags ORDER BY protocol_version DESC, flag_name; protocol_version | flag_name | flag_value ------------------+-------------------------------------------------------------+------------ 6 | accept_zklogin_in_multisig | f 6 | advance_epoch_start_time_in_safe_mode | t 6 | advance_to_highest_supported_protocol_version | f 6 | allow_receiving_object_id | f 6 | authority_capabilities_v2 | f [...] ```

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

indexer-alt: protocol_configs and feature_flags pipelines #20149

indexer-alt: protocol_configs and feature_flags pipelines #20149

Commits on Nov 4, 2024

indexer-alt: protocol_configs and feature_flags pipelines #20149

Are you sure you want to change the base?

indexer-alt: protocol_configs and feature_flags pipelines #20149

Commits on Nov 4, 2024