Oracle daemon for Lido decentralized staking service. Collects and reports Ethereum 2.0 beacon chain states (the number of visible validators and their summarized balances) to the Lido dApp contract running on Ethereum 1.0 side.
-
Upon the start daemon determines the reportable epoch and retrieves the list of validator keys to watch for.
-
Then it Gets the Lido-controlled validators from the reportable beacon state and summarizes their balances.
-
Constructs the transaction containing the
epochId,beaconBalance, andbeaconValidators. -
If the daemon has
MEMBER_PRIV_KEYin its environment (i.e. isn't running in dry-run mode), it signs and sends the transaction to the oracle contract (asking for user confirmation if running interactively). -
If the oracle runs in the daemon mode (with
DAEMON=1env ) it waitsSLEEPseconds and restarts the loop.
The oracle daemon requires fully-synced ETH1.0 and Beacon nodes. We highly recommend using geth and Lighthouse.
Note: Prysm beacon client is also supported, but has less API performance.
docker run -d --name geth -v $HOME/.geth:/root -p 30303:30303 -p 8545:8545 ethereum/client-go --http --http.addr=0.0.0.0
docker run -d --name lighthouse -v $HOME/.ligthouse:/root/.lighthouse -p 9000:9000 -p 5052:5052 sigp/lighthouse lighthouse beacon --http --http-address 0.0.0.0The oracle receives its configuration via ENVironment variables. You need to provide URIs of both nodes and the Lido contract address. The following snippet (adapted to your setup) will start the oracle in safe, read-only mode called Dry-run. It will run the single loop iteration, calculate the report and print it out instead of sending real TX.
export WEB3_PROVIDER_URI=$ETH1_NODE_RPC_ADDRESS
export BEACON_NODE=$ETH2_NODE_RPC_ADDRESS
export MEMBER_PRIV_KEY=$ORACLE_PRIVATE_KEY_0X_PREFIXED
export POOL_CONTRACT=0xae7ab96520DE3A18E5e111B5EaAb095312D7fE84
export STETH_PRICE_ORACLE_CONTRACT=0x3a6bd15abf19581e411621d669b6a2bbe741ffd6
export STETH_CURVE_POOL_CONTRACT=0xDC24316b9AE028F1497c275EB9192a3Ea0f67022
export DAEMON=0
export ORACLE_FROM_BLOCK=11595281
docker run -e WEB3_PROVIDER_URI -e BEACON_NODE -e POOL_CONTRACT -e DAEMON -e ORACLE_FROM_BLOCK -it lidofinance/oracle:2.0.0Other pre-built oracle images can be found in the Lido dockerhub.
See Other examples below for transactable modes.
WEB3_PROVIDER_URI- HTTP or WS URL of web3 Ethereum node (tested with Geth). Required.BEACON_NODE- HTTP endpoint of Beacon Node (Lighthouse recommended, also tested with Prysm). Required.POOL_CONTRACT- Lido contract in EIP-55 (mixed-case) hex format. Required. Example:0xae7ab96520DE3A18E5e111B5EaAb095312D7fE84STETH_CURVE_POOL_CONTRACT- address of Curve ETH/stETH stable swap pool. If provided together withSTETH_PRICE_ORACLE_CONTRACTstETH price oracle will be enabled.STETH_PRICE_ORACLE_CONTRACT- address of Lido's stable swap state oracle. If provided together withSTETH_CURVE_POOL_CONTRACTstETH price oracle will be enabled.STETH_PRICE_ORACLE_BLOCK_NUMBER_SHIFT- indent fromlatestblock number to be used in computation of new state for stable swap. Optional. Default:15DAEMON- withDAEMON=0runs the single iteration then quits.DAEMON=0in combination withMEMBER_PRIV_KEYruns interactively and asks user for confirmation before sending each TX. WithDAEMON=1runs autonomously (without confirmation) in an indefinite loop. Optional. Default:0MEMBER_PRIV_KEY- Hex-encoded private key of Oracle Quorum Member address. Optional. If omitted, the oracle runs in read-only (dry-run) mode. WARNING: KeepMEMBER_PRIV_KEYsafe. Since it keeps real Ether to pay gas, it should never be exposed outside.FORCE_DO_NOT_USE_IN_PRODUCTION- Do not use in production! The oracle makes the sanity checks on the collected data before reporting. Running inDAEMONmode, if data look suspicious, it skips sending TX. In enforced mode it gets reported even if it looks suspicious. It's unsafe and used for smoke testing purposes, NEVER use it in production! Optional. Default:0SLEEPseconds - The interval between iterations in Daemon mode. Default value: 60 s. Effective withDAEMON=1only.GAS_LIMIT- The pre-defined gasLimit for composed transaction. Defaulf value: 1 500 000. Effective in transactable mode (with givenMEMBER_PRIV_KEY)ORACLE_FROM_BLOCK- The earlist block to check for oracle events. Needed on mainnet first run to skip 5 minutes of scanning blockchain for events that are not there, recommended to be set to 11595281 on mainnet deploymentsPROMETHEUS_PREFIX- Prefix for all prometheus metrics. This is good practice having different prefixes for applications (recommended to uselido_oracle_) Optional. Default ''
- WARNING: The examples below are transactable and can potentially break the Lido. You must understand the protocol and what you are doing.
- WARNING: Keep your
MEMBER_PRIV_KEYsafe. Since it keeps real Ether to pay gas, it should never be exposed outside. - WARNING: Never use the
MEMBER_PRIV_KEYvalue given below. You will definitely lose all your Ethers if reuse that private key.
This mode is intended for controlled start and allows to double-check the report and its effects before its actual sending. Runs the single iteration and asks for confirmation via interactive [y/n] prompt before sending real TX to the network. You should be connected (attached) to the terminal to see this.
export WEB3_PROVIDER_URI=$ETH1_NODE_RPC_ADDRESS
export BEACON_NODE=$ETH2_NODE_RPC_ADDRESS
export MEMBER_PRIV_KEY=$ORACLE_PRIVATE_KEY_0X_PREFIXED
export POOL_CONTRACT=0xae7ab96520DE3A18E5e111B5EaAb095312D7fE84
export STETH_PRICE_ORACLE_CONTRACT=0x3a6bd15abf19581e411621d669b6a2bbe741ffd6
export STETH_CURVE_POOL_CONTRACT=0xDC24316b9AE028F1497c275EB9192a3Ea0f67022
export DAEMON=0
export ORACLE_FROM_BLOCK=11595281
docker run -e WEB3_PROVIDER_URI -e BEACON_NODE -e POOL_CONTRACT -e DAEMON -e MEMBER_PRIV_KEY -e ORACLE_FROM_BLOCK -it lidofinance/oracle:2.0.0Runs in the background with 1-hour pauses between consecutive iterations. To be used without human supervision (on later stages).
export WEB3_PROVIDER_URI=$ETH1_NODE_RPC_ADDRESS
export BEACON_NODE=$ETH2_NODE_RPC_ADDRESS
export MEMBER_PRIV_KEY=$ORACLE_PRIVATE_KEY_0X_PREFIXED
export POOL_CONTRACT=0xae7ab96520DE3A18E5e111B5EaAb095312D7fE84
export STETH_PRICE_ORACLE_CONTRACT=0x3a6bd15abf19581e411621d669b6a2bbe741ffd6
export STETH_CURVE_POOL_CONTRACT=0xDC24316b9AE028F1497c275EB9192a3Ea0f67022
export DAEMON=1
export SLEEP=300
export ORACLE_FROM_BLOCK=11595281
docker run -e WEB3_PROVIDER_URI -e BEACON_NODE -e POOL_CONTRACT -e DAEMON -e MEMBER_PRIV_KEY -e SLEEP -e ORACLE_FROM_BLOCK lidofinance/oracle:2.0.0Instead of downloading the image from dockerhub, you can build it yourself. This requires git and python3.8+.
./build.shTo build and push with the given version tag to the dockerhub:
TAG=0.1.3 PUSH=1 ./build.shPrometheus exporter is running on port 8000 and provides 5 logical groups of metrics.
Current Oracle daemon's state.
| name | description | frequency | goal |
|---|---|---|---|
| reportableFrame gauge |
the report could be sent or is sending | ||
| nowEthV1BlockNumber gauge |
ETH1 latest block number | every COUNTDOWN_SLEEP seconds | should be increasing constantly and be aligned with https://etherscan.io/blocks |
| daemonCountDown gauge |
time till the next oracle run in seconds | every COUNTDOWN_SLEEP seconds | should be decreasing down to 0 |
| finalizedEpoch gauge |
last finalized ETH2 epoch | every COUNTDOWN_SLEEP seconds | should go up at a rate of 1 per six munites |
Oracle process stats.
| name | description | frequency | goal |
|---|---|---|---|
| txSuccess histogram |
number of successful transactions | every SLEEP seconds | |
| txRevert histogram |
number of failed transactions | every SLEEP seconds | |
| process_virtual_memory_bytes gauge |
Virtual memory size in bytes. | every call | normal RAM consumption is ~200Mb |
| process_resident_memory_bytes gauge |
Resident memory size in bytes. | every call | normal RAM consumption is ~200Mb |
| process_start_time_seconds gauge |
Start time of the process since unix epoch in seconds. | every call | |
| process_cpu_seconds_total counter |
Total user and system CPU time spent in seconds. | every call | |
| process_open_fds gauge |
Number of open file descriptors. | every call | |
| process_max_fds gauge |
Maximum number of open file descriptors. | every call |
The previous and the current frame variables.
| name | description | frequency | goal |
|---|---|---|---|
| deltaSeconds gauge |
current.timestamp - previous.timestamp | every SLEEP seconds | should be approximately equal to the delay between reports |
| appearedValidators gauge |
current.beaconValidators - previous.beaconValidators | every SLEEP seconds | |
| currentEthV1BlockNumber gauge |
block number of the most current oracle stats check | every SLEEP seconds | should be constantly updated and be aligned with https://etherscan.io/blocks |
| currentValidatorsKeysNumber gauge |
len(validators_keys) | every time there is an unreported frame (1/day or potentially rarer) | |
| currentEpoch gauge |
every SLEEP seconds | ||
| currentTimestamp gauge |
every SLEEP seconds | ||
| currentBeaconValidators gauge |
every SLEEP seconds | ||
| currentBeaconBalance gauge |
every SLEEP seconds | ||
| currentBufferedBalance gauge |
every SLEEP seconds | ||
| currentDepositedValidators gauge |
every SLEEP seconds | ||
| currentActiveValidatorBalance gauge |
every SLEEP seconds | ||
| currentTotalPooledEther gauge |
every SLEEP seconds | ||
| currentTransientValidators gauge |
every SLEEP seconds | ||
| currentTransientBalance gauge |
every SLEEP seconds | ||
| prevEthV1BlockNumber gauge |
block number of the previous oracle stats check | every SLEEP seconds | should be constantly updated and be aligned with https://etherscan.io/blocks |
| prevEpoch gauge |
every SLEEP seconds | ||
| prevTimestamp gauge |
every SLEEP seconds | ||
| prevBeaconValidators gauge |
every SLEEP seconds | ||
| prevBeaconBalance gauge |
every SLEEP seconds | ||
| prevBufferedBalance gauge |
every SLEEP seconds | ||
| prevDepositedValidators gauge |
every SLEEP seconds | ||
| prevActiveValidatorBalance gauge |
every SLEEP seconds | ||
| prevTotalPooledEther gauge |
every SLEEP seconds | ||
| prevTransientValidators gauge |
every SLEEP seconds | ||
| prevTransientBalance gauge |
every SLEEP seconds |
Current stETH price in the pool and the price oracle.
| name | description | frequency | goal |
|---|---|---|---|
| stethOraclePrice gauge |
every call | ||
| stethPoolPrice gauge |
every call |
Exception counters.
| name | description |
|---|---|
| underpricedExceptionsCount gauge |
count of ValueError: replacement transaction underpriced |
| transactionTimeoutCount gauge |
count of web3.exceptions.TimeExhausted |
| beaconNodeTimeoutCount gauge |
count of beacon node connection timeouts |
| exceptionsCount gauge |
count of all other exceptions |
Metrics provided allow for multiple useful alerts on oracle health and performance. We strongly recommend setting up at least two alerts as follows:
- There were no Beacon oracle reports about the last finalized Beacon epoch for more than 30 minutes since that epoch has become finalized.
- alert: reported_frame
expr: currentEpoch > prevEpoch
for: 30m
labels:
severity: critical
annotations:
title: No report for current frame for 30 minutes
- Curve stETH pool live price (
stethPoolPricegauge) differs from the stETH oracle's price (stethOraclePricegauge) by more than 5% for at least 10 minutes.
- alert: peg
expr: abs((stethPoolPrice - stethOraclePrice)/stethOraclePrice) > 0.05
for: 10m
labels:
severity: critical
annotations:
title: Peg difference is greater than 5%
description: Peg difference is greater than 5% for more than 10 minutes.
2020 Lido [email protected]
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3 of the License, or any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.