Replay Verify scheduler #15103

areshand · 2024-10-28T22:34:53Z

Description

A basic scheduler that can schedule tasks based on config and collect results for reporting.
Next step is to put it on a real cluster and add it to github workflow
automation for creating snapshot from archive node, creating disk, PV, PVC for k8s cluster
fix a bug on replay-on-archive when no txn to be executed

How Has This Been Tested?

Tested on my local kube cluster

Key Areas to Review

Type of Change

Which Components or Systems Does This Change Impact?

Checklist

I have read and followed the CONTRIBUTING doc
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I identified and added all stakeholders and component owners affected by this change as reviewers
I tested both happy and unhappy path of the functionality
I have made corresponding changes to the documentation

trunk-io · 2024-10-28T22:34:56Z

⏱️ 2h 54m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
rust-cargo-deny	25m	🟩 🟩 ⬜ 🟩 🟩 (+11 more)
check-dynamic-deps	21m	🟩 🟩 🟩 🟩 🟩 (+12 more)
rust-move-tests	10m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
semgrep/ci	8m	🟩 🟩 🟩 🟩 🟩 (+12 more)
general-lints	7m	🟩 🟩 ⬜ 🟩 🟩 (+11 more)
rust-move-tests	7m	⬜

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
check-dynamic-deps	2m	1m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

msmouse

🪄

msmouse · 2024-10-30T20:54:58Z

testsuite/replay-verify/main.py

+from enum import Enum
+
+
+# Hyperparameters


what? is there any machine learning involved? 😂

removed the ones that are supposed to passed by config
The rest should be fixed

msmouse · 2024-10-30T20:59:03Z

testsuite/replay-verify/main.py

+    # Check if the environment variable already exists
+    for env_var in container["env"]:
+        if env_var["name"] == name:
+            env_var["value"] = value
+            return
+
+    # If it doesn't exist, add it
+    container["env"].append({"name": name, "value": value})


what, you can't just "upsert"?
(I've forgotten Python)

this is a vector of map/dict and each dict is {"name": name, "value": value}. The format is from k8s config.

testsuite/replay-verify/main.py

testsuite/replay-verify/README.md

perryjrandall · 2024-10-31T17:52:58Z

testsuite/replay-verify/replay-verify-worker-template.yaml

+  serviceAccountName: default  
+  restartPolicy: Never  # Pod restarts only if it fails  
+  containers:  
+    - name: replay-verify-worker  


probably gonna wanna generate this in code

this is updated dynamically instart()function. This file serves as a template

perryjrandall

You might wanna add replay-verify as a tool in poetry so you can poetry run the tool

https://gist.github.com/perryjrandall/cc38e1a47c49d7da807a512ce7283796

github-actions · 2024-11-12T19:10:28Z

✅ Forge suite `realistic_env_max_load` success on `957ee8f25fe21c155227ecde11b746eef469e43d`

two traffics test: inner traffic : committed: 14430.47 txn/s, latency: 2757.37 ms, (p50: 2700 ms, p70: 2700, p90: 2900 ms, p99: 3200 ms), latency samples: 5486780
two traffics test : committed: 100.09 txn/s, latency: 1603.13 ms, (p50: 1300 ms, p70: 1400, p90: 1600 ms, p99: 9400 ms), latency samples: 1760
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.038, avg: 1.584", "ConsensusProposalToOrdered: max: 0.309, avg: 0.292", "ConsensusOrderedToCommit: max: 0.363, avg: 0.357", "ConsensusProposalToCommit: max: 0.657, avg: 0.648"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.08s no progress at version 2682400 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.50s no progress at version 2682398 (avg 8.50s) [limit 15].
Test Ok

github-actions · 2024-11-12T19:12:20Z

✅ Forge suite `framework_upgrade` success on `fcd2dedf6ca61a23f4979e944c629dcdbdae5dca` ==> `957ee8f25fe21c155227ecde11b746eef469e43d`

Compatibility test results for fcd2dedf6ca61a23f4979e944c629dcdbdae5dca ==> 957ee8f25fe21c155227ecde11b746eef469e43d (PR)
Upgrade the nodes to version: 957ee8f25fe21c155227ecde11b746eef469e43d
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1413.10 txn/s, submitted: 1415.63 txn/s, failed submission: 2.53 txn/s, expired: 2.53 txn/s, latency: 2247.20 ms, (p50: 2100 ms, p70: 2300, p90: 3300 ms, p99: 5200 ms), latency samples: 122900
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1288.27 txn/s, submitted: 1289.81 txn/s, failed submission: 1.54 txn/s, expired: 1.54 txn/s, latency: 2307.37 ms, (p50: 2100 ms, p70: 2400, p90: 3900 ms, p99: 5200 ms), latency samples: 117240
5. check swarm health
Compatibility test for fcd2dedf6ca61a23f4979e944c629dcdbdae5dca ==> 957ee8f25fe21c155227ecde11b746eef469e43d passed
Upgrade the remaining nodes to version: 957ee8f25fe21c155227ecde11b746eef469e43d
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1313.48 txn/s, submitted: 1317.55 txn/s, failed submission: 4.07 txn/s, expired: 4.07 txn/s, latency: 2305.59 ms, (p50: 2100 ms, p70: 2400, p90: 3900 ms, p99: 5700 ms), latency samples: 116180
Test Ok

github-actions · 2024-11-12T19:13:31Z

✅ Forge suite `compat` success on `fcd2dedf6ca61a23f4979e944c629dcdbdae5dca` ==> `957ee8f25fe21c155227ecde11b746eef469e43d`

Compatibility test results for fcd2dedf6ca61a23f4979e944c629dcdbdae5dca ==> 957ee8f25fe21c155227ecde11b746eef469e43d (PR)
1. Check liveness of validators at old version: fcd2dedf6ca61a23f4979e944c629dcdbdae5dca
compatibility::simple-validator-upgrade::liveness-check : committed: 17053.15 txn/s, latency: 1973.82 ms, (p50: 1900 ms, p70: 2100, p90: 2200 ms, p99: 3300 ms), latency samples: 556620
2. Upgrading first Validator to new version: 957ee8f25fe21c155227ecde11b746eef469e43d
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7058.06 txn/s, latency: 4025.58 ms, (p50: 4400 ms, p70: 4800, p90: 4800 ms, p99: 4900 ms), latency samples: 130080
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7183.52 txn/s, latency: 4496.61 ms, (p50: 4900 ms, p70: 5000, p90: 5700 ms, p99: 6400 ms), latency samples: 237200
3. Upgrading rest of first batch to new version: 957ee8f25fe21c155227ecde11b746eef469e43d
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7749.55 txn/s, latency: 3735.14 ms, (p50: 4100 ms, p70: 4400, p90: 4500 ms, p99: 4600 ms), latency samples: 145320
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7607.30 txn/s, latency: 4226.81 ms, (p50: 4500 ms, p70: 4600, p90: 5400 ms, p99: 5900 ms), latency samples: 253620
4. upgrading second batch to new version: 957ee8f25fe21c155227ecde11b746eef469e43d
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 11377.12 txn/s, latency: 2446.94 ms, (p50: 2600 ms, p70: 2900, p90: 3200 ms, p99: 3300 ms), latency samples: 197360
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10757.14 txn/s, latency: 2905.02 ms, (p50: 2800 ms, p70: 3100, p90: 4500 ms, p99: 5400 ms), latency samples: 350140
5. check swarm health
Compatibility test for fcd2dedf6ca61a23f4979e944c629dcdbdae5dca ==> 957ee8f25fe21c155227ecde11b746eef469e43d passed
Test Ok

areshand force-pushed the new_replay_verify branch 5 times, most recently from 2d4bcf2 to bdb718b Compare October 30, 2024 17:19

areshand changed the title ~~New replay verify~~ Replay Verify scheduler Oct 30, 2024

areshand requested review from perryjrandall, sionescu and msmouse October 30, 2024 17:23

areshand marked this pull request as ready for review October 30, 2024 17:24

areshand force-pushed the new_replay_verify branch 2 times, most recently from c776074 to b40e99a Compare October 30, 2024 17:35

areshand requested review from lightmark and grao1991 as code owners October 30, 2024 17:35

areshand force-pushed the new_replay_verify branch from b40e99a to 78dfd06 Compare October 30, 2024 17:36

areshand requested a review from aluon October 30, 2024 17:40

msmouse approved these changes Oct 30, 2024

View reviewed changes

perryjrandall reviewed Oct 31, 2024

View reviewed changes

testsuite/replay-verify/README.md Outdated Show resolved Hide resolved

perryjrandall reviewed Oct 31, 2024

View reviewed changes

areshand force-pushed the new_replay_verify branch from 78dfd06 to 490666f Compare October 31, 2024 22:00

areshand requested a review from perryjrandall October 31, 2024 22:04

areshand force-pushed the new_replay_verify branch 7 times, most recently from 5311ad3 to 5f4e5e8 Compare November 3, 2024 17:19

areshand force-pushed the new_replay_verify branch 6 times, most recently from 26d9c75 to b49aa77 Compare November 11, 2024 22:50

areshand added the CICD:build-performance-images build performance docker image variants label Nov 11, 2024

areshand force-pushed the new_replay_verify branch from b49aa77 to 65a65fc Compare November 12, 2024 03:49

add replay verify scheduler and fix a minor bug in replay-on-archive

957ee8f

areshand force-pushed the new_replay_verify branch from 65a65fc to 957ee8f Compare November 12, 2024 17:41

grao1991 approved these changes Nov 12, 2024

View reviewed changes

areshand enabled auto-merge (rebase) November 12, 2024 18:34

This comment has been minimized.

Sign in to view

areshand merged commit 8cefc2e into main Nov 12, 2024
92 checks passed

areshand deleted the new_replay_verify branch November 12, 2024 19:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replay Verify scheduler #15103

Replay Verify scheduler #15103

areshand commented Oct 28, 2024 •

edited

Loading

trunk-io bot commented Oct 28, 2024 •

edited

Loading

msmouse left a comment

msmouse Oct 30, 2024

areshand Oct 31, 2024

msmouse Oct 30, 2024

areshand Oct 31, 2024 •

edited

Loading

perryjrandall Oct 31, 2024

areshand Oct 31, 2024

perryjrandall left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Nov 12, 2024

github-actions bot commented Nov 12, 2024

github-actions bot commented Nov 12, 2024

Replay Verify scheduler #15103

Replay Verify scheduler #15103

Conversation

areshand commented Oct 28, 2024 • edited Loading

Description

How Has This Been Tested?

Key Areas to Review

Type of Change

Which Components or Systems Does This Change Impact?

Checklist

trunk-io bot commented Oct 28, 2024 • edited Loading

msmouse left a comment

Choose a reason for hiding this comment

msmouse Oct 30, 2024

Choose a reason for hiding this comment

areshand Oct 31, 2024

Choose a reason for hiding this comment

msmouse Oct 30, 2024

Choose a reason for hiding this comment

areshand Oct 31, 2024 • edited Loading

Choose a reason for hiding this comment

perryjrandall Oct 31, 2024

Choose a reason for hiding this comment

areshand Oct 31, 2024

Choose a reason for hiding this comment

perryjrandall left a comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Nov 12, 2024

✅ Forge suite realistic_env_max_load success on 957ee8f25fe21c155227ecde11b746eef469e43d

github-actions bot commented Nov 12, 2024

✅ Forge suite framework_upgrade success on fcd2dedf6ca61a23f4979e944c629dcdbdae5dca ==> 957ee8f25fe21c155227ecde11b746eef469e43d

github-actions bot commented Nov 12, 2024

✅ Forge suite compat success on fcd2dedf6ca61a23f4979e944c629dcdbdae5dca ==> 957ee8f25fe21c155227ecde11b746eef469e43d

areshand commented Oct 28, 2024 •

edited

Loading

trunk-io bot commented Oct 28, 2024 •

edited

Loading

areshand Oct 31, 2024 •

edited

Loading

✅ Forge suite `realistic_env_max_load` success on `957ee8f25fe21c155227ecde11b746eef469e43d`

✅ Forge suite `framework_upgrade` success on `fcd2dedf6ca61a23f4979e944c629dcdbdae5dca` ==> `957ee8f25fe21c155227ecde11b746eef469e43d`

✅ Forge suite `compat` success on `fcd2dedf6ca61a23f4979e944c629dcdbdae5dca` ==> `957ee8f25fe21c155227ecde11b746eef469e43d`