A0-3237: sync tests (#1419)

# Description

Series of tests based on the synthetic-network framework (a docker tool
for simulating "real-life" network conditions). All introduced tests
attempts to modify nodes connectivity (e.g. creating connected subsets
of nodes) and verify if both block production and finalization are still
possible.

Description of how to run these tests is in `e2e-tests/README.md` in
section #Running e2e-tests that depend on synthetic-network, e.g.
`sync`, `high_latency` .

## Type of change

- New feature (non-breaking change which adds functionality)

# Checklist:

<!-- delete when not applicable to your PR -->

- I have added tests
- I have made neccessary updates to the Infrastructure

---------

Co-authored-by: Marcin <[email protected]>
Co-authored-by: kostekIV <[email protected]>
Co-authored-by: Mikolaj Gasior <[email protected]>
Co-authored-by: Grzegorz Gawryał <[email protected]>
Co-authored-by: timorl <[email protected]>
Co-authored-by: timorl <[email protected]>
Co-authored-by: timorl <[email protected]>

.github/scripts/run_e2e_test.sh

@@ -57,9 +57,12 @@ function set_randomized_test_params {
   NON_RESERVED_SEATS=$((${VALIDATOR_COUNT} - ${RESERVED_SEATS}))
 }
 
+NODE_URL=${NODE_URL:-"ws://127.0.0.1:9943"}
+NETWORK=${NETWORK:-"container:Node0"}
+
 ARGS=(
-  --network "container:Node0"
-  -e NODE_URL="ws://127.0.0.1:9943"
+  --network "${NETWORK}"
+  -e NODE_URL="${NODE_URL}"
   -e RUST_LOG=info
   -e VALIDATOR_COUNT
 )

.github/workflows/nightly-normal-session-e2e-tests.yml

@@ -48,6 +48,148 @@ jobs:
           compose-file: docker/docker-compose.synthetic-network.yml
         timeout-minutes: 40
 
+  run-e2e-sync-test-one_node_catching_up_and_then_becoming_necessary_for_consensus:
+    needs: [build-production-node-and-e2e-client-image]
+    name: Sync test using synthetic-network - one node catching up and then
+          becoming necessary for consensus
+    runs-on: ubuntu-20.04
+    steps:
+      - name: Checkout source code
+        uses: actions/checkout@v3
+
+      - name: Run e2e test
+        uses: ./.github/actions/run-e2e-test
+        env:
+          NETWORK: "synthetic-network"
+          NODE_URL: "ws://Node0:9943"
+          VALIDATOR_COUNT: 7
+        with:
+          test-case: test::sync::one_node_catching_up_and_then_becoming_necessary_for_consensus
+          image-path: aleph-release-synthetic-docker
+          node-image: aleph-node:syntheticnet
+          compose-file: docker/docker-compose.synthetic-network_sync-tests.yml
+          node-count: 7
+          min-validator-count: 7
+        timeout-minutes: 35
+
+  run-e2e-sync-test-one_node_catching_up:
+    needs: [build-production-node-and-e2e-client-image]
+    name: Sync test using synthetic-network - one node catching up
+    runs-on: ubuntu-20.04
+    steps:
+      - name: Checkout source code
+        uses: actions/checkout@v3
+
+      - name: Run e2e test
+        uses: ./.github/actions/run-e2e-test
+        env:
+          NETWORK: "synthetic-network"
+          NODE_URL: "ws://Node0:9943"
+          VALIDATOR_COUNT: 7
+        with:
+          test-case: test::sync::one_node_catching_up
+          image-path: aleph-release-synthetic-docker
+          node-image: aleph-node:syntheticnet
+          compose-file: docker/docker-compose.synthetic-network_sync-tests.yml
+          node-count: 7
+          min-validator-count: 7
+        timeout-minutes: 35
+
+  run-e2e-sync-test-into_two_groups_and_one_quorum_and_switch_quorum_between_them:
+    needs: [build-production-node-and-e2e-client-image]
+    name: Sync test using synthetic-network - into two groups and one quorum
+          and switch quorum between them
+    runs-on: ubuntu-20.04
+    steps:
+      - name: Checkout source code
+        uses: actions/checkout@v3
+
+      - name: Run e2e test
+        uses: ./.github/actions/run-e2e-test
+        env:
+          NETWORK: "synthetic-network"
+          NODE_URL: "ws://Node0:9943"
+          VALIDATOR_COUNT: 7
+        with:
+          test-case: test::sync::into_two_groups_and_one_quorum_and_switch_quorum_between_them
+          image-path: aleph-release-synthetic-docker
+          node-image: aleph-node:syntheticnet
+          compose-file: docker/docker-compose.synthetic-network_sync-tests.yml
+          node-count: 7
+          min-validator-count: 7
+        timeout-minutes: 35
+
+  run-e2e-sync-test-into_multiple_groups_of_two:
+    needs: [build-production-node-and-e2e-client-image]
+    name: Sync test using synthetic-network - into multiple groups of two
+    runs-on: ubuntu-20.04
+    steps:
+      - name: Checkout source code
+        uses: actions/checkout@v3
+
+      - name: Run e2e test
+        uses: ./.github/actions/run-e2e-test
+        env:
+          NETWORK: "synthetic-network"
+          NODE_URL: "ws://Node0:9943"
+          VALIDATOR_COUNT: 7
+        with:
+          test-case: test::sync::into_multiple_groups_of_two
+          image-path: aleph-release-synthetic-docker
+          node-image: aleph-node:syntheticnet
+          compose-file: docker/docker-compose.synthetic-network_sync-tests.yml
+          node-count: 7
+          min-validator-count: 7
+        timeout-minutes: 35
+
+  run-e2e-sync-test-into_two_equal_size_groups_with_no_quorum:
+    needs: [build-production-node-and-e2e-client-image]
+    name: Sync test using synthetic-network - into two equal size groups
+          with no quorum
+    runs-on: ubuntu-20.04
+    steps:
+      - name: Checkout source code
+        uses: actions/checkout@v3
+
+      - name: Run e2e test
+        uses: ./.github/actions/run-e2e-test
+        env:
+          NETWORK: "synthetic-network"
+          NODE_URL: "ws://Node0:9943"
+          VALIDATOR_COUNT: 7
+        with:
+          test-case: test::sync::into_two_equal_size_groups_with_no_quorum
+          image-path: aleph-release-synthetic-docker
+          node-image: aleph-node:syntheticnet
+          compose-file: docker/docker-compose.synthetic-network_sync-tests.yml
+          node-count: 7
+          min-validator-count: 7
+        timeout-minutes: 35
+
+  run-e2e-sync-test-into_two_groups_one_with_quorum:
+    needs: [build-production-node-and-e2e-client-image]
+    name: Sync test using synthetic-network - into two groups one with
+          quorum
+    runs-on: ubuntu-20.04
+    steps:
+      - name: Checkout source code
+        uses: actions/checkout@v3
+
+      - name: Run e2e test
+        uses: ./.github/actions/run-e2e-test
+        env:
+          NETWORK: "synthetic-network"
+          NODE_URL: "ws://Node0:9943"
+          VALIDATOR_COUNT: 7
+        with:
+          test-case: test::sync::into_two_groups_one_with_quorum
+          image-path: aleph-release-synthetic-docker
+          node-image: aleph-node:syntheticnet
+          compose-file: docker/docker-compose.synthetic-network_sync-tests.yml
+          node-count: 7
+          min-validator-count: 7
+        timeout-minutes: 35
+
   run-major-sync-test:
     needs: [build-production-node-and-runtime]
     name: Run major sync test

aleph-client/src/connections.rs

@@ -2,7 +2,7 @@ use std::{thread::sleep, time::Duration};
 
 use anyhow::anyhow;
 use codec::Decode;
-use log::info;
+use log::{debug, info};
 use serde::{Deserialize, Serialize};
 use subxt::{
     blocks::ExtrinsicEvents,
@@ -374,10 +374,11 @@ impl Connection {
     /// * `retries` - number of connection attempts
     async fn new_with_retries(address: &str, mut retries: u32) -> Connection {
         loop {
+            debug!(target: "aleph-client", "new_with_retries: address={address} retries_left={retries}");
             let client = SubxtClient::from_url(&address).await;
             match (retries, client) {
                 (_, Ok(client)) => return Connection { client },
-                (0, Err(e)) => panic!("{e:?}"),
+                (0, Err(e)) => panic!("new_with_retries failed for address {address}: {e:?}"),
                 _ => {
                     sleep(Duration::from_secs(Self::RETRY_WAIT_SECS));
                     retries -= 1;

docker/docker-compose.synthetic-network.yml

@@ -11,7 +11,6 @@ services:
       - NET_RAW
       - SYS_PTRACE
     environment:
-      - SYNTHETIC_NETWORK=10.77.0.0/16
       - PUBLIC_VALIDATOR_ADDRESS=Node0:30343
     ports:
       - 3000:80
@@ -28,7 +27,6 @@ services:
       - NET_RAW
       - SYS_PTRACE
     environment:
-      - SYNTHETIC_NETWORK=10.77.0.0/16
       - PUBLIC_VALIDATOR_ADDRESS=Node1:30344
       - BOOT_NODES=/dns4/Node0/tcp/30333/p2p/$BOOTNODE_PEER_ID
     ports:
@@ -46,7 +44,6 @@ services:
       - NET_RAW
       - SYS_PTRACE
     environment:
-      - SYNTHETIC_NETWORK=10.77.0.0/16
       - PUBLIC_VALIDATOR_ADDRESS=Node2:30345
       - BOOT_NODES=/dns4/Node0/tcp/30333/p2p/$BOOTNODE_PEER_ID
     ports:
@@ -64,7 +61,6 @@ services:
       - NET_RAW
       - SYS_PTRACE
     environment:
-      - SYNTHETIC_NETWORK=10.77.0.0/16
       - PUBLIC_VALIDATOR_ADDRESS=Node3:30346
       - BOOT_NODES=/dns4/Node0/tcp/30333/p2p/$BOOTNODE_PEER_ID
     ports:
@@ -82,15 +78,13 @@ services:
       - NET_RAW
       - SYS_PTRACE
     environment:
-      - SYNTHETIC_NETWORK=10.77.0.0/16
       - PUBLIC_VALIDATOR_ADDRESS=Node4:30347
       - BOOT_NODES=/dns4/Node0/tcp/30333/p2p/$BOOTNODE_PEER_ID
     ports:
       - 3004:80
 
 networks:
   synthetic-network:
-    ipam:
-      config:
-        - subnet: 10.77.0.0/16
+    name: synthetic-network
+    driver: bridge
 

docker/docker-compose.synthetic-network_sync-tests.yml

@@ -0,0 +1,83 @@
+services:
+  Node0:
+    extends:
+      file: docker-compose.synthetic-network.yml
+      service: Node0
+    environment:
+      - CUSTOM_ARGS="--max-nonfinalized-blocks=128"
+      - RPC_PORT=9943
+
+  Node1:
+    extends:
+      file: docker-compose.synthetic-network.yml
+      service: Node1
+    environment:
+      - CUSTOM_ARGS="--max-nonfinalized-blocks=128"
+      - RPC_PORT=9943
+
+  Node2:
+    extends:
+      file: docker-compose.synthetic-network.yml
+      service: Node2
+    environment:
+      - CUSTOM_ARGS="--max-nonfinalized-blocks=128"
+      - RPC_PORT=9943
+
+  Node3:
+    extends:
+      file: docker-compose.synthetic-network.yml
+      service: Node3
+    environment:
+      - CUSTOM_ARGS="--max-nonfinalized-blocks=128"
+      - RPC_PORT=9943
+
+  Node4:
+    extends:
+      file: docker-compose.synthetic-network.yml
+      service: Node4
+    environment:
+      - CUSTOM_ARGS="--max-nonfinalized-blocks=128"
+      - RPC_PORT=9943
+
+  Node5:
+    extends:
+      file: docker-compose.base.yml
+      service: Node5
+    image: aleph-node:syntheticnet
+    networks:
+      - synthetic-network
+    cap_add:
+      - NET_ADMIN
+      - NET_RAW
+      - SYS_PTRACE
+    environment:
+      - PUBLIC_VALIDATOR_ADDRESS=Node5:30348
+      - BOOT_NODES=/dns4/Node0/tcp/30333/p2p/$BOOTNODE_PEER_ID
+      - CUSTOM_ARGS="--max-nonfinalized-blocks=128"
+      - RPC_PORT=9943
+    ports:
+      - 3005:80
+
+  Node6:
+    extends:
+      file: docker-compose.base.yml
+      service: Node6
+    image: aleph-node:syntheticnet
+    networks:
+      - synthetic-network
+    cap_add:
+      - NET_ADMIN
+      - NET_RAW
+      - SYS_PTRACE
+    environment:
+      - PUBLIC_VALIDATOR_ADDRESS=Node6:30349
+      - BOOT_NODES=/dns4/Node0/tcp/30333/p2p/$BOOTNODE_PEER_ID
+      - CUSTOM_ARGS="--max-nonfinalized-blocks=128"
+      - RPC_PORT=9943
+    ports:
+      - 3006:80
+
+networks:
+  synthetic-network:
+    name: synthetic-network
+    driver: bridge

docker/docker_entrypoint.sh

@@ -54,7 +54,7 @@ ARGS=(
   --rpc-cors all
   --no-mdns
   --rpc-max-connections "${RPC_MAX_CONNECTIONS}"
-   --unsafe-rpc-external
+  --unsafe-rpc-external
   --enable-log-reloading
   --db-cache "${DB_CACHE}"
   --runtime-cache-size "${RUNTIME_CACHE_SIZE}"

e2e-tests/Cargo.toml

@@ -20,6 +20,7 @@ serde_json = "1.0"
 serial_test = "1.0.0"
 tokio = { version = "1.21.2", features = ["full"] }
 subxt = "0.30.1"
+url = "2.4.0"
 
 sp-core = { git = "https://github.com/Cardinal-Cryptography/substrate.git", branch = "aleph-v1.0.0", default-features = false, features = ["full_crypto"] }
 sp-runtime = { git = "https://github.com/Cardinal-Cryptography/substrate.git", branch = "aleph-v1.0.0", default-features = false }

e2e-tests/README.md

@@ -15,6 +15,38 @@ configuration of the launched nodes, see the documentation for a particular test
 
 Additional options are passed to the tests via env variables. See `src/config.rs` for docs on available options.
 
+## Running e2e-tests that depend on synthetic-network, e.g. `sync`, `high_latency`
+
+These tests require the synthetic-network component to work properly. synthetic-network is a docker
+based application that allows to manipulate network conditions between nodes, e.g. latency,
+bit-rate, etc. You can run these tests by building a custom docker image containing aleph-node with
+synthetic-network and executing e2e-tests within a docker container. Example:
+
+```bash
+# build aleph-node docker-image
+# we assume that aleph-node binary is stored at ./target/release/aleph-node
+aleph-node$ docker build -t aleph-node:latest -f docker/Dockerfile .
+
+# build e2e-tests
+aleph-node$ cd e2e-tests
+e2e-tests$ cargo test --release --no-run --locked
+# copy created binary to e2e-tests/target/release/, built test binary is in the last line of
+# the above command, e.g.
+# cp target/release/deps/aleph_e2e_client-44dc7cbed6112daa target/release/aleph-e2e-client
+e2e-tests$ docker build --tag aleph-e2e-client:latest -f Dockerfile .
+e2e-tests$ cd ..
+
+# run synthetic-network with aleph-node using docker-compose
+# by default, it should build for you a docker-image for synthetic-network
+aleph-node$ NODES_COUNT=7 DOCKER_COMPOSE=./docker/docker-compose.synthetic-network_sync-tests.yml ./scripts/synthetic-network/run_consensus_synthetic-network.sh
+
+# run e2e-tests
+# run tests for the block sync component
+aleph-node$ VALIDATOR_COUNT=7 NETWORK="synthetic-network" NODE_URL="ws://Node0:9944" ./.github/scripts/run_e2e_test.sh -t test::sync -m 7
+# run high-latency tests
+aleph-node$ OUT_LATENCY=500 ./.github/scripts/run_e2e_test.sh -t high_out_latency_for_all -m 5
+```
+
 ## Running on devnet (or other-net)
 
 You can also run the tests on some other network. For example, to run the contract test for the `adder` contract on
@@ -40,7 +72,7 @@ Run a feature net by adding an appropriate label to a pull request, ie `trigger:
 , then after its started run
 
 ```bash
-e2e-tests$ RUST_LOG=info NODE_URL=wss://ws-fe-a0-29025887979136.dev.azero.dev:443 cargo test --release finalization::finalization -- --nocapture 
+e2e-tests$ RUST_LOG=info NODE_URL=wss://ws-fe-a0-29025887979136.dev.azero.dev:443 cargo test --release finalization::finalization -- --nocapture
 ```
 
 where you can find your feature env address in https://github.com/Cardinal-Cryptography/aleph-node/deployments