Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test #1858

Draft
wants to merge 59 commits into
base: main
Choose a base branch
from
Draft

test #1858

wants to merge 59 commits into from

Commits on Nov 13, 2024

  1. revamped TokenBucket for rate-limiter: 

    - parametrized by TimeProvider instead of taking `now: Instant` as argument for `rate_limit`
- simplified rate_limit method
- re-aligned tests of TokenBucket
- TokenBucket can be set to rate_per_second = 0 - no data will be read
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    9e967ae View commit details
    Browse the repository at this point in the history

  2. rate_limiter: 

    - seperate RateLimiter implementations for [`tokio::io::AsyncRead`] and [`futures::AsyncRead`]
- using [`std::sync::Mutex`] for TokenBucket in RateLimiter - it should allow for `global` rate-limit, not just per connection
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    f896b4c View commit details
    Browse the repository at this point in the history

  3. RateLimitedAsyncRead and Write moved to rate_limiter crate

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    ab74c45 View commit details
    Browse the repository at this point in the history

  4. custom implementation of Transport with rate-limiting (only read) for… 

    … substrate's [`sc_client::network::NetworkWorker`]
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    0b17e45 View commit details
    Browse the repository at this point in the history

  5. intergrated rate-limiting for substrate [`sc_client::network::Network… 

    …Worker`]
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    03142e2 View commit details
    Browse the repository at this point in the history

  6. increased limits for the rate-limiter, both for alephbft and substrate

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    2eb7b72 View commit details
    Browse the repository at this point in the history

  7. - s/substrate_bit_rate/substrate_network_bit_rate 

    - s/alephbft_bit_rate/alephbft_network_bit_rate
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    fc806fc View commit details
    Browse the repository at this point in the history

  8. renamed local variable in build/transport.rs - it no longer referen… 

    …ces `TokenBucket`
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    0a41d22 View commit details
    Browse the repository at this point in the history

  9. removed unecessary pub-use to [TokenBucket] in rate_limiter/lib.rs

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    eb6dd52 View commit details
    Browse the repository at this point in the history

  10. removed unecessary pub in pub mod transport; in [`finality-aleph/… 

    …src/network/build/mod.rs`]
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    b5937b8 View commit details
    Browse the repository at this point in the history

  11. removed newline in [finality-aleph/src/network/build/mod.rs]

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    d0b0ce5 View commit details
    Browse the repository at this point in the history

  12. added impl Default for DefaultTimeProvider in [token_bucket.rs]

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    d9fdce1 View commit details
    Browse the repository at this point in the history

  13. removed unecessary .into() conversion of [SleepingRateLimiter] in… 

    … [`finality-aleph/src/network/build/transport.rs`]
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    6ce1663 View commit details
    Browse the repository at this point in the history

  14. - build_transport moved to [`finality-aleph/src/network/build/trans… 

    …port.rs`]

- removed references to [`libp2p`] from [`finality-aleph/src/network/build/mod.rs`]
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    019f9a9 View commit details
    Browse the repository at this point in the history

  15. Default macro for DefaultTimeProvider - review changes

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    0525c17 View commit details
    Browse the repository at this point in the history

  16. using parking_lot::Mutex in rate_limiter instead of std::sync::Mutex

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    2dd96c2 View commit details
    Browse the repository at this point in the history

  17. simplified TokenBucket algorithm: using only requested field instea… 

    …d of `requeste` and `available`. It should be also easier now to transform it into wait-free version.
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    bbe8e8a View commit details
    Browse the repository at this point in the history

  18. TokenBucket per connection instead of a shared one

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    e69894b View commit details
    Browse the repository at this point in the history

  19. using new API of substrate::NetworkWorker::new_with_custom_transport

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    3fe9404 View commit details
    Browse the repository at this point in the history

  20. TokenBucket return None when rate_limit is 0 instead of some arbitrar… 

    …y big value
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    7e680da View commit details
    Browse the repository at this point in the history

  21. default rate-limit for the sync network is now 1 MiB per connection… 

    … - so far only tested locally using 7 instances of aleph-node.
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    563de02 View commit details
    Browse the repository at this point in the history

  22. fmt for rate_limiter.rs

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    b049419 View commit details
    Browse the repository at this point in the history

  23. fixed unit-tests for token-bucket after recent changes of its api

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    f88d6ac View commit details
    Browse the repository at this point in the history

  24. rust-fmt after changes in rate-limiter

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    4fa4dc0 View commit details
    Browse the repository at this point in the history

  25. added type alias for `Box<dyn sc_network::config::NotificationService… 

    …>` in `network/substrate.rs`
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    33eb851 View commit details
    Browse the repository at this point in the history

  26. s/substrate_network_bit_rate_per_connection/bit_rate_per_connection i… 

    …n `finality-aleph/src/network/build/mod.rs`
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    f791005 View commit details
    Browse the repository at this point in the history

  27. better readability for TokenBucket.rs

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    e9cf1de View commit details
    Browse the repository at this point in the history

  28. better readability in TokenBucket.rs ++

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    a2f1eff View commit details
    Browse the repository at this point in the history

  29. rust-fmt in bin/node/src/service.rs

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    6fdecaa View commit details
    Browse the repository at this point in the history

  30. using enum Deadline {Never, Instant} instead of `Option<Option<Inst… 

    …ant>>` in TokenBucket
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    a379641 View commit details
    Browse the repository at this point in the history

  31. rust-fmt in tests for TokenBucket

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    dbeeca2 View commit details
    Browse the repository at this point in the history

  32. - simpler version of the TokenBucket (rate-limiter) 

    - added ShareTokenBucket, AsyncTokenBucket and ShredBandwidthManager - used for sharing bandwidth between multiple connections
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    f0dc89c View commit details
    Browse the repository at this point in the history

  33. added bunch of unit-tests for TokenBucket and SharedTokenBucket

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    374a45f View commit details
    Browse the repository at this point in the history

  34. - integration of the new SharedTokenBucket with both Substrate-based … 

    …and Socket-based networks (sync + alephbft)

- new default values for rate-limiters (sync + alephbft). 768Kib for alephbft and 5Mib for sync. These values were tested using t3a.xlarge aws instances - nodes were not able to handle more.
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    ba07522 View commit details
    Browse the repository at this point in the history

  35. added ALEPHBFT_NETWORK_BIT_RATE SUBSTRATE_NETWORK_BIT_RATE params to … 

    …docker/docker_entrypoint.sh to allow configuration of rate-limiting for both sync and alephbft networks
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    ba90a89 View commit details
    Browse the repository at this point in the history

  36. slightly refactored token_bucket.rs

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    6c7404f View commit details
    Browse the repository at this point in the history

  37. fixed and updated unit tests in token_bucket.rs

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    1f5b787 View commit details
    Browse the repository at this point in the history

  38. rust-fmt for token_bucket and related

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    b11b80c View commit details
    Browse the repository at this point in the history

  39. added more docs in token_bucket.rs

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    fc27ed5 View commit details
    Browse the repository at this point in the history

  40. more docs for token_bucket.rs

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    fa131da View commit details
    Browse the repository at this point in the history

  41. bumped version of rust-toolchain in nix/versions.nix

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    c8db058 View commit details
    Browse the repository at this point in the history

  42. improved docs in token_bucket.rs

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    e2e0bb3 View commit details
    Browse the repository at this point in the history

  43. removed no longer necessary types in rate-limiter crate - helpers rel… 

    …ated with impls of AsyncRead, etc.
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    0f26764 View commit details
    Browse the repository at this point in the history

  44. Added verbose share() method to SharedRateLimiter (and to other t… 

    …ypes in its hierarchy). It is more readable this way - previous version was using `Clone`.
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    8fd9ba9 View commit details
    Browse the repository at this point in the history

  45. rust-fmt for token_bucket.rs

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    d12abfd View commit details
    Browse the repository at this point in the history

  46. added flooding routines for both alephbft and sync networks.

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    ecce82b View commit details
    Browse the repository at this point in the history

  47. added FINALIZATION_WAIT param for the finalization e2e-test

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    e9b5076 View commit details
    Browse the repository at this point in the history

  48. new docker-compose definition for the network flooding tests

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    2e1f074 View commit details
    Browse the repository at this point in the history

  49. simplified docker-compose file for the network flooding test

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    fbbc564 View commit details
    Browse the repository at this point in the history

  50. building e2e-tests with --locked file - helps to keep its Cargo.loc… 

    …k file valid
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    19a2759 View commit details
    Browse the repository at this point in the history

  51. added the network-flooding tests to nightly e2e-pipeline

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    3d12e7f View commit details
    Browse the repository at this point in the history

  52. Revert "simplified docker-compose file for the network flooding test" 

    This reverts commit 74fc9b0.
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    f14095d View commit details
    Browse the repository at this point in the history

  53. explit env vars TIMEOUT_MINUTES="5m" FINALIZATION_WAIT=1 <script> for… 

    … before and after finalization tests
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    a1727d6 View commit details
    Browse the repository at this point in the history

  54. e2e-tests: finalization test was await expected finalized block + 1… 

    … instead of just `expected finalized block`
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    db3d50d View commit details
    Browse the repository at this point in the history

  55. halved network bit rates for the docker network flooding test (e2e ni… 

    …ghtly test)
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    c8d4130 View commit details
    Browse the repository at this point in the history

  56. halved (1Mib) rate-limit for the sync-network for the flooding test

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    cf35e5d View commit details
    Browse the repository at this point in the history

  57. halved rate-limit for the sync-network in flooding e2e-test - docker/… 

    …docker-compose_network_flooding_test.yml
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    6e833d1 View commit details
    Browse the repository at this point in the history

  58. halved rate-limit for alephbft network for the flooding e2e test - no… 

    …w its 393216 b/s
    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    182d692 View commit details
    Browse the repository at this point in the history

  59. WIP REMOVE commented out other nightly tests except flooding test

    @fixxxedpoint
    fixxxedpoint committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    fbe540c View commit details
    Browse the repository at this point in the history