Node debug build panics with debug_assert in network/transactions/fetcher

[cakevm]

Describe the bug

Using the latest reth (tag=1.1.0) in a exex context dies because of a debug_assert. See log for details.

Steps to reproduce

Can not tell. It happens two times after 5 min and 25 min.

Node logs

2024-10-11T13:01:14.171855Z  INFO Canonical chain committed number=20942541 hash=0xa58e0c8cbe8dbc644f4cd43cc69bdadf1d9e0ebd3f7bec6973955b3da3d9dbf0 elapsed=371.521573ms
2024-10-11T13:01:14.172145Z  INFO Pruner started tip_block_number=20942540
2024-10-11T13:01:14.172292Z  INFO Pruner finished tip_block_number=20942540 elapsed=132.719µs stats=[]
2024-10-11T13:01:25.287347Z  INFO Block added to canonical chain number=20942542 hash=0x429dcfebbcac97e7660d36a4deb291e015c5e853fa2d7891e033e1972ad922e9 peers=7 txs=190 gas=13.37 Mgas gas_throughput=18.70 Mgas/second full=44.6% base_fee=30.99gwei blobs=1 excess_blobs=6 elapsed=714.895292ms
2024-10-11T13:01:25.603900Z  INFO Canonical chain committed number=20942542 hash=0x429dcfebbcac97e7660d36a4deb291e015c5e853fa2d7891e033e1972ad922e9 elapsed=240.529852ms
thread 'tokio-runtime-worker' panicked at /home/cake/.cargo/git/checkouts/reth-36d3ea1d1152b20c/1ba631b/crates/net/network/src/transactions/fetcher.rs:915:9:
`0x8204422ed64281d05f750401e09478c0ae9b9abb7d3ce7bb3b28503942e7db56dd60d993ecce7dfeacf14ea04b3fcf1f8cfeb937a289a549a11e8d70c1c97a99` has been removed from `@active_peers` before inflight request(s) resolved, broken invariant `@active_peers` and `@inflight_requests`, `%peer_id`: 0x8204422ed64281d05f750401e09478c0ae9b9abb7d3ce7bb3b28503942e7db56dd60d993ecce7dfeacf14ea04b3fcf1f8cfeb937a289a549a11e8d70c1c97a99, `@hashes_fetch_inflight_and_pending_fetch` for `%requested_hashes`: [(0x2d96de998217dae387ddf1a969bbf98a44dbe57091f648850d04480a0a7b2a5e, Some((0, None)))]
stack backtrace:
   0: rust_begin_unwind
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:665:5
   1: core::panicking::panic_fmt
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panicking.rs:74:14
   2: reth_network::transactions::fetcher::TransactionFetcher::on_resolved_get_pooled_transactions_request_fut
             at /home/cake/.cargo/git/checkouts/reth-36d3ea1d1152b20c/1ba631b/crates/net/network/src/transactions/fetcher.rs:915:9
   3: <reth_network::transactions::fetcher::TransactionFetcher as futures_core::stream::Stream>::poll_next
             at /home/cake/.cargo/git/checkouts/reth-36d3ea1d1152b20c/1ba631b/crates/net/network/src/transactions/fetcher.rs:1056:37
   4: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/stream/mod.rs:1638:9
   5: <reth_network::transactions::TransactionsManager<Pool> as core::future::future::Future>::poll
             at /home/cake/.cargo/git/checkouts/reth-36d3ea1d1152b20c/1ba631b/crates/net/network/src/transactions/mod.rs:1312:13
   6: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::future::future::Future>::poll
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panic/unwind_safe.rs:297:9
   7: <futures_util::future::future::catch_unwind::CatchUnwind<Fut> as core::future::future::Future>::poll::{{closure}}
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/future/catch_unwind.rs:36:42
   8: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panic/unwind_safe.rs:272:9
   9: std::panicking::try::do_call
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:557:40
  10: __rust_try
  11: std::panicking::try
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:521:19
  12: std::panic::catch_unwind
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs:350:14
  13: <futures_util::future::future::catch_unwind::CatchUnwind<Fut> as core::future::future::Future>::poll
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/future/catch_unwind.rs:36:9
  14: <F as futures_core::future::TryFuture>::try_poll
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
  15: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_future/into_future.rs:34:9
  16: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/future/map.rs:55:37
  17: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
  18: <futures_util::future::try_future::MapErr<Fut,F> as core::future::future::Future>::poll
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
  19: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-futures-0.2.5/src/lib.rs:283:9
  20: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/future/future.rs:123:9
  21: futures_util::future::future::FutureExt::poll_unpin
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/future/mod.rs:558:9
  22: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/select.rs:118:35
  23: reth_tasks::TaskExecutor::spawn_critical_as::{{closure}}
             at /home/cake/.cargo/git/checkouts/reth-36d3ea1d1152b20c/1ba631b/crates/tasks/src/lib.rs:417:47
  24: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/future/future.rs:123:9
  25: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/core.rs:331:17
  26: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/loom/std/unsafe_cell.rs:16:9
  27: tokio::runtime::task::core::Core<T,S>::poll
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/core.rs:320:13
  28: tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:500:19
  29: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panic/unwind_safe.rs:272:9
  30: std::panicking::try::do_call
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:557:40
  31: __rust_try
  32: std::panicking::try
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:521:19
  33: std::panic::catch_unwind
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs:350:14
  34: tokio::runtime::task::harness::poll_future
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:488:18
  35: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:209:27
  36: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:154:15
  37: tokio::runtime::task::raw::poll
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/raw.rs:271:5
  38: tokio::runtime::task::raw::RawTask::poll
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/raw.rs:201:18
  39: tokio::runtime::task::LocalNotified<S>::run
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/mod.rs:436:9
  40: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:598:13
  41: tokio::runtime::coop::with_budget
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
  42: tokio::runtime::coop::budget
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
  43: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:597:9
  44: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:560:24
  45: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:513:21
  46: tokio::runtime::context::scoped::Scoped<T>::set
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/scoped.rs:40:9
  47: tokio::runtime::context::set_scheduler::{{closure}}
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context.rs:180:26
  48: std::thread::local::LocalKey<T>::try_with
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/local.rs:283:12
  49: std::thread::local::LocalKey<T>::with
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/local.rs:260:9
  50: tokio::runtime::context::set_scheduler
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context.rs:180:9
  51: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:508:9
  52: tokio::runtime::context::runtime::enter_runtime
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  53: tokio::runtime::scheduler::multi_thread::worker::run
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:500:5
  54: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:466:45
  55: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/blocking/task.rs:42:21
  56: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/core.rs:331:17
  57: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/loom/std/unsafe_cell.rs:16:9
  58: tokio::runtime::task::core::Core<T,S>::poll
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/core.rs:320:13
  59: tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:500:19
  60: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panic/unwind_safe.rs:272:9
  61: std::panicking::try::do_call
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:557:40
  62: __rust_try
  63: std::panicking::try
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:521:19
  64: std::panic::catch_unwind
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs:350:14
  65: tokio::runtime::task::harness::poll_future
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:488:18
  66: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:209:27
  67: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:154:15
  68: tokio::runtime::task::raw::poll
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/raw.rs:271:5
  69: tokio::runtime::task::raw::RawTask::poll
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/raw.rs:201:18
  70: tokio::runtime::task::UnownedTask<S>::run
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/mod.rs:473:9
  71: tokio::runtime::blocking::pool::Task::run
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/blocking/pool.rs:160:9
  72: tokio::runtime::blocking::pool::Inner::run
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/blocking/pool.rs:518:17
  73: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /home/cake/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/blocking/pool.rs:476:13
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2024-10-11T13:01:37.442694Z ERROR Critical task `p2p txpool` panicked: ``0x8204422ed64281d05f750401e09478c0ae9b9abb7d3ce7bb3b28503942e7db56dd60d993ecce7dfeacf14ea04b3fcf1f8cfeb937a289a549a11e8d70c1c97a99` has been removed from `@active_peers` before inflight request(s) resolved, broken invariant `@active_peers` and `@inflight_requests`, `%peer_id`: 0x8204422ed64281d05f750401e09478c0ae9b9abb7d3ce7bb3b28503942e7db56dd60d993ecce7dfeacf14ea04b3fcf1f8cfeb937a289a549a11e8d70c1c97a99, `@hashes_fetch_inflight_and_pending_fetch` for `%requested_hashes`: [(0x2d96de998217dae387ddf1a969bbf98a44dbe57091f648850d04480a0a7b2a5e, Some((0, None)))]`
2024-10-11T13:01:37.442902Z ERROR shutting down due to error
Error: Critical task `p2p txpool` panicked: ``0x8204422ed64281d05f750401e09478c0ae9b9abb7d3ce7bb3b28503942e7db56dd60d993ecce7dfeacf14ea04b3fcf1f8cfeb937a289a549a11e8d70c1c97a99` has been removed from `@active_peers` before inflight request(s) resolved, broken invariant `@active_peers` and `@inflight_requests`, `%peer_id`: 0x8204422ed64281d05f750401e09478c0ae9b9abb7d3ce7bb3b28503942e7db56dd60d993ecce7dfeacf14ea04b3fcf1f8cfeb937a289a549a11e8d70c1c97a99, `@hashes_fetch_inflight_and_pending_fetch` for `%requested_hashes`: [(0x2d96de998217dae387ddf1a969bbf98a44dbe57091f648850d04480a0a7b2a5e, Some((0, None)))]`


Another time:
2024-10-11T12:46:57.716374Z ERROR Critical task `p2p txpool` panicked: ``0x6223fbd7020a17b9b2f89bd66df1d5d16557fcab2cf04244da0f6a4fb04ea85bc263710d2268461cbf53731e16d218eb13742a269e6129dd89980abd06342c26` has been removed from `@active_peers` before inflight request(s) resolved, broken invariant `@active_peers` and `@inflight_requests`, `%peer_id`: 0x6223fbd7020a17b9b2f89bd66df1d5d16557fcab2cf04244da0f6a4fb04ea85bc263710d2268461cbf53731e16d218eb13742a269e6129dd89980abd06342c26, `@hashes_fetch_inflight_and_pending_fetch` for `%requested_hashes`: [(0x51feb14f2fccd3c8d41cc5345abd263de729f5924b30ae040443c9972b628959, Some((0, None)))]`
2024-10-11T12:46:57.716782Z ERROR shutting down due to error
Error: Critical task `p2p txpool` panicked: ``0x6223fbd7020a17b9b2f89bd66df1d5d16557fcab2cf04244da0f6a4fb04ea85bc263710d2268461cbf53731e16d218eb13742a269e6129dd89980abd06342c26` has been removed from `@active_peers` before inflight request(s) resolved, broken invariant `@active_peers` and `@inflight_requests`, `%peer_id`: 0x6223fbd7020a17b9b2f89bd66df1d5d16557fcab2cf04244da0f6a4fb04ea85bc263710d2268461cbf53731e16d218eb13742a269e6129dd89980abd06342c26, `@hashes_fetch_inflight_and_pending_fetch` for `%requested_hashes`: [(0x51feb14f2fccd3c8d41cc5345abd263de729f5924b30ae040443c9972b628959, Some((0, None)))]`

Platform(s)

Linux (x86)

What version/commit are you on?

debug build for tag=1.1.0

What database version are you on?

Current database version: 2
Local database version: 2

Which chain / network are you on?

eth mainnet

What type of node are you running?

Archive (default)

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• I agree to follow the Code of Conduct

[emhane]

this assertion will no longer hold since https://github.com/paradigmxyz/reth/pull/11598/files, as removing the peer when session closes will remove the peer before req returns or fails. will remove the debug assertion.