[e3] domains.GetDiffset(...): not found

[wmitsuda]

yesterday's main, commit 39b62b4fda39510251f6ceac89d81f5a94ff8f50

holesky logs, but I had it on mainnet as well.

TLDR; node entered a stuck state after an unwind.

INFO[10-10|20:03:02.554] head updated                             head=0x28bfa98929aa21f775b7b7152390e42649abf004e89de8e138ff4275c20bd0ae hash
=0x28bfa98929aa21f775b7b7152390e42649abf004e89de8e138ff4275c20bd0ae number=2508019 execution=234.13725ms mgas/s=44.74 average mgas/s=151.14 co
mmit=309.708µs alloc=10.7GB sys=23.7GB
INFO[10-10|20:03:03.533] Timings: Post-Forkchoice (slower than 50ms) Prune Execution=445ms
INFO[10-10|20:03:42.063] [4/6 Execution] Done Commit every block  blk=2508022 blks=3 blk/s=3.5 txs=204 tx/s=238 gas/s=44.53M buf=5.6MB/512.0MB
 stepsInDB=0.00 step=62.1 alloc=12.2GB sys=23.7GB
INFO[10-10|20:03:42.154] cl reorg                                 new_head_slot=2724918 fork_slot=2724915 old_canonical=0x7422a67c93f6f832c267
59887b0e953481d80dbf33bfe434cc1696e85cd506b2 new_canonical=0x6599fcd9116803cd7b749eaac84f54f029e5f93db1c0d91cfb4cbb3488f63a16
INFO[10-10|20:03:42.525] head updated                             head=0x90845967329bd00242f307859ff4d08ff9fdc3c6351767b9aa147a3585abd476 hash
=0x90845967329bd00242f307859ff4d08ff9fdc3c6351767b9aa147a3585abd476 commit=591.416µs alloc=12.3GB sys=23.7GB
INFO[10-10|20:03:43.304] Timings: Post-Forkchoice (slower than 50ms) Prune Execution=344ms
INFO[10-10|20:03:51.609] head updated                             head=0xcb60133835bc04a2ba843c32d67a248e870f009856e7b847d1fae8972654385d hash
=0xcb60133835bc04a2ba843c32d67a248e870f009856e7b847d1fae8972654385d number=2508023 execution=194.758291ms mgas/s=38.08 average mgas/s=150.63 c
ommit=115.25µs alloc=13.2GB sys=23.7GB
INFO[10-10|20:03:52.462] Timings: Post-Forkchoice (slower than 50ms) Prune Execution=333ms
WARN[10-10|20:04:05.795] [Gossip] No peers to publish to for topic topic=/eth2/69ae0e99/beacon_attestation_21/ssz_snappy
INFO[10-10|20:04:06.662] head updated                             head=0x257617090aebbba0f70fa62f9b1396fd9a8af444b4c5755399ad95665b814ead hash
=0x257617090aebbba0f70fa62f9b1396fd9a8af444b4c5755399ad95665b814ead number=2508024 execution=542.140542ms mgas/s=51.58 average mgas/s=150.17 c
ommit=331.916µs alloc=14.0GB sys=23.7GB
INFO[10-10|20:04:07.383] Timings: Post-Forkchoice (slower than 50ms) Prune Execution=257ms
INFO[10-10|20:04:11.523] P2P                                      app=caplin peers=8
INFO[10-10|20:04:14.604] head updated                             head=0xace0cece8ae6c87aaa76bc2f6673546b38a18e16054b9f7f3bb9db871dc90a27 hash
=0xace0cece8ae6c87aaa76bc2f6673546b38a18e16054b9f7f3bb9db871dc90a27 number=2508025 execution=232.228042ms mgas/s=46.30 average mgas/s=149.70 c
ommit=113.584µs alloc=14.6GB sys=23.7GB
INFO[10-10|20:04:15.328] Timings: Post-Forkchoice (slower than 50ms) Prune Execution=229ms
WARN[10-10|20:04:15.646] [Gossip] No peers to publish to for topic topic=/eth2/69ae0e99/beacon_attestation_21/ssz_snappy
WARN[10-10|20:04:16.140] [Gossip] No peers to publish to for topic topic=/eth2/69ae0e99/beacon_attestation_36/ssz_snappy
INFO[10-10|20:04:16.827] [snapshots:blocks:retire] Stat           blocks=2.50M indices=2.50M alloc=8.1GB sys=23.7GB
INFO[10-10|20:09:06.891] [mem] memory stats                       alloc=10.3GB sys=23.7GB
WARN[10-10|20:09:06.891] [p2p] No GoodPeers
INFO[10-10|20:09:08.689] [txpool] stat                            pending=327 baseFee=390 queued=1053 alloc=10.4GB sys=23.7GB
INFO[10-10|20:09:40.597] P2P                                      app=caplin peers=3
WARN[10-10|20:18:36.403] [engine] CurrentHeader                   err="ethereumExecutionModule.CurrentHeader: could not begin database tx cont
ext deadline exceeded"
INFO[10-10|20:18:54.969] [3/6 Senders] Started                    from=2508025 to=2508093
INFO[10-10|20:18:55.002] [4/6 Execution] starting                 from=2508026 to=2508093 fromTxNum=97017933 offsetFromBlockBeginning=0 initia
lCycle=false useExternalTx=true
INFO[10-10|20:18:55.918] cl reorg                                 new_head_slot=2724994 fork_slot=2724921 old_canonical=0xa6c8cf18c50d5f52cb8e
ca4b7c9a84d900c33532edf8562b454b1b8dd2332499 new_canonical=0xc14ab11f7855caaeb7d0acde4e55866ccf165b2e0495e49231348681833e981c
INFO[10-10|20:18:58.874] P2P                                      app=caplin peers=1
INFO[10-10|20:27:22.134] [4/6 Execution] Done                     blk=2508093 blks=68 blk/s=2.9 txs=4942 tx/s=212 gas/s=44.11M buf=83.3MB/512.
0MB stepsInDB=0.00 step=62.1 alloc=13.9GB sys=23.9GB
INFO[10-10|20:27:24.613] head updated                             head=0x4b10d5de9e195bcec472197c21d6838b8539c8bcf836aab46c1e5d54a8b855d0 hash
=0x4b10d5de9e195bcec472197c21d6838b8539c8bcf836aab46c1e5d54a8b855d0 commit=3.345625ms alloc=14.0GB sys=23.9GB
INFO[10-10|20:27:25.556] Timings: Post-Forkchoice (slower than 50ms) Prune Execution=608ms
INFO[10-10|20:27:25.720] [4/6 Execution] Unwind Execution         from=2508093 to=2508092
INFO[10-10|20:27:47.670] [4/6 Execution] Unwind Execution         from=2508093 to=2508092
INFO[10-10|20:28:12.749] P2P                                      app=caplin peers=2
INFO[10-10|20:28:39.043] [mem] memory stats                       alloc=14.5GB sys=23.9GB
WARN[10-10|20:28:39.043] [p2p] No GoodPeers
INFO[10-10|20:28:40.842] [txpool] stat                            pending=327 baseFee=390 queued=1053 alloc=15.4GB sys=23.9GB
INFO[10-10|20:29:12.749] P2P                                      app=caplin peers=1
INFO[10-10|20:29:21.086] [4/6 Execution] Unwind Execution         from=2508093 to=2508092
INFO[10-10|20:29:33.360] [4/6 Execution] Unwind Execution         from=2508093 to=2508092
INFO[10-10|20:29:59.564] [4/6 Execution] Unwind Execution         from=2508093 to=2508092
INFO[10-10|20:30:12.748] P2P                                      app=caplin peers=7
INFO[10-10|20:30:25.035] [4/6 Execution] Unwind Execution         from=2508093 to=2508092
INFO[10-10|20:36:52.414] [Caplin] Forward Sync                    app=caplin stage=ForwardSync from=2724928 to=2725084
INFO[10-10|20:37:00.874] P2P                                      app=caplin peers=0
INFO[10-10|20:37:27.168] [mem] memory stats                       alloc=9.0GB sys=23.9GB
WARN[10-10|20:37:27.168] [p2p] No GoodPeers
INFO[10-10|20:37:28.966] [txpool] stat                            pending=327 baseFee=390 queued=1053 alloc=9.1GB sys=23.9GB
INFO[10-10|20:38:00.874] P2P                                      app=caplin peers=0
INFO[10-10|20:38:13.817] [Caplin] Forward Sync                    app=caplin stage=ForwardSync progress=2724941 distance-from-chain-tip=28m36s
 estimated-time-remaining=5m30s
INFO[10-10|20:38:25.643] [Caplin] Forward Sync                    app=caplin stage=ForwardSync progress=2724992 distance-from-chain-tip=18m24s
 estimated-time-remaining=54s
INFO[10-10|20:46:04.045] P2P                                      app=caplin peers=0
INFO[10-10|20:46:13.420] [Caplin] Forward Sync                    app=caplin stage=ForwardSync progress=2725030 distance-from-chain-tip=10m48s
 estimated-time-remaining=42s
INFO[10-10|20:46:32.869] [Caplin] Forward Sync                    app=caplin stage=ForwardSync progress=2725068 distance-from-chain-tip=3m12s
estimated-time-remaining=12s
INFO[10-10|20:47:04.052] P2P                                      app=caplin peers=6
INFO[10-10|20:47:27.508] [4/6 Execution] Unwind Execution         from=2508093 to=2508092
WARN[10-10|20:47:27.509] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] domains.GetDiffset(2508093, 0x4b10d5de9e195bcec472197c21d6838b8539c8bcf836aab46c1e5d54a8b855d0): not found"
INFO[10-10|20:47:30.346] [mem] memory stats                       alloc=11.5GB sys=26.1GB
WARN[10-10|20:47:30.346] [p2p] No GoodPeers
INFO[10-10|20:47:32.145] [txpool] stat                            pending=463 baseFee=441 queued=1091 alloc=11.5GB sys=26.1GB
INFO[10-10|20:47:38.182] [4/6 Execution] Unwind Execution         from=2508093 to=2508092
WARN[10-10|20:47:38.182] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] domains.GetDiffset(2508093, 0x4b10d5de9e195bcec472197c21d6838b8539c8bcf836aab46c1e5d54a8b855d0): not found"
WARN[10-10|20:47:46.870] [Gossip] No peers to publish to for topic topic=/eth2/69ae0e99/beacon_attestation_21/ssz_snappy
INFO[10-10|20:47:50.798] [4/6 Execution] Unwind Execution         from=2508093 to=2508092
WARN[10-10|20:47:50.798] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] domains.GetDiffset(2508093, 0x4b10d5de9e195bcec472197c21d6838b8539c8bcf836aab46c1e5d54a8b855d0): not found"
INFO[10-10|20:47:30.346] [mem] memory stats                       alloc=11.5GB sys=26.1GB
WARN[10-10|20:47:30.346] [p2p] No GoodPeers
INFO[10-10|20:47:32.145] [txpool] stat                            pending=463 baseFee=441 queued=1091 alloc=11.5GB sys=26.1GB
INFO[10-10|20:47:38.182] [4/6 Execution] Unwind Execution         from=2508093 to=2508092
WARN[10-10|20:47:38.182] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] domains.GetDiffset(2508093, 0x4b10d5de9e195bcec472197c21d6838b8539c8bcf836aab46c1e5d54a8b855d0): not found"
WARN[10-10|20:47:46.870] [Gossip] No peers to publish to for topic topic=/eth2/69ae0e99/beacon_attestation_21/ssz_snappy
INFO[10-10|20:47:50.798] [4/6 Execution] Unwind Execution         from=2508093 to=2508092
WARN[10-10|20:47:50.798] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] domains.GetDiffset(2508093, 0x4b10d5de9e195bcec472197c21d6838b8539c8bcf836aab46c1e5d54a8b855d0): not found"
INFO[10-10|20:48:01.916] [4/6 Execution] Unwind Execution         from=2508093 to=2508092
WARN[10-10|20:48:01.917] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] domains.GetDiffset(2508093, 0x4b10d5de9e195bcec472197c21d6838b8539c8bcf836aab46c1e5d54a8b855d0): not found"
INFO[10-10|20:48:04.052] P2P                                      app=caplin peers=4
INFO[10-10|20:48:14.909] [4/6 Execution] Unwind Execution         from=2508093 to=2508092
WARN[10-10|20:48:14.910] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] domains.GetDiffset(2508093, 0x4b10d5de9e195bcec472197c21d6838b8539c8bcf836aab46c1e5d54a8b855d0): not found"
INFO[10-10|20:48:27.723] [4/6 Execution] Unwind Execution         from=2508093 to=2508092
WARN[10-10|20:48:27.723] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] domains.GetDiffset(2508093, 0x4b10d5de9e195bcec472197c21d6838b8539c8bcf836aab46c1e5d54a8b855d0): not found"
INFO[10-10|20:48:42.624] [4/6 Execution] Unwind Execution         from=2508093 to=2508092
WARN[10-10|20:48:42.625] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] domains.GetDiffset(2508093, 0x4b10d5de9e195bcec472197c21d6838b8539c8bcf836aab46c1e5d54a8b855d0): not found"