[Perf] Greedily verify Executions before block generation

[vicsn]

Motivation

When processing hundreds of transmissions during block generation, transaction verification is a large bottleneck, taking up 90% of compute. By verifying transactions ahead of time, verification will be cached in partially_verified_transactions, moving compute out of the hot path.

Some useful details:

• The current PR does not mean to change any consensus logic. We still sign proposals without awaiting succesful verification of transmissions, just like before.
• Solutions are not verified because there are only 4 per block, and ARC-0043: Extending the Puzzle to a Full SNARK ARCs#77 might change the game entirely.
• Deployments are not verified, because in the honest case there won't be a lot of them, and they consume a lot of memory so would require a lot more complexity (a queue and thread steadily popping from it)

Test Plan

• Deployed a network succesfully.

[raychu86]

Relevant PR - #3098.

We initially did not verify transactions from other validator's proposals to prevent a block halting issue. In your case, I believe this is safe because it doesn't interfere with the consensus rules.

However, it would be interesting to see the implications of this kind of change. I worry that this may open up avenues for malicious nodes to slow down other validators by spamming them with transactions.

[vicsn]

I worry that this may open up avenues for malicious nodes to slow down other validators by spamming them with transactions.

Valid concern, but note that:

• this only impacts execution verification, which is light (takes a couple ms).
• validators currently are only at 50% average CPU saturation. It is theoretically possible that the early tx verification eats up resources from, say, certificate deserialization. But in practice my E2E stress test shows a large speedup. Happy to share grafana links etc.
• assuming the cache works as intended, we are not increasing compute, merely moving it (you used the term frontloading before).

[raychu86]

LGTM as long as we feel confident that this shouldn't introduce any new attack vectors!