increase forge node startup timeout

[aluon]

Description

Increase the startup timeout for nodes during Forge tests. Pods sometimes take longer to startup when they get rescheduled to a different node and attach a pvc. Also added a separate check for pod status that could help us distinguish between infra errors vs other issues

How Has This Been Tested?

https://github.com/aptos-labs/aptos-core/actions/runs/11750653293

Key Areas to Review

Type of Change

• New feature
• Bug fix
• Breaking change
• Performance improvement
• Refactoring
• Dependency update
• Documentation update
• Tests

Which Components or Systems Does This Change Impact?

• Validator Node
• Full Node (API, Indexer, etc.)
• Move/Aptos Virtual Machine
• Aptos Framework
• Aptos CLI/SDK
• Developer Infrastructure
• Move Compiler
• Other (specify)

Checklist

• I have read and followed the CONTRIBUTING doc
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I identified and added all stakeholders and component owners affected by this change as reviewers
• I tested both happy and unhappy path of the functionality
• I have made corresponding changes to the documentation

[trunk-io]

⏱️ 1h 9m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
execution-performance / test-target-determinator	17m	🟩 🟩 🟩 🟩
test-target-determinator	9m	🟩 ⬜ 🟩
check	9m	🟩 ⬜ 🟩
rust-cargo-deny	5m	🟩 ⬜ 🟩
rust-doc-tests	5m	🟩
fetch-last-released-docker-image-tag	4m	🟩 ⬜ 🟩
check-dynamic-deps	4m	🟩 🟩 🟩 🟩
rust-doc-tests	4m	🟥
semgrep/ci	2m	🟩 🟩 🟩 🟩
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
general-lints	1m	🟩 🟩 🟩
rust-move-tests	1m	⬜
rust-doc-tests	1m	⬜
execution-performance / single-node-performance	41s	🟩 🟩 🟩 🟩

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
execution-performance / single-node-performance	10s	19m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions]

Forge is running suite compat on fcd2dedf6ca61a23f4979e944c629dcdbdae5dca ==> 298f89ef037337f05570ee98d38526f1e4e2f2c7

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite realistic_env_max_load success on 298f89ef037337f05570ee98d38526f1e4e2f2c7

two traffics test: inner traffic : committed: 14532.96 txn/s, latency: 2735.62 ms, (p50: 2700 ms, p70: 2700, p90: 2900 ms, p99: 3000 ms), latency samples: 5525780
two traffics test : committed: 100.03 txn/s, latency: 1461.15 ms, (p50: 1400 ms, p70: 1400, p90: 1500 ms, p99: 8400 ms), latency samples: 1800
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 1.998, avg: 1.566", "ConsensusProposalToOrdered: max: 0.318, avg: 0.291", "ConsensusOrderedToCommit: max: 0.359, avg: 0.347", "ConsensusProposalToCommit: max: 0.648, avg: 0.639"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.88s no progress at version 2470552 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.59s no progress at version 2470550 (avg 8.59s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking