AOSS 500/502 responses repeatedly observed under heavy load

[alexdunnjpl]

Checked for duplicates

No - I haven't checked

🐛 Describe the bug

When running reindexing sweeper against MCP ATM without any flow control, read requests begin to fail with HTTP502 at ~90% progress during the initial attempt to iterate through all 3.3M docs, and the errors persist for some minutes afterwards, during which time the AOSS dev dashboard is also unavailable, returning HTTP500.

🕵️ Expected behavior

I expect the system to remain stable.

📜 To Reproduce

Run the reindexer sweeper as of 9339ede0 and wait for failure.

🖥 Environment Info

• Version of this software [e.g. vX.Y.Z]
• Operating System: [e.g. MacOSX with Docker Desktop vX.Y]
  ...

📚 Version of Software Used

No response

🩺 Test Data / Additional context

No response

🦄 Related requirements

🦄 #xyz

⚙️ Engineering Details

No response

🎉 Integration & Test

No response