[Chore] Improve test flakes

[jhamon]

Problem

I frequently see recurring test flakes, most often at the cleanup stage of test_configure_index_with_deletion_protection because the index cannot be deleted while still in an "upgrading" state. The index is in that state while configuration changes are being applied.

Solution

• In deletion protection test, wait for the index to be ready before attempting the delete.
• In dependency tests which do a basic sanity test, loosen the assertions. We're not really intending to test the backend functionality with these tests; we just want to make sure network calls are successfully made with a variety of different grpc / protobuff / etc dependency versions.
• Remove some assertions that rely on non-deterministic behavior on the backend. E.g. how long it takes for a serverless index to be considered ready.
• Check for deletion protection status before trying to delete.
• Don't fail the job when cleanup fails; even if we re-run the entire job, there will still be an orphaned index from a previous run that needs to be cleaned up. Best to just attempt to delete, then let the nightly cleanup job garbage collect anything left over.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update
• Infrastructure change (CI configs, etc)
• Non-code change (docs, etc)
• None of the above: (explain here)

[rohanshah18]

LGTM

[haruska]

small nit: because of logic ordering, you end up sleeping for 2 seconds before failing on the last iteration.

[jhamon]

This is what comes from outsourcing too much typing to github copilot 🤣

[haruska]

why both a break and boolean flag? could just loop/break?

[haruska]

I believe you can delete in other states as well.

[jhamon]

You would think, wouldn't you? Maybe this is true in some cases, but a lot of flakes seen in the past are about trying to delete while the index is "upgrading". I.e. if you called configure index too recently, your delete request will be rejected.

[austin-denoble]

I actually changed all my integration tests to check both status.state and status.ready. There are situations where they are not in sync, and the index is marked as Ready but then the state is something else.

I'd maybe recommend checking both just to be thorough, it's helped with some of my flakiness. I think @aulorbe reported this to the db team at some point.

[austin-denoble]

LGTM, this also seems in line with some of the approaches we've had to take in other clients to massage some of these issues, specifically around cleanup.

I basically just retry the cleanup delete call a set number of times before bailing in Go.