Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce integration test retries #1546

Merged
merged 6 commits into from
Feb 12, 2024
Merged

Reduce integration test retries #1546

merged 6 commits into from
Feb 12, 2024

Conversation

robbycochran
Copy link
Collaborator

@robbycochran robbycochran commented Feb 11, 2024

Description

Problem

Integration tests taking unneeded time due to retries on operations involving removed containers, as well as unnecessarily long health checks and redundant cleanups.

Solution

  • Implemented early exits in retry functions for remove, stop, and kill container operations if the container is no longer present based on the output message.
  • Reduced collector health check wait time from 30s to 5s. This improves the common case where collector is healthy before 30 sec has elapsed.
  • Eliminated redundant calls to collector cleanup functions.
  • Corrected an error in the container name for jsonlabel, ending repeated cleanup retries.

Impact

Integration test suite execution time (including VM provisioning) decreased significantly, from approximately 77 minutes to 29 minutes, representing a ~2.75x speedup.

Checklist

  • Investigated and inspected CI test results
  • Updated documentation accordingly

Automated testing
- [ ] Added unit tests
- [ ] Added integration tests
- [ ] Added regression tests

Testing Performed

Tested locally and on CI with all tests labels

- Exit Retry fn early if container doesn't exist on rm, stop, and kill
- Shorten collector health check tick (30s -> 5s)
- Small fixes for collector and jsonlabel cleanup
@robbycochran robbycochran requested a review from a team as a code owner February 11, 2024 17:37
Copy link
Collaborator

@Stringy Stringy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a pretty incredible speed up, and for relatively few changes. LGTM! Thanks Robby!

@robbycochran robbycochran merged commit 2db86d1 into master Feb 12, 2024
59 checks passed
@robbycochran robbycochran deleted the rc-reduce-test-retries branch February 12, 2024 17:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants