Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pull all test images to avoid timing issues during tests #1349

Merged
merged 3 commits into from
Oct 2, 2023

Conversation

Stringy
Copy link
Collaborator

@Stringy Stringy commented Sep 29, 2023

Description

This PR will pre-pull test images on CI to make test execution times more consistent. There is currently a situation where test setup might start various containers and then wait to pull an image. When the timing of events is important/crucial, this waiting can cause flakes.

This behaviour is disabled when running individual test suites because the upfront performance penalty is significant at that level. I have yet to figure out a neat way of only pulling the images that the requested test uses, but that can be looked at in a follow up if it is deemed useful.

Checklist

  • Investigated and inspected CI test results
  • Updated documentation accordingly

Automated testing

  • Added unit tests
  • Added integration tests
  • Added regression tests

If any of these don't apply, please comment below.

Testing Performed

Tested locally, with timings:

# with pre-pulling images:

➜  collector git:(giles/ROX-14542-pre-pull-all-images) ✗ time make -C ansible/ integration-tests COLLECTION_METHODS=core-bpf COLLECTOR_IMAGE=quay.io/stackrox-io/collector:3.16.0 COLLECTOR_TEST=TestProcessNetwork
ansible-playbook \
		-i dev \
		-e job_id="ghutton" \
		-e @secrets.yml \
		--tags run-tests \
		integration-tests.yml
...................
# run-test-target : Run integration tests *********************************************************************************
  * collector-dev-garden-linux-any-777da73a-ghutton- changed=True --  -------------------------
......

# STATS *******************************************************************************************************************
collector-dev-garden-linux-any-777da73a-ghutton    : ok=26	changed=13	failed=0	unreachable=0	rescued=0	ignored=0

________________________________________________________
Executed in  376.03 secs    fish           external
   usr time   52.34 secs    0.13 millis   52.34 secs
   sys time   13.40 secs    1.55 millis   13.40 secs

# without pre-pulling images:

➜  collector git:(giles/ROX-14542-pre-pull-all-images) ✗ time make -C ansible/ integration-tests COLLECTION_METHODS=core-bpf COLLECTOR_IMAGE=quay.io/stackrox-io/collector:3.16.0 COLLECTOR_TEST=TestProcessNetwork
ansible-playbook \
		-i dev \
		-e job_id="ghutton" \
		-e @secrets.yml \
		--tags run-tests \
		integration-tests.yml
.............
# run-test-target : Run integration tests *********************************************************************************
  * collector-dev-garden-linux-any-777da73a-ghutton- changed=True --  -------------------------
......

# STATS *******************************************************************************************************************
collector-dev-garden-linux-any-777da73a-ghutton    : ok=20	changed=9	failed=0	unreachable=0	rescued=0	ignored=0

________________________________________________________
Executed in  228.68 secs    fish           external
   usr time   44.81 secs    0.14 millis   44.81 secs
   sys time   10.31 secs    1.60 millis   10.31 secs

Copy link
Collaborator

@Molter73 Molter73 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@github-actions
Copy link

Kernel Method Without Collector Time (secs) With Collector Time (secs) Baseline median (secs) Collector median (secs) PValue

@Stringy Stringy merged commit a5f90a6 into master Oct 2, 2023
43 checks passed
@Stringy Stringy deleted the giles/ROX-14542-pre-pull-all-images branch October 2, 2023 08:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants