Run all tests against S3 and S3 Express #90

dnanuti · 2023-12-06T15:06:03Z

Currently, we are running just a subset of tests for S3 Express One Zone.
Enhance Github CI actions and variables to run all tests against both S3 and S3 Express One Zone. Address tests flakiness by creating per test assets using BucketPrefixFixture, instead of using static assets. Add storage class as part of test fixture.

These changes require the following environment variables for running the integration tests locally:

to run them against S3:

S3_REGION, S3_BUCKET, CI_PREFIX

to run them against S3 Express:

S3_EXPRESS_REGION, S3_EXPRESS_BUCKET, CI_PREFIX, CI_STORAGE_CLASS=EXPRESS_ONEZONE

Description

Update automated test runs to cover S3 Express One Zone.

Testing

Locally run pytest commands with CI parameters

By submitting this pull request, I confirm that my contribution is made under the terms of BSD 3-Clause License and I agree to the terms of the LICENSE.

muddyfish · 2023-12-07T12:06:26Z

.github/workflows/python-integration.yml

@@ -79,11 +79,33 @@ jobs:
          python -m pip install -e "s3torchconnectorclient[test,e2e]"
          python -m pip install -e "s3torchconnector[test,e2e]"

-      - name: s3torchconnector integration tests
-        run: pytest s3torchconnector/tst/e2e -n auto
+      - name: s3torchconnector S3 integration tests


Should we add s3 express/standard to our matrix? That would decrease duplication here.

Definitely, let me just get it to work!

muddyfish · 2023-12-07T12:08:22Z

s3torchconnectorclient/python/tst/integration/conftest.py

+    storage_class = getenv("CI_STORAGE_CLASS")
+    assert prefix == "" or prefix.endswith("/")
+
+    nonce = random.randrange(2**64)


random.getrandbits might be better here, or even just a UUID

This is the current behaviour, I haven't changed it, just added extra logic.

I am happy to update it if you could please detail your reasoning 😄

I have no strong opinion here if you don't think it looks nicer :)

muddyfish · 2023-12-08T15:06:43Z

.github/workflows/python-integration.yml

@@ -29,10 +25,10 @@ jobs:
      fail-fast: false
      matrix:
        python-version: ["3.8", "3.9", "3.10", "3.11"]
+        test-run: [ "S3", "S3 Express"]


Could we use paired values like described in this StackOverflow answer? https://stackoverflow.com/a/68940067

Thanks a lot! That's awesome, I looked for something similar, but didn't manage to find it.

muddyfish · 2023-12-08T15:07:24Z

s3torchconnector/tst/e2e/conftest.py

@@ -14,8 +13,9 @@

 def getenv(var: str) -> str:
    v = os.getenv(var)
-    if v is None:
-        raise Exception(f"required environment variable {var} is not set")
+    optional = ["CI_STORAGE_CLASS"]


Why not have 'optional' be a default argument to the function?

I did not intend on keeping this, but added.

muddyfish · 2023-12-08T15:08:47Z

s3torchconnector/tst/e2e/test_e2e_s3datasets.py

@@ -29,7 +29,10 @@ def test_s3mapdataset_images_10_from_prefix(image_directory):

    # Intentional usage to emphasize the accessor dataset[index] works.
    for index in range(len(dataset)):
-        key = f"{image_directory.prefix}img{index:03d}.jpg"
+        key = dataset[index].key
+        if image_directory.storage_class is None:


Could we do a not equals express in case we want to try other storage classes?

muddyfish · 2023-12-08T15:19:57Z

s3torchconnectorclient/python/tst/integration/test_mountpoint_s3_integration.py

-        next(client.get_object(f"{bucket}-{uuid.uuid4()}", key))
+        next(
+            client.get_object(
+                f"{sample_directory.bucket}-{uuid.uuid4()}", sample_directory.prefix


I think this is missing the key, but that shouldn't affect the outcome of the test

Enhance Github CI actions and variables to run all tests against both S3 and S3 Express. Address test flakiness by creating per test assets using BucketPrefixFixture, instead of static assets. Remove hypothesis setup from test_mountpoint_client_pickles() as it was flaky. We will add this back when we align on the limits of the parameters.

dnanuti had a problem deploying to integration-tests December 6, 2023 15:06 — with GitHub Actions Failure

dnanuti changed the title ~~Run all tests against S3 and S3 Express~~ WIP: Run all tests against S3 and S3 Express Dec 6, 2023

muddyfish reviewed Dec 7, 2023

View reviewed changes

dnanuti force-pushed the fix/tests-stability branch from fc2dd71 to 2883276 Compare December 7, 2023 13:58

dnanuti had a problem deploying to integration-tests December 7, 2023 13:58 — with GitHub Actions Failure

dnanuti had a problem deploying to integration-tests December 7, 2023 14:05 — with GitHub Actions Failure

dnanuti had a problem deploying to integration-tests December 7, 2023 14:12 — with GitHub Actions Failure

dnanuti had a problem deploying to integration-tests December 7, 2023 14:33 — with GitHub Actions Failure

dnanuti had a problem deploying to integration-tests December 7, 2023 14:39 — with GitHub Actions Failure

dnanuti had a problem deploying to integration-tests December 7, 2023 14:45 — with GitHub Actions Failure

muddyfish reviewed Dec 8, 2023

View reviewed changes

dnanuti dismissed fuatbasik’s stale review via b0aa930 December 8, 2023 15:32

dnanuti had a problem deploying to integration-tests December 8, 2023 15:32 — with GitHub Actions Failure

dnanuti had a problem deploying to integration-tests December 8, 2023 15:35 — with GitHub Actions Failure

dnanuti force-pushed the fix/tests-stability branch from 977806e to fe5fad8 Compare December 8, 2023 15:38

dnanuti had a problem deploying to integration-tests December 8, 2023 15:38 — with GitHub Actions Error

dnanuti had a problem deploying to integration-tests December 8, 2023 15:39 — with GitHub Actions Error

dnanuti force-pushed the fix/tests-stability branch from fe5fad8 to 303f6b2 Compare December 8, 2023 16:02

dnanuti temporarily deployed to integration-tests December 8, 2023 16:02 — with GitHub Actions Inactive

muddyfish approved these changes Dec 8, 2023

View reviewed changes

dnanuti force-pushed the fix/tests-stability branch from 303f6b2 to 4685582 Compare December 8, 2023 16:11

dnanuti temporarily deployed to integration-tests December 8, 2023 16:11 — with GitHub Actions Inactive

dnanuti merged commit 1f8fdd0 into awslabs:main Dec 8, 2023
14 checks passed

dnanuti deleted the fix/tests-stability branch April 11, 2024 14:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run all tests against S3 and S3 Express #90

Run all tests against S3 and S3 Express #90

dnanuti commented Dec 6, 2023 •

edited

Loading

muddyfish Dec 7, 2023

dnanuti Dec 7, 2023 •

edited

Loading

muddyfish Dec 7, 2023

dnanuti Dec 7, 2023

dnanuti Dec 7, 2023

muddyfish Dec 8, 2023

muddyfish Dec 8, 2023

dnanuti Dec 8, 2023

muddyfish Dec 8, 2023

dnanuti Dec 8, 2023

muddyfish Dec 8, 2023

dnanuti Dec 8, 2023

muddyfish Dec 8, 2023

Run all tests against S3 and S3 Express #90

Run all tests against S3 and S3 Express #90

Conversation

dnanuti commented Dec 6, 2023 • edited Loading

Description

Testing

Choose a reason for hiding this comment

dnanuti Dec 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dnanuti commented Dec 6, 2023 •

edited

Loading

dnanuti Dec 7, 2023 •

edited

Loading