Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 publishing speedup #1171

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

S3 publishing speedup #1171

wants to merge 2 commits into from

Commits on Apr 20, 2023

  1. Support scanning S3 bucket in parallel

    Add setting `parallelListingRequests` to an S3 publish endpoint. If set
    to a value greater than 1, aptly will use up to that number of parallel
    HTTP requests to list the S3 bucket.
    
    Fix the test S3 server: in case on multiple CommonPrefixes being
    returned, S3 specs require multiple CommonPrefix XML tags (and not a
    single CommonPrefix with multiple Prefix tags).
    cavedon committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    52a135b View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2023

  1. Cache bucket content by prefix

    When a publishing uses a publish prefix, instead of listing the contents
    of the whole bucket under the storage prefix, only list the contents of
    the bucket under the storage prefix and publish prefix, and cache it by
    publish prefix.
    This speeds up publish operations under a prefix.
    cavedon committed Apr 24, 2023
    Configuration menu
    Copy the full SHA
    d90a1ad View commit details
    Browse the repository at this point in the history