-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AWS SigV4 support in Fetch Migration #394
AWS SigV4 support in Fetch Migration #394
Conversation
60bc424
to
10a43d3
Compare
Codecov Report
@@ Coverage Diff @@
## main #394 +/- ##
============================================
- Coverage 63.64% 63.55% -0.10%
+ Complexity 718 715 -3
============================================
Files 82 82
Lines 3298 3298
Branches 303 303
============================================
- Hits 2099 2096 -3
- Misses 1011 1014 +3
Partials 188 188
Flags with carried forward coverage won't be shown. Click here to find out more. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you update this library?
https://github.com/opensearch-project/opensearch-migrations/pull/394/checks?check_run_id=18495377062
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lgtm. Minor changes -- no need to re-review. One thing to double check is the documentation. I believe this was documented in a sample yml but didn't double check.
# Utility method to creat a test config section | ||
def create_config_section(plugin_config: dict) -> dict: | ||
valid_plugin = dict() | ||
valid_plugin[random.choice(SUPPORTED_ENDPOINTS)] = plugin_config | ||
config_section = copy.deepcopy(BASE_CONFIG_SECTION) | ||
config_section[TEST_KEY].append(valid_plugin) | ||
return config_section |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Out of curiosity, can you explain what this is doing? especially the random.choice
part please.
NIT: missing 'e' in "create" in the comment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Data Prepper pipeline configuration file is composed of several "sections" - source
and sink
being the two familiar ones. Each section can be configured with one or more plugins. Every plugin has a name, and an associated plugin configuration.
This utility method helps in the creation of one of those "sections" for testing. The random.choice
logic randomly picks a supported endpoint as the plugin name, and uses the supplied plugin_config
argument as its plugin configuration value. Instead of source
or sink
, the section name uses the value of TEST_KEY
Fixed the typo as a part of the latest commit.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm going to go ahead and merge this. @okhasawn if you have any other comments, please continue to provide them and i'll incorporate them in a follow-up PR.
…tInfo class This change adds a class method to construct API endpoint paths instead of having callers compute this on their own. It also includes better/normalized handling of slashes in endpoint and path URIs. A minor refactoring of logging in migration_monitor.py is also included, which improves unit test code coverage. Signed-off-by: Kartik Ganesh <[email protected]>
Also adding package dependencies on AWSSigV4 libraries Signed-off-by: Kartik Ganesh <[email protected]>
Signed-off-by: Kartik Ganesh <[email protected]>
Signed-off-by: Kartik Ganesh <[email protected]>
Signed-off-by: Kartik Ganesh <[email protected]>
10a43d3
to
d684a10
Compare
Signed-off-by: Kartik Ganesh <[email protected]>
d684a10
to
98008f0
Compare
Description
This PR enables support for SigV4 signed requests to AWS OpenSearch / OpenSearch Serverless endpoints in Fetch Migration, using the requests-aws4auth library. To achieve this, this change adds several components:
1 - All “endpoint” related logic is now encapsulated in the
EndpointInfo
class, including logic to construct API paths instead of having callers compute this.2 - Construction of
EndpointInfo
instances from the Data Prepper pipeline configuration (and its plugin configuration sub-sections) has been moved out ofmetadata_migration.py
to a newendpoint_utils.py
file for better abstraction.3 - Use of SigV4 is inferred from the supplied DP pipeline (separately for source and sink), including detection of the
serverless
key to change the service name (‘es’ vs. ‘aoss’)4 - Since
AWS4Auth
requires aregion
argument, Fetch Migration first checks the plugin configuration for an explicitly defined region. If this is not present (since it is an optional parameter), the code attempts to derive the region based on the service endpoint URL (since generated endpoint URLs usually include the region). If a region cannot be inferred, aValueError
is thrown.Unit tests for all of these components have been added (or existing ones updated). A minor refactoring of logging in
migration_monitor.py
is also included, which improves unit test code coverage.Testing
Migration to an OpenSearch Serverless collection has also been tested and verified via the CDK.
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.