Sync devDeploy with recent Replayer auth changes #279

lewijacn · 2023-08-22T15:32:59Z

Description

Adjust devDeploy.sh with recent Replayer auth changes. Also remove additional "orchestration" logic from devDeploy surrounding setting the auth header, which will be taxing to keep up to date and probably doesn't belong there.

Issues Resolved

None

Testing

Manual deployment testing

Check List

New functionality includes testing
- All tests pass, including unit test, integration test and doctest
New functionality has been documented
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Tanner Lewis <[email protected]>

codecov · 2023-08-22T15:37:07Z

Codecov Report

Merging #279 (61a7ad1) into main (b7c3da0) will not change coverage.
Report is 18 commits behind head on main.
The diff coverage is 71.28%.

❗ Current head 61a7ad1 differs from pull request most recent head bd5a6b1. Consider uploading reports for the commit bd5a6b1 to get more accurate results

@@            Coverage Diff            @@
##               main     #279   +/-   ##
=========================================
  Coverage     61.93%   61.93%           
  Complexity      617      617           
=========================================
  Files            82       82           
  Lines          3129     3129           
  Branches        292      292           
=========================================
  Hits           1938     1938           
  Misses         1014     1014           
  Partials        177      177

Flag	Coverage Δ
unittests	`61.93% <71.28%> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...n/java/org/opensearch/migrations/replay/Utils.java	`40.00% <ø> (ø)`
.../datahandlers/http/NettyJsonContentCompressor.java	`88.37% <ø> (ø)`
...ions/transform/RemovingAuthTransformerFactory.java	`0.00% <0.00%> (ø)`
.../opensearch/migrations/replay/TrafficReplayer.java	`7.17% <12.90%> (ø)`
...nsearch/migrations/transform/IAuthTransformer.java	`28.57% <28.57%> (ø)`
...replay/PacketToTransformingHttpHandlerFactory.java	`63.63% <75.00%> (ø)`
.../org/opensearch/migrations/replay/SigV4Signer.java	`77.58% <77.58%> (ø)`
.../datahandlers/http/NettyJsonContentAuthSigner.java	`86.36% <86.36%> (ø)`
...datahandlers/http/RequestPipelineOrchestrator.java	`96.36% <92.30%> (ø)`
...tyDecodedHttpRequestPreliminaryConvertHandler.java	`94.50% <96.00%> (ø)`
... and 16 more

gregschohn

Please look at the comment on 'extra args'. Approving this since it should unbreak my previous changes.

gregschohn · 2023-08-22T16:49:04Z

deployment/copilot/README.md

+There is a level of precedence that will determine which or if any Auth header should be added to outgoing Replayer requests, which is listed below.
+1. If the user provides an explicit auth header option to the Replayer, such as providing a static value auth header or using a user and secret arn pair, this mechanism will be used for the auth header of outgoing requests. The options can be found as Parameters [here](../../TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java)
+2. If the devDeploy script is used and its Replayer command has not been altered (as in the case of 1.) and CDK deploys a target cluster with a configured FGAC user (see `fineGrainedManagerUserName` and `fineGrainedManagerUserSecretManagerKeyARN` CDK context options [here](../cdk/opensearch-service-migration/README.md)) or is running in demo mode (see `enableDemoAdmin` CDK context option), this user and secret arn pair will be provided in the Replay command for the Auth header
+3. If the user provides no auth header option and incoming captured requests have an auth header, this auth header will be reused for outgoing requests
+4. If the user provides no auth header option and incoming captured requests have no auth header, then no auth header will be used for outgoing requests


This should probably moved to the Replayer's README and linked from here. That doesn't need to be done now though.

Have went ahead and moved and referenced here

gregschohn · 2023-08-22T16:57:43Z

deployment/copilot/devDeploy.sh

-    --auth-header-value)
-      REPLAY_AUTH_HEADER="$2"
-      shift # past argument
-      shift # past value
-      ;;
-    --aws-auth-header-user)
-      REPLAY_AWS_AUTH_HEADER_USER="$2"
-      shift # past argument
-      shift # past value
-      ;;
-    --aws-auth-header-secret)
-      REPLAY_AWS_AUTH_HEADER_SECRET="$2"
-      shift # past argument
-      shift # past value
-      ;;


Could you have the dev-deploy script take in the arguments to use for the replayer? Something like "--extraFlags" and then let the developer tack on any auth flags or anything else that might be necessary. As it is, it will be painful for a dev to play around w/ different options.

And I would say that none of the arguments are checked - which should be fine if this is a "developer" script

I am a little conflicted with what we should do here. I don't want the additional tax of actually knowing what arguments the Replayer takes which makes something like "--extraFlags" seem preferrable, however this also becomes a balancing act of does my default command already have an argument that is being passed in with "--extraFlags", it seems it would be easier for the user to just update the command in the script to what they need.

What might be slightly convenient is to allow a user to specify all the arguments for the Replayer (not sure how useful this is) in the devDeploy script, or do something like "--replayerAuthFlags" which allows specifying auth flags and defaults to our normal setting but can be overriden with whatever other auth flags

Lmk your thoughts here

gregschohn · 2023-08-22T16:58:57Z

deployment/copilot/devDeploy.sh

-# Gather CDK output which includes export commands needed by Copilot, and make them available to the environment
-found_exports=$(grep -o "export [a-zA-Z0-9_]*=[^\\;\"]*" cdkOutput.json)
-eval "$(grep -o "export [a-zA-Z0-9_]*=[^\\;\"]*" cdkOutput.json)"
+# Collect export commands from CDK output, which are needed by Copilot, wrap the values in quotes and make them available to the environment


nit - are you finding/gathering commands or values?

Signed-off-by: Tanner Lewis <[email protected]>

opensearch-trigger-bot · 2023-08-22T19:35:39Z

The backport to capture-and-replay-v0.1.0 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-capture-and-replay-v0.1.0 capture-and-replay-v0.1.0
# Navigate to the new working tree
pushd ../.worktrees/backport-capture-and-replay-v0.1.0
# Create a new branch
git switch --create backport/backport-279-to-capture-and-replay-v0.1.0
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 4165dc48496deb0670d370d7c478fabc69b5c702
# Push it to GitHub
git push --set-upstream origin backport/backport-279-to-capture-and-replay-v0.1.0
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-capture-and-replay-v0.1.0

Then, create a pull request where the base branch is capture-and-replay-v0.1.0 and the compare/head branch is backport/backport-279-to-capture-and-replay-v0.1.0.

) * Sync devDeploy with recent Replayer auth changes Signed-off-by: Tanner Lewis <[email protected]> (cherry picked from commit 4165dc4)

* Sync devDeploy with recent Replayer auth changes Signed-off-by: Tanner Lewis <[email protected]> (cherry picked from commit 4165dc4)

* main: (27 commits) Align capture proxy for docker and Copilot and add cat indices convenience script MIGRATIONS-1289: Add end of segment message to capture serializer (opensearch-project#276) Sync devDeploy with recent Replayer auth changes (opensearch-project#279) Update documentation for auth header Sync devDeploy with recent Replayer auth changes Fix linter error Handle chunked transfer encoding Make path search case-insensitive-ish and don't output bytes to a json PR feedback around command line arguments Address review comments Bugfix to apply the default signer class's content checksum when there isn't already one present. This pushes the success rate for signatures to 100% in the first 120 messages from the opensearch benchmark workload when rerun against a cluster that uses sigv4. Add error handling, progress bar, cleanup Bugfixes to get SigV4 working in at least simple cases w/ and w/out payload streams. A couple other minor improvements and cleanups were thrown in here too, most notably, tracking the shadow request packets that are sent, which is very helpful to debug signing problems. Add documentation and some cleanup/todos Add script to transform tuples to human readable format Review comment and basic documentation Update script with params and defaults for copilot vs docker Cleanup after reviewing the changes on the branch in aggregate. Cleanup and bugfixes to get SigV4 more correctly wired with the rest of the replayer. Further changes to unify the replayer's auth story and to tie sigv4 signing all the way back to the command line. ... Signed-off-by: Greg Schohn <[email protected]> # Conflicts: # TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/PacketToTransformingHttpHandlerFactory.java # TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java # TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/Utils.java # TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/datahandlers/NettyPacketToHttpConsumer.java # TrafficCapture/trafficReplayer/src/test/java/org/opensearch/migrations/replay/PayloadRepackingTest.java

Sync devDeploy with recent Replayer auth changes

61a7ad1

Signed-off-by: Tanner Lewis <[email protected]>

lewijacn requested review from chelma, gregschohn, kartg, mikaylathompson, okhasawn and sumobrian as code owners August 22, 2023 15:32

gregschohn approved these changes Aug 22, 2023

View reviewed changes

Update documentation for auth header

bd5a6b1

Signed-off-by: Tanner Lewis <[email protected]>

sumobrian added backport capture-and-replace-v0.1.0 backport capture-and-replay-v0.1.0 and removed backport capture-and-replace-v0.1.0 labels Aug 22, 2023

lewijacn merged commit 4165dc4 into opensearch-project:main Aug 22, 2023
2 of 5 checks passed

lewijacn added backport capture-and-replay-v0.1.0 and removed backport capture-and-replay-v0.1.0 labels Aug 22, 2023

lewijacn added a commit that referenced this pull request Aug 23, 2023

Sync devDeploy with recent Replayer auth changes (#279) (#281)

7706b40

* Sync devDeploy with recent Replayer auth changes Signed-off-by: Tanner Lewis <[email protected]> (cherry picked from commit 4165dc4)

lewijacn deleted the adjust-dev-deploy-auth branch March 29, 2024 17:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sync devDeploy with recent Replayer auth changes #279

Sync devDeploy with recent Replayer auth changes #279

lewijacn commented Aug 22, 2023

codecov bot commented Aug 22, 2023 •

edited

Loading

gregschohn left a comment

gregschohn Aug 22, 2023

lewijacn Aug 22, 2023

gregschohn Aug 22, 2023

lewijacn Aug 22, 2023 •

edited

Loading

gregschohn Aug 22, 2023

lewijacn Aug 22, 2023

opensearch-trigger-bot bot commented Aug 22, 2023

Sync devDeploy with recent Replayer auth changes #279

Sync devDeploy with recent Replayer auth changes #279

Conversation

lewijacn commented Aug 22, 2023

Description

Issues Resolved

Testing

Check List

codecov bot commented Aug 22, 2023 • edited Loading

Codecov Report

gregschohn left a comment

Choose a reason for hiding this comment

gregschohn Aug 22, 2023

Choose a reason for hiding this comment

lewijacn Aug 22, 2023

Choose a reason for hiding this comment

gregschohn Aug 22, 2023

Choose a reason for hiding this comment

lewijacn Aug 22, 2023 • edited Loading

Choose a reason for hiding this comment

gregschohn Aug 22, 2023

Choose a reason for hiding this comment

lewijacn Aug 22, 2023

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Aug 22, 2023

codecov bot commented Aug 22, 2023 •

edited

Loading

lewijacn Aug 22, 2023 •

edited

Loading