Replayer Transformation support #937

gregschohn · 2024-09-08T14:06:23Z

Description

Json transformations can work when bodies aren't present and when the bodies are of 'ndjson' types (like OpenSearch bulk requests).

There was also a test (testMalformedPayload_andTypeMappingUri_IsPassedThrough) that sent malformed JSON in a payload. The test didn't do much and a comment admitted that we weren't likely doing the right thing. Moving a buffer to Unpooled caused the test to fail w/ a leak. Upon closer inspection, errors while parsing JSON didn't cause the payload contents to be dumped into the inlinedBinaryBody key of the payload. Now that content is captured and passed forward (plus, refCounted appropriately, though still as an Unpooled to minimize risk from transformations [see code comment]). In the case of this existing test, which has been moved to HttpJsonTransformingConsumerTest, the JsonTransformerForOpenSearch23PlusTargetTransformerProvider that the test uses throws an exception when the json payload isn't present. That was also not being handled properly. Now it is handled by marking the transformation as an error w/ the exception wrapped in a "TransformationException". In that case, the transformation status is marked as an error and the contents of the entire message will be empty. It feels more appropriate that if a transformation threw to be as conservative as possible and not put anything on the wire, in case some part of it was sensitive.
Lastly, I refactored the AggregatedRawResponse into a base class for AggregatedRawResult. We use that class to accumulate transformation results. It was extremely confusing to see an HTTP response and the word response on so many of the fields. Now the word 'result' is used and the usage in various contexts makes more sense.

Category Enhancement/Bug fix
Why these changes are required? So that users can transform contents of http requests. This was a feature that was added nearly 18 months ago, but has been incomplete as shown by some simple OpenSearch Benchmark test runs.
What is the old behavior before changes and new behavior after changes? Before, a number of requests would be garbled in various ways when payload transformations were present. Now they should be passed through much more reasonably. This at least gets us to the point where we can now see some non-trivial bugs.

Issues Resolved

https://opensearch.atlassian.net/browse/MIGRATIONS-1984

Testing

Unit testing and docker compose testing.

Check List

New functionality includes testing
- All tests pass, including unit test, integration test and doctest
New functionality has been documented
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…via a base64 encoded string + a test sample script. Add a test/sample script to excise specific elements, when present, from a json tree and also drop a component of the URI path. Signed-off-by: Greg Schohn <[email protected]>

…er transforms for docs w/out bodies or with malformed bodies. Signed-off-by: Greg Schohn <[email protected]>

codecov · 2024-09-08T14:07:18Z

Codecov Report

Attention: Patch coverage is 79.41176% with 42 lines in your changes missing coverage. Please review.

Project coverage is 80.81%. Comparing base (5fd341d) to head (a4a9405).
Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
.../opensearch/migrations/replay/TrafficReplayer.java	0.00%	14 Missing ⚠️
...nsearch/migrations/replay/AggregatedRawResult.java	81.57%	5 Missing and 2 partials ⚠️
.../replay/datahandlers/PayloadAccessFaultingMap.java	57.14%	4 Missing and 2 partials ⚠️
...ahandlers/http/NettyJsonBodyAccumulateHandler.java	86.04%	2 Missing and 4 partials ⚠️
...tahandlers/http/NettyJsonBodySerializeHandler.java	84.61%	2 Missing and 2 partials ⚠️
...lay/datahandlers/http/TransformationException.java	50.00%	2 Missing ⚠️
...igrations/replay/datahandlers/JsonAccumulator.java	90.90%	1 Missing ⚠️
...atahandlers/http/HttpJsonTransformingConsumer.java	91.66%	0 Missing and 1 partial ⚠️
...datahandlers/http/NettyJsonBodyConvertHandler.java	90.90%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #937      +/-   ##
============================================
+ Coverage     80.69%   80.81%   +0.11%     
- Complexity     2691     2726      +35     
============================================
  Files           398      400       +2     
  Lines         15317    15404      +87     
  Branches        956      967      +11     
============================================
+ Hits          12360    12448      +88     
+ Misses         2385     2382       -3     
- Partials        572      574       +2

Flag	Coverage Δ
gradle-test	`76.83% <79.41%> (+0.18%)`	⬆️
python-test	`92.87% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

These are deserialized into a list of json objects and are accessed via a new top-level payload key so that they're distinguishable from a single json object. Any leftover bytes are now appended after json objects. Both the single-json and ndjson-list (mutually exclusive) and the leftover bytes are all accessible for transformations. I've also added a transformation script to the replayer's command for docker-compose to exercise json body transformations. Signed-off-by: Greg Schohn <[email protected]>

…ents its own EntrySet... Accesses to the TreeMap derived object weren't causing faults from transformations, so the it never looked like anybody was trying to access payload contents, causing the transforming pipeline to NOT parse the json or pass the bodies to the transformers. Also, when there is no payload at all, we'll pass that through as an empty byte array. It might be better to pass no payload at all, but that will be a slightly greater change. At the moment, this change allows the OpenSearch Benchmark test run traffic to completely move over to the target. Signed-off-by: Greg Schohn <[email protected]>

…rmations from docker-compose as the default. On the last point, I'll continue to look into differences that could be causing e2e tests to fail. Signed-off-by: Greg Schohn <[email protected]>

peternied · 2024-09-09T13:20:45Z

TrafficCapture/dockerSolution/src/main/docker/docker-compose.yml

@@ -78,7 +78,7 @@ services:
        condition: service_started
      opensearchtarget:
        condition: service_started
-    command: /bin/sh -c "/runJavaWithClasspath.sh org.opensearch.migrations.replay.TrafficReplayer --speedup-factor 2 https://opensearchtarget:9200 --auth-header-value Basic\\ YWRtaW46bXlTdHJvbmdQYXNzd29yZDEyMyE= --insecure --kafka-traffic-brokers kafka:9092 --kafka-traffic-topic logging-traffic-topic --kafka-traffic-group-id logging-group-default --otelCollectorEndpoint http://otel-collector:4317"
+    command: /bin/sh -c "/runJavaWithClasspath.sh org.opensearch.migrations.replay.TrafficReplayer --speedup-factor 2 https://opensearchtarget:9200 --auth-header-value Basic\\ YWRtaW46bXlTdHJvbmdQYXNzd29yZDEyMyE= --insecure --kafka-traffic-brokers kafka:9092 --kafka-traffic-topic logging-traffic-topic --kafka-traffic-group-id logging-group-default --otelCollectorEndpoint http://otel-collector:4317" #--transformer-config-base64 W3sgIkpzb25Kb2x0VHJhbnNmb3JtZXJQcm92aWRlciI6ClsKICB7CiAgICAic2NyaXB0IjogewogICAgICAib3BlcmF0aW9uIjogInNoaWZ0IiwKICAgICAgInNwZWMiOiB7CiAgICAgICAgIm1ldGhvZCI6ICJtZXRob2QiLAogICAgICAgICJVUkkiOiAiVVJJIiwKICAgICAgICAiaGVhZGVycyI6ICJoZWFkZXJzIiwKICAgICAgICAicGF5bG9hZCI6IHsKICAgICAgICAgICJpbmxpbmVkSnNvbkJvZHkiOiB7CiAgICAgICAgICAgICJ0b3AiOiB7CiAgICAgICAgICAgICAgInRhZ1RvRXhjaXNlIjogewogICAgICAgICAgICAgICAgIioiOiAicGF5bG9hZC5pbmxpbmVkSnNvbkJvZHkudG9wLiYiIAogICAgICAgICAgICAgIH0sCiAgICAgICAgICAgICAgIioiOiAicGF5bG9hZC5pbmxpbmVkSnNvbkJvZHkudG9wLiYiCiAgICAgICAgICAgIH0sCiAgICAgICAgICAiKiI6ICJwYXlsb2FkLmlubGluZWRKc29uQm9keS4mIgogICAgICAgICAgfQogICAgICAgIH0KICAgICAgfQogICAgfQogIH0sIAogewogICAic2NyaXB0IjogewogICAgICJvcGVyYXRpb24iOiAibW9kaWZ5LW92ZXJ3cml0ZS1iZXRhIiwKICAgICAic3BlYyI6IHsKICAgICAgICJVUkkiOiAiPXNwbGl0KCcvZXh0cmFUaGluZ1RvUmVtb3ZlJyxAKDEsJikpIgogICAgIH0KICB9CiB9LAogewogICAic2NyaXB0IjogewogICAgICJvcGVyYXRpb24iOiAibW9kaWZ5LW92ZXJ3cml0ZS1iZXRhIiwKICAgICAic3BlYyI6IHsKICAgICAgICJVUkkiOiAiPWpvaW4oJycsQCgxLCYpKSIKICAgICB9CiAgfQogfQpdCn1dCg=="


This command becomes unreadable with the base64 encoded here - can we instead load it from a file path to improve comprehension?

I'm open to accepting base64 encoding directly over the command line, but what I'm struggling with is understanding the escaping/quotes on this like. You have this section:

#--transformer-config-base64

which seems to imply the entire base64 block is part of a non-actionable comment, and the final double quote at the end of the line seems to confirm that (e.g. the line has a non-even number of quotes).

If that's the case - should this either be deleted or added to the actual command?

@peternied - I don't expect customers to be loading this from a file. They could, but that would require them to get a file onto a replayer container. For our CDK deployment, it's much easier to add an extra argument to pass to the replayer than to add an additional file. As such, I was hoping to test something closer to what more customers would be doing.
@chelma - It's a comment for now. I'm debugging through why its inclusion caused some of the E2E tests to fail.

I don't expect customers to be loading this from a file.

How do you expect customer will be building up the configuration? I would expect them to be iterating in a text editor, does your workflow expect them to process then and then copy/paste into a cdk variable and then redeploy.

If we are using a file on a shared EFS volume that text editor could be vim on the migration console, this could be reloaded on boot of the container. This means no deployed is required, just stop/start on the TC instance... or at least that is where my head is at.

Please file a new feature for setting up the EFS volume for that. We can tackle that separately.

chelma

Thanks for the PR. This is my first time looking through this code (really should have investigated it earlier), and I'm struggling a bit to see how we'd take something like the HttpJsonTransformingConsumer and generalize its mechanisms to work for, say, the transformations we need for RFS. Curious what your thoughts are on that?

chelma · 2024-09-09T14:52:47Z

TrafficCapture/dockerSolution/src/main/docker/docker-compose.yml

@@ -78,7 +78,7 @@ services:
        condition: service_started
      opensearchtarget:
        condition: service_started
-    command: /bin/sh -c "/runJavaWithClasspath.sh org.opensearch.migrations.replay.TrafficReplayer --speedup-factor 2 https://opensearchtarget:9200 --auth-header-value Basic\\ YWRtaW46bXlTdHJvbmdQYXNzd29yZDEyMyE= --insecure --kafka-traffic-brokers kafka:9092 --kafka-traffic-topic logging-traffic-topic --kafka-traffic-group-id logging-group-default --otelCollectorEndpoint http://otel-collector:4317"
+    command: /bin/sh -c "/runJavaWithClasspath.sh org.opensearch.migrations.replay.TrafficReplayer --speedup-factor 2 https://opensearchtarget:9200 --auth-header-value Basic\\ YWRtaW46bXlTdHJvbmdQYXNzd29yZDEyMyE= --insecure --kafka-traffic-brokers kafka:9092 --kafka-traffic-topic logging-traffic-topic --kafka-traffic-group-id logging-group-default --otelCollectorEndpoint http://otel-collector:4317" #--transformer-config-base64 W3sgIkpzb25Kb2x0VHJhbnNmb3JtZXJQcm92aWRlciI6ClsKICB7CiAgICAic2NyaXB0IjogewogICAgICAib3BlcmF0aW9uIjogInNoaWZ0IiwKICAgICAgInNwZWMiOiB7CiAgICAgICAgIm1ldGhvZCI6ICJtZXRob2QiLAogICAgICAgICJVUkkiOiAiVVJJIiwKICAgICAgICAiaGVhZGVycyI6ICJoZWFkZXJzIiwKICAgICAgICAicGF5bG9hZCI6IHsKICAgICAgICAgICJpbmxpbmVkSnNvbkJvZHkiOiB7CiAgICAgICAgICAgICJ0b3AiOiB7CiAgICAgICAgICAgICAgInRhZ1RvRXhjaXNlIjogewogICAgICAgICAgICAgICAgIioiOiAicGF5bG9hZC5pbmxpbmVkSnNvbkJvZHkudG9wLiYiIAogICAgICAgICAgICAgIH0sCiAgICAgICAgICAgICAgIioiOiAicGF5bG9hZC5pbmxpbmVkSnNvbkJvZHkudG9wLiYiCiAgICAgICAgICAgIH0sCiAgICAgICAgICAiKiI6ICJwYXlsb2FkLmlubGluZWRKc29uQm9keS4mIgogICAgICAgICAgfQogICAgICAgIH0KICAgICAgfQogICAgfQogIH0sIAogewogICAic2NyaXB0IjogewogICAgICJvcGVyYXRpb24iOiAibW9kaWZ5LW92ZXJ3cml0ZS1iZXRhIiwKICAgICAic3BlYyI6IHsKICAgICAgICJVUkkiOiAiPXNwbGl0KCcvZXh0cmFUaGluZ1RvUmVtb3ZlJyxAKDEsJikpIgogICAgIH0KICB9CiB9LAogewogICAic2NyaXB0IjogewogICAgICJvcGVyYXRpb24iOiAibW9kaWZ5LW92ZXJ3cml0ZS1iZXRhIiwKICAgICAic3BlYyI6IHsKICAgICAgICJVUkkiOiAiPWpvaW4oJycsQCgxLCYpKSIKICAgICB9CiAgfQogfQpdCn1dCg=="


I'm open to accepting base64 encoding directly over the command line, but what I'm struggling with is understanding the escaping/quotes on this like. You have this section:

#--transformer-config-base64

which seems to imply the entire base64 block is part of a non-actionable comment, and the final double quote at the end of the line seems to confirm that (e.g. the line has a non-even number of quotes).

If that's the case - should this either be deleted or added to the actual command?

chelma · 2024-09-09T14:55:26Z

TrafficCapture/trafficReplayer/README.md

@@ -140,8 +148,9 @@ To run only one transformer without any configuration, the `--transformer-config
 be set to the name of the transformer (e.g. 'JsonTransformerForOpenSearch23PlusTargetTransformerProvider', 
 without quotes or any json surrounding it).

-The user can also specify a file to read the transformations from using the `--transformer-config-file`, but can't use
-both transformer options.
+The user can also specify a file to read the transformations from using the `--transformer-config-file`.  Users can


Curious if you have a plan for how to get the user-supplied --transformer-config-file onto the same host/container has the transformer code in a cloud-based (or non-laptop) deployment?

I'm not planning for that w/ the CDK deployment today, which is why I put a Base64 option in there too. Further out, it might make sense to pull it via a parameter store and write it as part of an init task. I'm going to try not bother with changing things like container initialization until I'm done w/ the K8s investigation.

Makes sense, thanks.

chelma · 2024-09-09T14:57:03Z

...cCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java

-            "--insecure" }, arity = 0, description = "Do not check the server's certificate")
+        @Parameter(
+            required = false,
+            names = {"--insecure" },


Non-blocking nitpick - unbalanced spacing inside the curly braces

chelma · 2024-09-09T14:57:08Z

...cCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java

-            REMOVE_AUTH_HEADER_VALUE_ARG }, arity = 0, description = "Remove the authorization header if present and do not replace it with anything.  "
+        @Parameter(
+            required = false,
+            names = {REMOVE_AUTH_HEADER_VALUE_ARG },


Non-blocking nitpick - unbalanced spacing inside the curly braces

chelma · 2024-09-09T14:57:21Z

...cCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java

-            AWS_AUTH_HEADER_USER_AND_SECRET_ARG }, arity = 2, description = "<USERNAME> <SECRET_ARN> pair to specify "
+        @Parameter(
+            required = false, names = {
+            AWS_AUTH_HEADER_USER_AND_SECRET_ARG },


Non-blocking nitpick - unbalanced spacing inside the curly braces

chelma · 2024-09-09T14:57:36Z

...cCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java

-        @Parameter(required = false, names = {
-            AWS_AUTH_HEADER_USER_AND_SECRET_ARG }, arity = 2, description = "<USERNAME> <SECRET_ARN> pair to specify "
+        @Parameter(
+            required = false, names = {


Non-blocking nitpick - inconsistent new lines

gregschohn · 2024-09-09T17:43:44Z

Thanks for the PR. This is my first time looking through this code (really should have investigated it earlier), and I'm struggling a bit to see how we'd take something like the HttpJsonTransformingConsumer and generalize its mechanisms to work for, say, the transformations we need for RFS. Curious what your thoughts are on that?

As a developer and a user, I would expect to see extensibility in a similar way for metadata migration and document migration. That means having a simple dependency-free API from the metadata/document migration processes to call into hot-loadable classes (via ServiceProvider). Beyond that, I'd like to see the same translation environments supported for each of our 3 migration applications.

Having a shared CompositeTransformer for all of these could be a start. That composite transformer may eventually handle conditionals and maybe is represented as a graph or tree. At that point, it should be shared by all of the applications that need to support transformations.

On the other side - what is being transformed, I'm not sure what we'll want to share. Ideally, it's a very consistent environment that develops. For example, the CompositeTransformer and TransformationLoader (that configures transformations) should be consistently shared. Admittedly, both of these are super simple/lightweight today, but complexity and value aren't necessarily correlated.

Further into the future, I'd suspect that we'll want to develop "document" migrations for metadata and backfill document migrations. These document migrations should be able to be pulled into the replayer since that's going to need to do all of the transformations of the other two systems since RESTful calls can go into the source w/ those document types. I suspect that we'll build up a library of transforms, each which may be configured. That library will be usable from all of the applications and may also allow transformations to be embedded/extended, etc.

So - things to care about for today. What are document types. Right now, for the replayer there are 3 kinds of documents - json, ndjson, binary. Is that enough? I suspect that for document and metadata migration, those will likely just need to care about json transformations. Ideally, the replayer will be able to consume those shared transformations in the future.

Signed-off-by: Greg Schohn <[email protected]> # Conflicts: # TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/datahandlers/NettyPacketToHttpConsumer.java # TrafficCapture/trafficReplayer/src/test/java/org/opensearch/migrations/replay/datahandlers/http/HttpJsonTransformingConsumerTest.java

...cCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java

...ficReplayer/src/main/java/org/opensearch/migrations/replay/datahandlers/JsonAccumulator.java

.../java/org/opensearch/migrations/replay/datahandlers/http/NettyJsonBodyAccumulateHandler.java

There were some refcount bugfixes for partial json streams and command line parsing. More importantly, there was a test (testMalformedPayload_andTypeMappingUri_IsPassedThrough) that sent malformed JSON in a payload. The test didn't do much and a comment admitted that we weren't likely doing the right thing. Moving a buffer to Unpooled caused the test to fail w/ a leak. Upon closer inspection, errors while parsing JSON didn't cause the payload contents to be dumped into the inlinedBinaryBody key of the payload. Now that content is captured and passed forward (plus, refCounted appropriately, though still as an Unpooled to minimize risk from transformations [see code comment]). In the case of this existing test, which has been moved to HttpJsonTransformingConsumerTest, the JsonTransformerForOpenSearch23PlusTargetTransformerProvider that the test uses throws an exception when the json payload isn't present. That was also not being handled properly. Now it is handled by marking the transformation as an error w/ the exception wrapped in a "TransformationException". In that case, the transformation status is marked as an error and the contents of the entire message will be empty. It feels more appropriate that if a transformation threw to be as conservative as possible and not put anything on the wire, in case some part of it was sensitive. Lastly, I refactored the AggregatedRawResponse into a base class for AggregatedRawResult. We use that class to accumulate transformation results. It was extremely confusing to see an HTTP response and the word response on so many of the fields. Now the word 'result' is used and the usage in various contexts makes more sense. Signed-off-by: Greg Schohn <[email protected]>

gregschohn added 2 commits September 7, 2024 09:01

Further improvements to transformation - most notably to support head…

1b3794f

…er transforms for docs w/out bodies or with malformed bodies. Signed-off-by: Greg Schohn <[email protected]>

gregschohn changed the title ~~Minor transformation ux help~~ Replayer Transformation support Sep 8, 2024

gregschohn force-pushed the MinorTransformationUXHelp branch from 8274fe8 to b39eb8e Compare September 8, 2024 14:13

gregschohn added 2 commits September 8, 2024 12:13

Touchup README, make a unit test a better example, and remove transfo…

4d27024

…rmations from docker-compose as the default. On the last point, I'll continue to look into differences that could be causing e2e tests to fail. Signed-off-by: Greg Schohn <[email protected]>

gregschohn force-pushed the MinorTransformationUXHelp branch from 374b82f to 4d27024 Compare September 8, 2024 18:06

gregschohn marked this pull request as ready for review September 8, 2024 18:07

gregschohn requested review from AndreKurait, chelma, lewijacn, mikaylathompson, okhasawn, peternied and sumobrian as code owners September 8, 2024 18:07

Merge branch 'main' into MinorTransformationUXHelp

20d69bd

peternied reviewed Sep 9, 2024

View reviewed changes

chelma reviewed Sep 9, 2024

View reviewed changes

AndreKurait requested changes Sep 18, 2024

View reviewed changes

AndreKurait reviewed Sep 19, 2024

View reviewed changes

.../java/org/opensearch/migrations/replay/datahandlers/http/NettyJsonBodyAccumulateHandler.java Show resolved Hide resolved

AndreKurait approved these changes Sep 19, 2024

View reviewed changes

gregschohn force-pushed the MinorTransformationUXHelp branch from 279945b to c48b6c4 Compare September 19, 2024 15:42

Merge branch 'main' into MinorTransformationUXHelp

a4a9405

gregschohn merged commit 5a8ed42 into opensearch-project:main Sep 19, 2024
14 of 15 checks passed

gregschohn deleted the MinorTransformationUXHelp branch September 20, 2024 16:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replayer Transformation support #937

Replayer Transformation support #937

gregschohn commented Sep 8, 2024 •

edited

Loading

codecov bot commented Sep 8, 2024 •

edited

Loading

peternied Sep 9, 2024

chelma Sep 9, 2024

gregschohn Sep 9, 2024

peternied Sep 9, 2024

gregschohn Sep 9, 2024

chelma left a comment

chelma Sep 9, 2024

chelma Sep 9, 2024

gregschohn Sep 9, 2024 •

edited

Loading

chelma Sep 9, 2024

chelma Sep 9, 2024

chelma Sep 9, 2024

chelma Sep 9, 2024

chelma Sep 9, 2024

gregschohn commented Sep 9, 2024

Replayer Transformation support #937

Replayer Transformation support #937

Conversation

gregschohn commented Sep 8, 2024 • edited Loading

Description

Issues Resolved

Testing

Check List

codecov bot commented Sep 8, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chelma left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gregschohn Sep 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gregschohn commented Sep 9, 2024

gregschohn commented Sep 8, 2024 •

edited

Loading

codecov bot commented Sep 8, 2024 •

edited

Loading

gregschohn Sep 9, 2024 •

edited

Loading