-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Console] Move humanReadableTuples
into console lib & add tests
#981
[Console] Move humanReadableTuples
into console lib & add tests
#981
Conversation
Signed-off-by: Mikayla Thompson <[email protected]>
Signed-off-by: Mikayla Thompson <[email protected]>
Signed-off-by: Mikayla Thompson <[email protected]>
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #981 +/- ##
============================================
+ Coverage 80.80% 80.90% +0.10%
Complexity 2725 2725
============================================
Files 400 402 +2
Lines 15408 15582 +174
Branches 971 971
============================================
+ Hits 12450 12607 +157
- Misses 2383 2400 +17
Partials 575 575
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you can tackle the nits in < 30 minutes, it would be worth it. Otherwise, maybe hold off until we think about other ways to store/process this data.
TrafficCapture/README.md
Outdated
@@ -110,27 +110,46 @@ The body of the messages is sometimes gzipped which makes it difficult to repres | |||
and responses is base64 encoded before it is logged. This makes the files stable, but not human-readable. | |||
|
|||
We have provided a utility script that can parse these files and output them to a human-readable format: the bodies are | |||
base64 decoded, un-gzipped if applicable, and parsed as JSON if applicable. They're then saved back to JSON format on disk. | |||
base64 decoded and parsed as JSON if applicable. They're then saved back to JSON format to stdout or disk. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: s/or disk/or file/
@@ -0,0 +1,49 @@ | |||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit - how is this tuple invalid? Maybe rename the file to be slightly more specific.
@@ -0,0 +1,4 @@ | |||
{"sourceRequest": {"Request-URI": "/", "Method": "GET", "HTTP-Version": "HTTP/1.1", "Host": "capture-proxy:9200", "User-Agent": "python-requests/2.32.3", "Accept-Encoding": "gzip, deflate, zstd", "Accept": "*/*", "Connection": "keep-alive", "Authorization": "Basic YWRtaW46YWRtaW4=", "body": ""}, "sourceResponse": {"HTTP-Version": {"keepAliveDefault": true}, "Status-Code": 200, "Reason-Phrase": "OK", "response_time_ms": 374, "content-type": "application/json; charset=UTF-8", "content-length": "538", "body": {"name": "3cc068ad54eb", "cluster_name": "docker-cluster", "cluster_uuid": "PIAawDwSQKSRLtoaXC-oWg", "version": {"number": "7.10.2", "build_flavor": "oss", "build_type": "docker", "build_hash": "747e1cc71def077253878a59143c1f785afa92b9", "build_date": "2021-01-13T00:42:12.435326Z", "build_snapshot": false, "lucene_version": "8.7.0", "minimum_wire_compatibility_version": "6.8.0", "minimum_index_compatibility_version": "6.0.0-beta1"}, "tagline": "You Know, for Search"}}, "targetRequest": {"Request-URI": "/", "Method": "GET", "HTTP-Version": "HTTP/1.1", "Host": "opensearchtarget", "User-Agent": "python-requests/2.32.3", "Accept-Encoding": "gzip, deflate, zstd", "Accept": "*/*", "Connection": "keep-alive", "Authorization": "Basic YWRtaW46bXlTdHJvbmdQYXNzd29yZDEyMyE=", "body": ""}, "targetResponses": [{"HTTP-Version": {"keepAliveDefault": true}, "Status-Code": 200, "Reason-Phrase": "OK", "response_time_ms": 341, "content-type": "application/json; charset=UTF-8", "content-length": "568", "body": {"name": "e6b3a4370763", "cluster_name": "docker-cluster", "cluster_uuid": "88zlik7ORFuFT78zduradA", "version": {"distribution": "opensearch", "number": "2.15.0", "build_type": "tar", "build_hash": "61dbcd0795c9bfe9b81e5762175414bc38bbcadf", "build_date": "2024-06-20T03:27:32.562036890Z", "build_snapshot": false, "lucene_version": "9.10.0", "minimum_wire_compatibility_version": "7.10.0", "minimum_index_compatibility_version": "7.0.0"}, "tagline": "The OpenSearch Project: https://opensearch.org/"}}], "connectionId": "0242acfffe12000b-0000000a-00000003-4c7bb8f4aefa3189-c0544dbf.0", "numRequests": 1, "numErrors": 0} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: consider pretty-formatting all of these, maybe one per file and put them into a directory structure. Then you can cons them together with the python equivalent of (cat foo/*).
It's tough to figure out what the errors are in each one. Being able to name each will let the developer know what the faults or examples are for too.
Signed-off-by: Mikayla Thompson <[email protected]>
Signed-off-by: Mikayla Thompson <[email protected]>
Signed-off-by: Mikayla Thompson <[email protected]>
Description
I scrapped the old
humanReadableTuples.py
file, added the logic to the console library, removed the unnecessary parts (the replayer now handles un-chunking & un-gzipping), and added tests.Follow-up task
(will create an issue for it)
Add a tuple conversion/check step to the E2E tests
Issues Resolved
Should fix roughly 150 lines of uncovered code, plus some bugs. I suspect there's a JIRA about this as well. (Edit: https://opensearch.atlassian.net/browse/MIGRATIONS-1325)
There were also two or three Sonarqube "code smells" associated with the former file that are resolved.
Testing
Unit tests and a integration test at the CLI command level that converts a real (small) tuple file generated by the replayer.
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.