Snapshot status #757

AndreKurait · 2024-06-21T21:02:16Z

Description

Implements several changes for performance and usability:

Implements Snapshot Status
Adds Snapshot Max Throughput parameter to Create
Makes Snapshot Create Asynchronous with --wait option
Modifies RFS to support ES 7.17
Changes the RFS ECS Worker to use the right document migration main class
Adds Bash-Completion to migration console with support for console-link library
Fixes RFS logging pattern in Cloudwatch

Category: New Features
Why these changes are required? Usability improvements for RFS and Migration Console
What is the old behavior before changes and new behavior after changes?

Issues Resolved

MIGRATIONS-1792

Is this a backport? If so, please add backport PR # and/or commits #

Testing

Lots of manual testing locally and in the cloud

Follow up for more documentation and E2E cloud testing https://opensearch.atlassian.net/browse/MIGRATIONS-1808

Check List

New functionality includes testing
- All tests pass, including unit test, integration test and doctest
New functionality has been documented
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov · 2024-06-21T21:03:44Z

Codecov Report

Attention: Patch coverage is 70.58824% with 45 lines in your changes missing coverage. Please review.

Project coverage is 88.13%. Comparing base (73f2092) to head (32b0d9f).
Report is 3 commits behind head on main.

Files	Patch %	Lines
...e/lib/console_link/console_link/models/snapshot.py	73.27%	31 Missing ⚠️
...rationConsole/lib/console_link/console_link/cli.py	61.53%	10 Missing ⚠️
...le/lib/console_link/console_link/logic/snapshot.py	50.00%	4 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##               main     #757       +/-   ##
=============================================
+ Coverage     68.50%   88.13%   +19.63%     
=============================================
  Files           270       48      -222     
  Lines         11175     3059     -8116     
  Branches        736        0      -736     
=============================================
- Hits           7655     2696     -4959     
+ Misses         3118      363     -2755     
+ Partials        402        0      -402

Flag	Coverage Δ
gradle-test	`?`
python-test	`88.13% <70.58%> (-0.75%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mikaylathompson

Looks good! A couple questions, but no blockers.

mikaylathompson · 2024-06-25T15:14:11Z

RFS/src/main/java/com/rfs/common/SnapshotShardUnpacker.java

@@ -72,40 +70,10 @@ public Path unpack() {
        }
    }

-    @Override
-    public void close() {


do we care about these things not getting cleaned up?

No, we want to recycle tasks between shards with ECS. When this was in there, we were deleting the shard date before we reindexed it

mikaylathompson · 2024-06-25T15:20:18Z

...erSolution/src/main/docker/migrationConsole/lib/console_link/console_link/models/snapshot.py

+    IN_PROGRESS = "IN_PROGRESS"
+
+
+def convert_snapshot_state_to_status(snapshot_state: str) -> Tuple[SnapshotStatus, str]:


I'm not sure I understand why we should have State & Status and what the difference is. Should we just use these States throughout?

State is the cluster response enum whereas Status is what we define. I had originally defined it when we were also storing status in a separate doc.

This helps us maintain a consistent Status despite state renaming across ES/OS versions

This doesn't feel right, we are embedding version handling deep in the console. I understand getting this in position for the demo, but this blends the responsibilities of the system quite a bit - it is going to make our codebase hard to maintain very quickly.

@AndreKurait What do you think of this, how should we capture the responsibility questions?

Moved this to just return the state string that the cluster responds with

mikaylathompson · 2024-06-25T15:24:40Z

...ficCapture/dockerSolution/src/main/docker/migrationConsole/loadServicesFromParameterStore.sh

+# Add sourcing of the completion script to .bashrc for persistence across sessions
+echo '. /etc/bash_completion' >> ~/.bashrc
+
+echo "Bash completion for console command has been set up and enabled."


Thanks for adding this!

Signed-off-by: Andre Kurait <[email protected]> Add Snapshot Status updates Signed-off-by: Andre Kurait <[email protected]> Snapshot improvements Signed-off-by: Andre Kurait <[email protected]> Use new Rfs Document Migration Main Signed-off-by: Andre Kurait <[email protected]> Fix create snapshot Signed-off-by: Andre Kurait <[email protected]> Updates Signed-off-by: Andre Kurait <[email protected]> Improvements Signed-off-by: Andre Kurait <[email protected]> Improve RFS log4j2.xml Signed-off-by: Andre Kurait <[email protected]> revert documentReindexer Signed-off-by: Andre Kurait <[email protected]> add logging for documentReindexer Signed-off-by: Andre Kurait <[email protected]> remove logging Signed-off-by: Andre Kurait <[email protected]> Update printing based on feedback Signed-off-by: Andre Kurait <[email protected]>

Signed-off-by: Andre Kurait <[email protected]>

peternied · 2024-06-25T17:02:56Z

...erSolution/src/main/docker/migrationConsole/lib/console_link/console_link/models/snapshot.py

+    IN_PROGRESS = "IN_PROGRESS"
+
+
+def convert_snapshot_state_to_status(snapshot_state: str) -> Tuple[SnapshotStatus, str]:


This doesn't feel right, we are embedding version handling deep in the console. I understand getting this in position for the demo, but this blends the responsibilities of the system quite a bit - it is going to make our codebase hard to maintain very quickly.

@AndreKurait What do you think of this, how should we capture the responsibility questions?

...erSolution/src/main/docker/migrationConsole/lib/console_link/console_link/models/snapshot.py

Signed-off-by: Andre Kurait <[email protected]>

peternied

Thanks for the updates @AndreKurait

AndreKurait force-pushed the SnapshotStatus branch 6 times, most recently from bdff322 to f268d4c Compare June 25, 2024 02:19

AndreKurait marked this pull request as ready for review June 25, 2024 15:03

AndreKurait requested review from chelma, gregschohn, lewijacn, mikaylathompson, okhasawn, peternied and sumobrian as code owners June 25, 2024 15:03

mikaylathompson approved these changes Jun 25, 2024

View reviewed changes

AndreKurait force-pushed the SnapshotStatus branch 3 times, most recently from e7597ba to 3351c06 Compare June 25, 2024 15:55

AndreKurait added 5 commits June 25, 2024 10:57

Add bash completion for migration console link cli

8690b9d

Signed-off-by: Andre Kurait <[email protected]>

Add Snapshot Max Throughput

0baf011

Signed-off-by: Andre Kurait <[email protected]>

Fix imports in snapshot

fdf77f8

Signed-off-by: Andre Kurait <[email protected]>

Fix linting

656d36e

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait force-pushed the SnapshotStatus branch 5 times, most recently from af22643 to 2c3058b Compare June 25, 2024 16:30

Update snapshot tests

95aa33b

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait force-pushed the SnapshotStatus branch from 2c3058b to 95aa33b Compare June 25, 2024 16:34

peternied requested changes Jun 25, 2024

View reviewed changes

Remove snapshot status enum

2a45284

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait force-pushed the SnapshotStatus branch 4 times, most recently from ae734ee to 57016fc Compare June 25, 2024 17:57

Add Test for Snapshot Status

32b0d9f

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait force-pushed the SnapshotStatus branch from 57016fc to 32b0d9f Compare June 25, 2024 17:59

peternied approved these changes Jun 25, 2024

View reviewed changes

AndreKurait merged commit b62d0a3 into opensearch-project:main Jun 25, 2024
13 checks passed

AndreKurait deleted the SnapshotStatus branch June 25, 2024 18:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Snapshot status #757

Snapshot status #757

AndreKurait commented Jun 21, 2024 •

edited

Loading

codecov bot commented Jun 21, 2024 •

edited

Loading

mikaylathompson left a comment

mikaylathompson Jun 25, 2024

AndreKurait Jun 25, 2024

mikaylathompson Jun 25, 2024

AndreKurait Jun 25, 2024

peternied Jun 25, 2024

AndreKurait Jun 25, 2024

mikaylathompson Jun 25, 2024

peternied Jun 25, 2024

peternied left a comment

		IN_PROGRESS = "IN_PROGRESS"


		def convert_snapshot_state_to_status(snapshot_state: str) -> Tuple[SnapshotStatus, str]:

Snapshot status #757

Snapshot status #757

Conversation

AndreKurait commented Jun 21, 2024 • edited Loading

Description

Issues Resolved

Testing

Check List

codecov bot commented Jun 21, 2024 • edited Loading

Codecov Report

mikaylathompson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peternied left a comment

Choose a reason for hiding this comment

AndreKurait commented Jun 21, 2024 •

edited

Loading

codecov bot commented Jun 21, 2024 •

edited

Loading