Updates to Orchestrator API for OSI #805

lewijacn · 2024-07-09T22:02:23Z

Description

This changes introduces the following additions to the Console Orchestrator API

Update OSI Create API model to use URI field instead of host and port and remove unnecessary migration type field
Add serializer to verify received data for all API actions
Add delete action as another OSI API action
Add assume role ability to all OSI actions, this allows passing in a separate role for creating/updating/deleting pipelines that is different than the default task role of the container

Issues Resolved

https://opensearch.atlassian.net/browse/MIGRATIONS-1843

Testing

Unit testing, Cloud testing

Check List

New functionality includes testing
- All tests pass, including unit test, integration test and doctest
New functionality has been documented
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…ine pattern, add assume role for all actions Signed-off-by: Tanner Lewis <[email protected]>

codecov · 2024-07-09T22:03:33Z

Codecov Report

Attention: Patch coverage is 98.42520% with 2 lines in your changes missing coverage. Please review.

Project coverage is 88.45%. Comparing base (88b2de7) to head (6dfa278).
Report is 15 commits behind head on main.

Files	Patch %	Lines
...console_api/console_api/apps/orchestrator/views.py	93.93%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #805      +/-   ##
==========================================
+ Coverage   88.18%   88.45%   +0.26%     
==========================================
  Files          56       67      +11     
  Lines        3478     3723     +245     
==========================================
+ Hits         3067     3293     +226     
- Misses        411      430      +19

Flag	Coverage Δ
python-test	`88.39% <100.00%> (+0.20%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Tanner Lewis <[email protected]>

# Conflicts: # TrafficCapture/dockerSolution/src/main/docker/migrationConsole/lib/console_link/tests/test_osi_utils.py

Signed-off-by: Tanner Lewis <[email protected]>

mikaylathompson

A few questions, but no major concerns

mikaylathompson · 2024-07-18T16:41:17Z

.github/workflows/CI.yml

+      - uses: codecov/codecov-action@v4
+        with:
+          files: ./coverage.xml
+


Note for a follow up change in the sonarqube stuff to incorporate this

Created a JIRA issue: Add Console API test coverage to Sonarqube

Thanks for creating this, yes we need to add

mikaylathompson · 2024-07-18T16:43:25Z

TrafficCapture/dockerSolution/src/main/docker/migrationConsole/console_api/Pipfile

@@ -13,6 +13,8 @@ sqlparse = "==0.5.0"
 pyopenssl = "*"

 [dev-packages]
+coverage = "*"
+moto = "*"


It looks like you can install moto with just the specific dependencies you want to mock (https://docs.getmoto.org/en/latest/docs/getting_started.html#installing-moto) -- worth doing? I assume we're mocking a pretty narrow list

Went ahead and added the specifier for sts for moto, from my testing all decorators like mock_sts have been removed in favor of using the single mock_aws

mikaylathompson · 2024-07-18T16:49:04Z

...c/main/docker/migrationConsole/console_api/console_api/apps/orchestrator/tests/test_views.py

+}
+
+
+# Moto has not yet implemented mocking for OSI API actions, using @patch for these calls


So moto is basically handling/mocking creating the boto3 client? It looks like we could add the OSI actions directly to moto (like https://docs.getmoto.org/en/latest/docs/services/patching_other_services.html or via PR to their repo), but I think this approach is fine with me for now

Switched to using patch for moto like you mentioned to give a little more flexibility here. Thanks for suggestion

mikaylathompson · 2024-07-18T16:52:39Z

...Solution/src/main/docker/migrationConsole/console_api/console_api/apps/orchestrator/views.py

+        create_pipeline_from_json(osi_client=determine_osi_client(request_data=request_data, action_name=action_name),
+                                  input_json=request_data,
+                                  pipeline_template_path=PIPELINE_TEMPLATE_PATH)
+    except InvalidAuthParameters as i:


Minor note that at some point, it would be great to have a test that exercised this branch, but not a blocker for this PR

Have added a test for this now 👍

mikaylathompson · 2024-07-18T16:53:22Z

...Solution/src/main/docker/migrationConsole/console_api/console_api/apps/orchestrator/views.py

+def osi_start_migration(request):
+    return osi_update_workflow(request=request, osi_action_func=start_pipeline, action_name='Start')
+
+
 @api_view(['POST'])
 @parser_classes([JSONParser])
 def osi_stop_migration(request):
-    request_data = request.data
-    logger.info(pretty_request(request, request_data))
+    return osi_update_workflow(request=request, osi_action_func=stop_pipeline, action_name='Stop')


Is there/should there be validation that the pipeline exists for these actions? What happens if it doesn't?

I imagine users will get a 500 with the error message, that the pipeline doesn't exist. But agree this is likely a common error case we should test for and be more specific for like a 404 response code. Have created a task around this here: https://opensearch.atlassian.net/browse/MIGRATIONS-1877

mikaylathompson · 2024-07-18T16:54:55Z

...rSolution/src/main/docker/migrationConsole/lib/console_link/console_link/models/osi_utils.py

@@ -57,6 +58,20 @@ def __init__(self, config: Dict) -> None:
            self.tags = convert_str_tags_to_dict(tags)


+def get_assume_role_session(role_arn, session_name) -> boto3.Session:


I'd love to see an example of what the caller sees if this fails. It seems like one of the more likely to fail pieces, given the general complication of IAM stuff.

From my testing, users will get a 500 and see the boto3 error message along the lines of Unable to assume role: .... for x reason. Which may be clear enough for now, but potentially something we could make more obvious

Signed-off-by: Tanner Lewis <[email protected]>

Update API model for OSI, add serializer for update, add delete pipel…

059c5ca

…ine pattern, add assume role for all actions Signed-off-by: Tanner Lewis <[email protected]>

lewijacn added 8 commits July 9, 2024 20:39

Adding test cases and minor fixes

3a6e927

Signed-off-by: Tanner Lewis <[email protected]>

Update pipfile lock

25e3531

Signed-off-by: Tanner Lewis <[email protected]>

Add tests for django

0d0bc2c

Signed-off-by: Tanner Lewis <[email protected]>

Merge remote-tracking branch 'origin/main' into osi-api-updates

c25fa2a

# Conflicts: # TrafficCapture/dockerSolution/src/main/docker/migrationConsole/lib/console_link/tests/test_osi_utils.py

API testing updates

5262dac

Signed-off-by: Tanner Lewis <[email protected]>

Set default region for test cases

ac29db8

Signed-off-by: Tanner Lewis <[email protected]>

Merge remote-tracking branch 'origin/main' into osi-api-updates

15627cb

Signed-off-by: Tanner Lewis <[email protected]>

Update pipfile lock

9a3f4ec

Signed-off-by: Tanner Lewis <[email protected]>

lewijacn marked this pull request as ready for review July 16, 2024 14:51

lewijacn requested review from AndreKurait, chelma, gregschohn, mikaylathompson, okhasawn, peternied and sumobrian as code owners July 16, 2024 14:51

mikaylathompson approved these changes Jul 18, 2024

View reviewed changes

lewijacn added 2 commits July 18, 2024 19:05

Updates per PR feedback

7bb1613

Signed-off-by: Tanner Lewis <[email protected]>

Update for flake8

6dfa278

Signed-off-by: Tanner Lewis <[email protected]>

lewijacn merged commit 600022b into opensearch-project:main Jul 19, 2024
15 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updates to Orchestrator API for OSI #805

Updates to Orchestrator API for OSI #805

lewijacn commented Jul 9, 2024 •

edited

Loading

codecov bot commented Jul 9, 2024 •

edited

Loading

mikaylathompson left a comment

mikaylathompson Jul 18, 2024

mikaylathompson Jul 18, 2024

lewijacn Jul 18, 2024

mikaylathompson Jul 18, 2024

lewijacn Jul 18, 2024

mikaylathompson Jul 18, 2024

lewijacn Jul 18, 2024

mikaylathompson Jul 18, 2024

lewijacn Jul 18, 2024

mikaylathompson Jul 18, 2024

lewijacn Jul 18, 2024

mikaylathompson Jul 18, 2024

lewijacn Jul 18, 2024

		}


		# Moto has not yet implemented mocking for OSI API actions, using @patch for these calls

		@@ -57,6 +58,20 @@ def __init__(self, config: Dict) -> None:
		self.tags = convert_str_tags_to_dict(tags)


		def get_assume_role_session(role_arn, session_name) -> boto3.Session:

Updates to Orchestrator API for OSI #805

Updates to Orchestrator API for OSI #805

Conversation

lewijacn commented Jul 9, 2024 • edited Loading

Description

Issues Resolved

Testing

Check List

codecov bot commented Jul 9, 2024 • edited Loading

Codecov Report

mikaylathompson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lewijacn commented Jul 9, 2024 •

edited

Loading

codecov bot commented Jul 9, 2024 •

edited

Loading