Basic TransformationRule: IndexMappingTypeRemoval #783

peternied · 2024-06-28T17:34:06Z

Description

Adding in a formal transformation rule into the codebase, these should be applicable with in RFS, TrafficCapture, or other systems by supporting the Entity interface on types that are supported. Created and wired up IndexMappingTypeRemoval into RFS.

Note: This is a minimal building block for future metadata / transformation work; replayer support, refactoring existing transformations, version management, rules grouping, audit entries are out of scope for this pull request. There can be considerable investment in this space built off of this change.

Testing

Example test cases for the transformer and end to end test to support.

Check List

New functionality includes testing
- All tests pass, including unit test, integration test and doctest
New functionality has been documented
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Peter Nied <[email protected]>

…l_of_types Signed-off-by: Peter Nied <[email protected]>

Signed-off-by: Peter Nied <[email protected]>

DocumentsFromSnapshotMigration/src/main/java/com/rfs/RfsMigrateDocuments.java

RFS/src/main/java/com/rfs/models/GlobalMetadata.java

RFS/src/main/java/com/rfs/transformers/Transformer_ES_6_8_to_OS_2_11.java

RFS/src/main/java/com/rfs/transformers/Transformer_ES_7_10_OS_2_11.java

RFS/src/main/java/com/rfs/version_es_6_8/IndexMetadataData_ES_6_8.java

coreUtilities/build.gradle

coreUtilities/src/main/java/org/opensearch/migrations/transformation/entity/Index.java

...es/src/main/java/org/opensearch/migrations/transformation/rules/IndexMappingTypeRemoval.java

chelma

Overall, I like where things are going. Had some comments/questions though. Thanks!

RFS/src/main/java/com/rfs/models/GlobalMetadata.java

RFS/src/main/java/com/rfs/transformers/Transformer.java

RFS/src/main/java/com/rfs/transformers/Transformer_ES_6_8_to_OS_2_11.java

RFS/src/test/java/com/rfs/integration/EndToEndTest.java

...es/src/main/java/org/opensearch/migrations/transformation/rules/IndexMappingTypeRemoval.java

Signed-off-by: Peter Nied <[email protected]>

RFS/src/main/java/com/rfs/transformers/Transformer_ES_6_8_to_OS_2_11.java

Signed-off-by: Peter Nied <[email protected]>

RFS/src/main/java/com/rfs/transformers/Transformer_ES_6_8_to_OS_2_11.java

RFS/src/main/java/com/rfs/version_os_2_11/IndexCreator_OS_2_11.java

Signed-off-by: Peter Nied <[email protected]>

…e test Signed-off-by: Peter Nied <[email protected]>

…her version updates looks like a bad version container doesn't start Signed-off-by: Peter Nied <[email protected]>

Signed-off-by: Peter Nied <[email protected]>

Missing ' RFS Dynamically selects transformer based on target cluster type', tracked with https://opensearch.atlassian.net/browse/MIGRATIONS-1827 Signed-off-by: Peter Nied <[email protected]>

Signed-off-by: Peter Nied <[email protected]>

Follow up issue https://opensearch.atlassian.net/browse/MIGRATIONS-1827 Signed-off-by: Peter Nied <[email protected]>

Signed-off-by: Peter Nied <[email protected]>

chelma

Thanks for all the work on this one!

RFS/src/main/java/com/rfs/version_es_6_8/IndexMetadataData_ES_6_8.java

RFS/src/main/java/com/rfs/models/IndexMetadata.java

Signed-off-by: Peter Nied <[email protected]>

peternied added 29 commits June 6, 2024 22:50

Draft doc

d1615eb

Signed-off-by: Peter Nied <[email protected]>

Editing pass and more details about rules engine

44b0a7e

Signed-off-by: Peter Nied <[email protected]>

Include reference link

cf62c1b

Signed-off-by: Peter Nied <[email protected]>

Move prototype code into files

e06343e

Signed-off-by: Peter Nied <[email protected]>

traffic capture request processing

8545bab

Signed-off-by: Peter Nied <[email protected]>

Seperate metadata migration doc

403561f

Signed-off-by: Peter Nied <[email protected]>

Initial changes

af4f095

Signed-off-by: Peter Nied <[email protected]>

Rename types

991b9ab

Signed-off-by: Peter Nied <[email protected]>

Merge remote-tracking branch 'origin/main' into transformations

ff7bcba

Signed-off-by: Peter Nied <[email protected]>

Move into the migrations folder

49e8555

Signed-off-by: Peter Nied <[email protected]>

Test stubs

d73c257

Signed-off-by: Peter Nied <[email protected]>

Add interfacce for groupings

d279c40

Signed-off-by: Peter Nied <[email protected]>

IndexMapping Type Removal can apply check

8918ccd

Signed-off-by: Peter Nied <[email protected]>

Initial transformer and tests

1699bb8

Signed-off-by: Peter Nied <[email protected]>

Shift all transformation code into core utils

cd9c805

Signed-off-by: Peter Nied <[email protected]>

Remove static data classes

fdb394e

Signed-off-by: Peter Nied <[email protected]>

Merge remote-tracking branch 'origin/main' into transformation_remova…

5c565e9

…l_of_types Signed-off-by: Peter Nied <[email protected]>

Cleaning up a handful more of the parts of the refactor

93d6049

Signed-off-by: Peter Nied <[email protected]>

Merge branch 'transformation_removal_of_types' into transformations

863cb31

Signed-off-by: Peter Nied <[email protected]>

Use transformations for type mapping removal

c1cda29

Signed-off-by: Peter Nied <[email protected]>

Remove unneeded test file

e45f1b3

Signed-off-by: Peter Nied <[email protected]>

migrateFrom_ES_v6_8 is working!

c09409b

Signed-off-by: Peter Nied <[email protected]>

Tests prints some proof of what it did

1a46fae

Signed-off-by: Peter Nied <[email protected]>

Cleanup test case a bit

b21dd5c

Signed-off-by: Peter Nied <[email protected]>

Merge remote-tracking branch 'origin/main' into transformations

443df79

Signed-off-by: Peter Nied <[email protected]>

Clean up container naming

fc000c6

Signed-off-by: Peter Nied <[email protected]>

Cleaning up core

bcbd213

Signed-off-by: Peter Nied <[email protected]>

Cleanup white space and javadocs

4f78c43

Signed-off-by: Peter Nied <[email protected]>

Minor doc update

e423390

Signed-off-by: Peter Nied <[email protected]>

peternied requested a review from AndreKurait as a code owner June 28, 2024 17:34

peternied added 3 commits June 28, 2024 18:03

Minimized unneeded changes

78a31ea

Signed-off-by: Peter Nied <[email protected]>

Fix invalid characters in javadoc

b1eff64

Signed-off-by: Peter Nied <[email protected]>

Fix build break

cbdb405

Signed-off-by: Peter Nied <[email protected]>

gregschohn reviewed Jul 1, 2024

View reviewed changes

chelma requested changes Jul 1, 2024

View reviewed changes

peternied added 2 commits July 2, 2024 15:32

Remove switch to RuntimeException

e061abe

Signed-off-by: Peter Nied <[email protected]>

Updates from PR feedback

b2af659

Signed-off-by: Peter Nied <[email protected]>

chelma reviewed Jul 3, 2024

View reviewed changes

RFS/src/main/java/com/rfs/transformers/Transformer_ES_6_8_to_OS_2_11.java Show resolved Hide resolved

RFS/src/main/java/com/rfs/transformers/Transformer_ES_6_8_to_OS_2_11.java Outdated Show resolved Hide resolved

peternied added 7 commits July 8, 2024 17:35

Seperate list of index and index template transformations

d713deb

Signed-off-by: Peter Nied <[email protected]>

Rename raw -> rawJson to make it clearer what is being actioned on

7ae3e66

Signed-off-by: Peter Nied <[email protected]>

Merge remote-tracking branch 'origin/main' into transformer-min

5635e09

Signed-off-by: Peter Nied <[email protected]>

Move to new transformation high level gradle project

3016306

Signed-off-by: Peter Nied <[email protected]>

Fix build break

0ccec1b

Signed-off-by: Peter Nied <[email protected]>

Fix broken test case due to lack of OS 1.3 support

3e4af5e

Signed-off-by: Peter Nied <[email protected]>

Refactor parameters for Connection Details into a shard location

a8fce88

Signed-off-by: Peter Nied <[email protected]>

chelma reviewed Jul 8, 2024

View reviewed changes

peternied added 9 commits July 8, 2024 21:54

Wrong type used

677eeab

Signed-off-by: Peter Nied <[email protected]>

Fix error where reindexed-logs appears on the cluster unrelated to th…

f8f7dd2

…e test Signed-off-by: Peter Nied <[email protected]>

Lock down all docker versions, jaeger saw a push to latest with no ot…

65dab08

…her version updates looks like a bad version container doesn't start Signed-off-by: Peter Nied <[email protected]>

Remove sneaky 'f' character

ea275a1

Signed-off-by: Peter Nied <[email protected]>

Disable testing against targeting 1.3

00fe75f

Missing ' RFS Dynamically selects transformer based on target cluster type', tracked with https://opensearch.atlassian.net/browse/MIGRATIONS-1827 Signed-off-by: Peter Nied <[email protected]>

Merge branch 'reliable-cat-indices' into transformer-min

caf639f

Signed-off-by: Peter Nied <[email protected]>

Merge branch 'reliable-cat-indices' into transformer-min

8b50d82

Signed-off-by: Peter Nied <[email protected]>

Stop running end to end tests on OpenSearch 1.3

eaa439c

Follow up issue https://opensearch.atlassian.net/browse/MIGRATIONS-1827 Signed-off-by: Peter Nied <[email protected]>

Transition from old to new types

a750ff9

Signed-off-by: Peter Nied <[email protected]>

gregschohn approved these changes Jul 9, 2024

View reviewed changes

chelma approved these changes Jul 10, 2024

View reviewed changes

RFS/src/main/java/com/rfs/version_es_6_8/IndexMetadataData_ES_6_8.java Show resolved Hide resolved

RFS/src/main/java/com/rfs/models/IndexMetadata.java Show resolved Hide resolved

Merge from main

65d6cef

Signed-off-by: Peter Nied <[email protected]>

peternied merged commit c6b1cc9 into opensearch-project:main Jul 10, 2024
11 of 13 checks passed

peternied deleted the transformer-min branch July 10, 2024 15:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Basic TransformationRule: IndexMappingTypeRemoval #783

Basic TransformationRule: IndexMappingTypeRemoval #783

peternied commented Jun 28, 2024

chelma left a comment

chelma left a comment

Basic TransformationRule: IndexMappingTypeRemoval #783

Basic TransformationRule: IndexMappingTypeRemoval #783

Conversation

peternied commented Jun 28, 2024

Description

Testing

Check List

chelma left a comment

Choose a reason for hiding this comment

chelma left a comment

Choose a reason for hiding this comment