Flexible Transformation Framework for Multi-Hop and Post-Fork Migrations

[sumobrian]

Is your feature request related to a problem?

Request and response transformations are needed to support changes from the source to the target. For example, indices created in Elasticsearch 7.0 and later could only contain a single mapping type. This change aimed to simplify the data model and improve performance and maintainability. Therefore, if you're migrating from Elasticsearch 6.x to a version beyond 7.0 (such as OpenSearch 1.x), you should be aware that the concept of multiple types within an index is no longer supported, and your mappings need to be adjusted accordingly to fit the single-type-per-index model.

Transformations on responses should also be supported. This serves multiple use cases, including extracting sensitive data when saving it to disk and being able to maintain a generated key on the source while propagating data to the target.

What solution would you like?

To support request and response transformations as part of multi-hop migrations (e.g., migrating from Elasticsearch 6.x to OpenSearch 2.x), the proposal is to develop a transformation framework within OpenSearch Migrations. This framework should allow the definition of reusable transformations for both requests and responses, ensuring data consistency and compliance across different versions and platforms. The goal is to build a suite of transformations that simplify the migration process, with support for community contributions to expand and refine these transformations over time.

To achieve this, the framework should:

• Provide predefined transformations for common version-based changes (e.g., converting multi-type mappings to single-type).
• Allow users to define custom transformation logic using a well-documented API or scripting mechanism.
• Include a validation mechanism to ensure transformations are applied correctly.
• Offer detailed guidelines and templates to encourage external contributions, making it easier for the community to add support for additional transformations.
• Complete solution support. That is, support added to Metadata Migrations, Capture and Replay, and Reindex from Snapshot

What alternatives have you considered?

• Application-Level Transformations: Implementing transformation logic directly in the application code, but this approach requires more custom development that may be applied inconsistently across migration products. In addition, it is not easily extended.

Do you have any additional context?

This framework is crucial for supporting multi-hop migrations (e.g., Elasticsearch 6.8 to OpenSearch 2.x) and post-fork migrations (For example, Elasticsearch 8.x to OpenSearch 2.x). By enabling request and response transformations, the framework can help bridge the gap between different Elasticsearch versions and OpenSearch. Additionally, by making the transformation logic open and extensible, we aim to build a community-driven suite of transformations that adapt to various use cases and scenarios over time.