-
Notifications
You must be signed in to change notification settings - Fork 27
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Added Elasticsearch 6.8 support to Document Migrations (#921)
* Added a speculative 5.6 docker-compose to RFS Signed-off-by: Chris Helma <[email protected]> * Checkpoint: adding version-specification to doc migration Signed-off-by: Chris Helma <[email protected]> * Created a 5.6 snapshot w/ updates & deletes, put in RFS test-resources Signed-off-by: Chris Helma <[email protected]> * Added 2 ES 6.8 snapshots to facilitate RFS testing Signed-off-by: Chris Helma <[email protected]> * Added a native ES 6.8 snapshot compatible w/ Lucene 8.X Signed-off-by: Chris Helma <[email protected]> * Unit test updates Signed-off-by: Chris Helma <[email protected]> * Removed an unnecessary local snapshot Signed-off-by: Chris Helma <[email protected]> * Ran spotlessApply Signed-off-by: Chris Helma <[email protected]> * Updates per PR comments Signed-off-by: Chris Helma <[email protected]> * Added ES 6.8 to RFS EndToEnd tests; refactored those tests Signed-off-by: Chris Helma <[email protected]> * More updates per PR comments Signed-off-by: Chris Helma <[email protected]> --------- Signed-off-by: Chris Helma <[email protected]>
- Loading branch information
Showing
76 changed files
with
996 additions
and
540 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
20 changes: 20 additions & 0 deletions
20
DocumentsFromSnapshotMigration/docker/TestSource_ES_5_6/Dockerfile
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
FROM docker.elastic.co/elasticsearch/elasticsearch:5.6.16 AS base | ||
|
||
# Configure Elastic | ||
ENV ELASTIC_SEARCH_CONFIG_FILE=/usr/share/elasticsearch/config/elasticsearch.yml | ||
RUN echo "discovery.type: single-node" >> $ELASTIC_SEARCH_CONFIG_FILE | ||
RUN echo "xpack.security.enabled: false" >> $ELASTIC_SEARCH_CONFIG_FILE | ||
RUN echo "bootstrap.system_call_filter: false" >> $ELASTIC_SEARCH_CONFIG_FILE | ||
ENV PATH=${PATH}:/usr/share/elasticsearch/jdk/bin/ | ||
|
||
# Make our snapshot directory | ||
USER root | ||
RUN mkdir /snapshots && chown elasticsearch /snapshots | ||
USER elasticsearch | ||
|
||
# We do not install the S3 Repo plugin here, because it is not compatible with modern | ||
# IAM Roles. Specifically, it does not support the AWS_SESSION_TOKEN environment variable. | ||
# We will instead take snapshots into a mounted local volume. | ||
|
||
# Additionally, we will rely on the base image's default entrypoint command to start the | ||
# Elasticsearch service. |
47 changes: 47 additions & 0 deletions
47
DocumentsFromSnapshotMigration/docker/docker-compose-es56.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,47 @@ | ||
version: '3.7' | ||
services: | ||
|
||
elasticsearchsource: | ||
image: 'migrations/empty_elasticsearch_source_5_6:latest' | ||
platform: linux/amd64 | ||
networks: | ||
- migrations | ||
environment: | ||
- path.repo=/snapshots | ||
- AWS_ACCESS_KEY_ID=${access_key} | ||
- AWS_SECRET_ACCESS_KEY=${secret_key} | ||
- AWS_SESSION_TOKEN=${session_token} | ||
ports: | ||
- '19200:9200' | ||
volumes: | ||
- ./snapshots:/snapshots | ||
|
||
reindex-from-snapshot: | ||
image: 'migrations/reindex_from_snapshot:latest' | ||
depends_on: | ||
elasticsearchsource: | ||
condition: service_started | ||
opensearchtarget: | ||
condition: service_started | ||
networks: | ||
- migrations | ||
environment: | ||
- AWS_ACCESS_KEY_ID=${access_key} | ||
- AWS_SECRET_ACCESS_KEY=${secret_key} | ||
- AWS_SESSION_TOKEN=${session_token} | ||
volumes: | ||
- ./snapshots:/snapshots | ||
|
||
opensearchtarget: | ||
image: 'opensearchproject/opensearch:2.11.1' | ||
environment: | ||
- discovery.type=single-node | ||
- plugins.security.disabled=true | ||
networks: | ||
- migrations | ||
ports: | ||
- "29200:9200" | ||
|
||
networks: | ||
migrations: | ||
driver: bridge |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.