Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fetch Migration] Enable migration for identical, empty target cluster index #390

Merged
merged 7 commits into from
Nov 11, 2023
Merged

[Fetch Migration] Enable migration for identical, empty target cluster index #390

merged 7 commits into from
Nov 11, 2023

Commits on Nov 10, 2023

  1. doc_count now doc-counts by index 

    This is encapsulated in an IndexDocCount class, which also track the total doc count

Signed-off-by: Kartik Ganesh <[email protected]>
    @kartg
    kartg committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    87bea49 View commit details
    Browse the repository at this point in the history

  2. [Fetch Migration] Support data migration for identical, empty target … 

    …cluster indices

This change includes refactoring the index difference logic to an IndexDiff class, and an additional network call to the target cluster to fetch document counts for identical indices to determine which ones are empty.

Signed-off-by: Kartik Ganesh <[email protected]>
    @kartg
    kartg committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    84f6949 View commit details
    Browse the repository at this point in the history

  3. [Fetch Migration] Tweaks to pipeline template file and task definition 

    This change adds documentation to the dp_pipeline_template.yaml file and changes some reasonable defaults. It also bumps the Fetch Migration task definition to 1 CPu and 4 GB memory.

Signed-off-by: Kartik Ganesh <[email protected]>
    @kartg
    kartg committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    a6a5f15 View commit details
    Browse the repository at this point in the history

  4. Bugfix to jsonpath library name in requirements.txt 

    Signed-off-by: Kartik Ganesh <[email protected]>
    @kartg
    kartg committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    e5522ce View commit details
    Browse the repository at this point in the history

  5. Minor - add unit test for endpoint_info URL normalization 

    Signed-off-by: Kartik Ganesh <[email protected]>
    @kartg
    kartg committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    7f57689 View commit details
    Browse the repository at this point in the history

  6. Incorporating PR feedback 

    Signed-off-by: Kartik Ganesh <[email protected]>
    @kartg
    kartg committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    54bead7 View commit details
    Browse the repository at this point in the history

  7. [Bugfix] Use 'r+' instead of 'rw' in fetch_orchestrator.py 

    Signed-off-by: Kartik Ganesh <[email protected]>
    @kartg
    kartg committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    d327ab5 View commit details
    Browse the repository at this point in the history