Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert interval parameter to cron schedule for self serve replication #238

Merged

Conversation

chenselena
Copy link
Collaborator

@chenselena chenselena commented Oct 24, 2024

Summary

This PR adds cron schedule to the replication config for the self serve replication API and converts the interval parameter to a cron schedule. Cron schedule is needed as an input for the replication job that performs the cross cluster data copy. The library cronutils is added to perform the cron expression generation.

The interval parameter can be validated as 12H, 1D, 2D, 3D and the cron schedule is generated based on:

  • 12H schedule dictates replication should trigger every 12 hours
  • The cron schedule should have an X hour to start from midnight where X can range from 0-23 and Y minute where Y is in [0, 15, 30, 45]. E.g 12hours can lead to schedule “0 30 3/12 ? * * *” -> trigger at 3:30am and 3:30pm every day.
  • If interval is not provided, a daily replication schedule should be set up with daily schedules with X hour starting from midnight.
  • X is randomized from 0-23 and Y is randomized as a value of [0, 15, 30, 45] to spread out the cron and avoid job clusters around a time.

Examples of:
12H: 0 15 23/12 ? * * *
1D: 0 45 10 ? * * *
2D: 0 30 8 ? * */2 *
3D: 0 0 8 ? * */3 *

Changes

  • Client-facing API Changes
  • Internal API Changes
  • Bug Fixes
  • New Features
  • Performance Improvements
  • Code Style
  • Refactoring
  • Documentation
  • Tests

For all the boxes checked, please include additional details of the changes made in this pull request.

Testing Done

  • Manually Tested on local docker setup. Please include commands ran, and their output.
  • Added new tests for the changes made.
  • Updated existing tests to reflect the changes made.
  • No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
  • Some other form of testing like staging or soak time in production. Please explain.

For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.

Unit tests added to check that the cron schedule was created successfully.

POST http://localhost:8000/v1/databases/u_tableowner/tables for a 1D schedule:

{
    "policies": {
        "sharingEnabled": "true",
        "replication": {
            "config": [
                {
                    "destination": "clusterA"                
                 }
            ]
        }
    }
}

response:

        "replication": {
            "config": [
                {
                    "destination": "clusterA",
                    "interval": "1D",
                    "cronSchedule": "0 0 10 ? * * *"
                }
            ]
        }

12H schedule:

"replication": {
            "config": [
                {
                    "destination": "clusterA",
                    "interval": "12H",
                    "cronSchedule": "0 0 23/12 ? * * *"
                }
            ]
        }

Additional Information

  • Breaking Changes
  • Deprecations
  • Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

@chenselena chenselena force-pushed the selchen/add-cron-schedule-to-replication branch from 4d32593 to 8d864e1 Compare October 24, 2024 22:30
@chenselena chenselena changed the title Convert interval input to cron schedule for self serve replication Convert interval parameter to cron schedule for self serve replication Oct 24, 2024
@chenselena chenselena force-pushed the selchen/add-cron-schedule-to-replication branch 3 times, most recently from 3653da3 to 8791b33 Compare October 25, 2024 15:49
@chenselena chenselena force-pushed the selchen/add-cron-schedule-to-replication branch from 8791b33 to 3a197f1 Compare October 25, 2024 16:46
@chenselena chenselena marked this pull request as ready for review October 25, 2024 16:57
@chenselena chenselena force-pushed the selchen/add-cron-schedule-to-replication branch from 040cb01 to d5ab815 Compare October 28, 2024 21:47
Copy link
Collaborator

@rohitkum2506 rohitkum2506 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks Selena.

Copy link
Member

@abhisheknath2011 abhisheknath2011 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @chenselena for addressing the comments!

@chenselena chenselena merged commit 9be2378 into linkedin:main Oct 31, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants