Introduce support for end_date parameter #181
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is intended to be used to pull back date ranges that have historically not been ingested, without needing to sync all data. For example, if you are missing data from 2023-01-01 to 2023-02-01, you could set the start and end dates to these values to retrieve just that month of data, rather than potentially syncing an extra 9 months of data. This can also be used to incrementally test your ELT workflows to ensure they are compatible with older data schemas.
|
Hi @OSalama, thanks for your contribution! In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes. |
|
You did it @OSalama! Thank you for signing the Singer Contribution License Agreement. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of change
Introduce support for an
end_dateparameterwhich can be used to control the sync's
stop_time.This is intended to be used to pull back
date ranges that have historically not
been ingested, without needing to sync
all data. For example, if you are missing
data from 2023-01-01 to 2023-02-01,
you could set the start and end dates
to these values to retrieve just that
month of data, rather than potentially
syncing an extra 9 months of data.
This can also be used to incrementally
test your ELT workflows to ensure they
are compatible with older data schemas.
Also requested by #72
QA steps
Risks
Rollback steps