Replies: 1 comment 1 reply
-
@sryza Thanks for putting this together! |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Dagster offers the ability to do "single-run range backfills", i.e. launch a run that backfills multiple contiguous partitions. Currently, you opt in to this functionality by flipping a checkbox in the modal that you use to launch the backfill. This RFC proposes:
Motivation
Why (1):
Why (2):
Proposal
20 partitions per backfill:
All partitions in a single run:
One backfill strategy per multi-asset:
Open questions:
max_materializations_per_minute
tomax_partitions_per_minute
? Otherwise, we'd risk backfilling large amounts of data in a single run.Beta Was this translation helpful? Give feedback.
All reactions