You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, we have five ways to run a microbatch model 🥴
creating/recreating the model
batches are run from model's defined begin to "now"
incrementally with no extra args
the batches run are determined by the current time and the lookback
f.x. if you have a batch_size=day, lookback=2, and it is "currently" 2024-10-17 13:04:39 then three batches are run: 2024-10-15, 2024-10-16, and 2024-10-17
incrementally with both --event-time-start and --event-time-end specified
batches are run >= --event-time-start and < --event-time-end
f.x. if you have a batch_size=day, --event-time-start "2024-10-01", and --event-time-end "2024-10-04" then three batches will be run: 2024-10-01, 2024-10-02, 2024-10-03
incrementally with only--event-time-start specified
batches will be run from --event-time-start to "now"
f.x. if you have a batch_size=day, --event-time-start "2024-10-13", and it's currently midday on 2024-10-17, then five batches will be run: 2024-10-13, 2024-10-14, 2024-10-15, 2024-10-16, and 2024-10-17
incrementally with only--event-time-end specified
batches will run the --event-time-end batch plus prior batches according to the period
f.x. if you have a batch_size=day, lookback=2, and --event-time-end "2024-10-04", then three batches will be run: 2024-10-01, 2024-10-02, 2024-10-03
Through further discussion, we believe (4) and (5) are confusing, unexpected, and dangerous. Ways (4) and (5) weren't intentional implementations, but accidental realities created by implementations for (1), (2), and (3). As such, we should require that the specification of --event-time-end necessitates --event-time-start and vice versa.
The text was updated successfully, but these errors were encountered:
Currently, we have five ways to run a microbatch model 🥴
begin
to "now"batch_size=day
,lookback=2
, and it is "currently"2024-10-17 13:04:39
then three batches are run:2024-10-15
,2024-10-16
, and2024-10-17
--event-time-start
and--event-time-end
specified--event-time-start
and <--event-time-end
batch_size=day
,--event-time-start "2024-10-01"
, and--event-time-end "2024-10-04"
then three batches will be run:2024-10-01
,2024-10-02
,2024-10-03
--event-time-start
specified--event-time-start
to "now"batch_size=day
,--event-time-start "2024-10-13"
, and it's currently midday on 2024-10-17, then five batches will be run:2024-10-13
,2024-10-14
,2024-10-15
,2024-10-16
, and2024-10-17
--event-time-end
specified--event-time-end
batch plus prior batches according to theperiod
batch_size=day
,lookback=2
, and--event-time-end "2024-10-04"
, then three batches will be run:2024-10-01
,2024-10-02
,2024-10-03
Through further discussion, we believe (4) and (5) are confusing, unexpected, and dangerous. Ways (4) and (5) weren't intentional implementations, but accidental realities created by implementations for (1), (2), and (3). As such, we should require that the specification of
--event-time-end
necessitates--event-time-start
and vice versa.The text was updated successfully, but these errors were encountered: