[RFC] Jobless automation

[smackesey]

Summary

In the past Dagster has emphasized the concept of jobs. Jobs appear at the top of our left nav, and user-defined jobs are currently required to leverage Dagster’s schedules and sensors.

We are moving towards a world where users typically only need to think about assets, rather than how assets and jobs interrelate. In the Dagster UI this is already the case. In the asset graph view, users can directly materialize assets without reference to any job. We are now continuing this theme in our core python APIs.

Going forward, jobs will fully leave center stage and become an “advanced” Dagster concept. “Automation” (a category encompassing schedules and sensors) will swap places with “jobs” in proximity to the core Dagster experience. Below are some specific planned changes:

Jobless schedules/sensors

Currently, schedules and sensors need to target a job. If you want to define a target that will only ever be executed by a schedule/sensor, you need to first define a separate job. Here is a basic example from our current schedule tutorial:

from dagster import (
    AssetSelection,
    Definitions,
    ScheduleDefinition,
    define_asset_job,
    load_assets_from_modules,
)

from . import assets

all_assets = load_assets_from_modules([assets])

# Define a job that will materialize the assets
hackernews_job = define_asset_job("hackernews_job", selection=AssetSelection.all())

# Addition: a ScheduleDefinition the job it should run and a cron schedule of how frequently to run it
hackernews_schedule = ScheduleDefinition(
    job=hackernews_job,
    cron_schedule="0 * * * *",  # every hour
)

defs = Definitions(
    assets=all_assets,
    schedules=[hackernews_schedule],
)

After the planned changes to schedule/sensor APIs:

from dagster import (
    Definitions,
    ScheduleDefinition,
    load_assets_from_modules,
)

from . import assets

defs = Definitions(
    schedules=[
        ScheduleDefinition(
            name="hackernews_schedule",
            target=load_assets_from_modules([assets]),
            cron_schedule="0 * * * *",  # every hour
        )
    ]
)

Notable differences between two examples:

• No longer need to import or use define_asset_job and AssetSelection.
• No longer need to pass asset definitions to assets parameter of Definitions
• Uses new target instead of job argument of ScheduleDefinition, and therefore the entire notion of “job” is not introduced in simple use cases.

Assets passed to schedules and sensors automatically included in Definitions

Currently, every asset referenced by another definition needs to be passed to the assets argument of Definitions. With the ability to pass assets directly as an automation target, we can relax this constraint. Assets passed only as automation targets will be automatically included in the set of assets stored on a Definitions object and thus able to be referenced by other definitions:

@asset
def foo(): ...

@schedule(cron_schedule="* * * * *", target=[foo])
def my_schedule(): ...

@sensor(target=AssetSelection.all())  # This selection will include asset `foo`
def my_sensor(): ...

# This works even though no `assets` are passed
defs = Definitions(
    schedules=[my_schedule],
    sensors=[my_sensor],
)

New top-level “Automation” page and elimination of the left nav

Dagster’s left nav currently puts jobs front and center and references the schedules/sensors that target them as an afterthought. We are making changes to instead emphasize automation (schedules/sensors) and de-emphasize jobs.

The Timeline will now be organized by automations by default:

The left nav is being removed, and the “Sensors”, “Schedules”, and “Auto-materialization” views from “Overview” will be merged into a single top-level “Automation” view. The contents of “Automation” are grouped by code location and can be filtered by Name, Status, Type, or Tag:

A top level “Jobs” view will be accessible from the main nav. If the user has not defined any jobs (a likely scenario in a world with jobless automation), this view will be hidden. Jobs are grouped by code location and can be filtered by Name or Tag:

These changes will begin to roll out experimentally in 1.8 and you can enable them by enabling the Experimental Nav feature flag in your user settings:

Conclusion

The Python changes outlined above will ship with Dagster 1.8. We believe these changes will simplify the experience of most Dagster users. Please post any questions, comments or concerns below.

[PedramNavid]

This is really exciting stuff. One note, here you do

            target=load_assets_from_modules([assets]),

Often I don't want to load everything from a module, but select particular assets. One feature jobs provided was a way to put a little circle around some group of assets I care about and then say run this thing.

Here are some ways I would expect this to work, without knowing anything about the underlying API:

@asset
def my_asset(): ...

@asset
def my_other_asset(): ...

my_selection = AssetSelection.groups("marketing")

...
target=[my_asset, my_other_asset]

...
target=['my_asset', 'my_other_asset']
...
target=[my_selection]

Would this work with the new jobless API?

[nicklausroach]

Yeah this is something @braunjj and I discussed a while back. I basically want to think of a run as a combination of an instigator (schedule, sensor, etc) and an asset selection. Like @PedramNavid said, I really just thought of jobs as a "lasso" around a set of assets, I would honestly even prefer we rename the Jobs tab to "Asset Selections" or something like that.

[smackesey]

Here is the type signature for target:

        target (Optional[Union[CoercibleToAssetSelection, AssetsDefinition, JobDefinition, UnresolvedAssetJobDefinition]]):
            The target that the schedule will execute.

And here's CoercibleToAssetSelection:

CoercibleToAssetSelection: TypeAlias = Union[
    str,
    Sequence[str],
    Sequence[AssetKey],
    Sequence[Union["AssetsDefinition", "SourceAsset"]],
    "AssetSelection",
]

So a lot of different formats can be passed to target.

[PedramNavid]

One more question, how would this work with partitions?

[smackesey]

Can you be more specific? Coming back to this topic after leave but nothing jumps out to me as problematic wrt partitions.

[PedramNavid]

I may have a thought 3 weeks ago but I can't for the life of me remember what it was.