Standardizing non-data dependencies

[abkfenris]

Currently when an op depends on the completion of another, but not on it's specific outputs, the suggestion is to use Nothing for non-data dependencies.

Issues with the current method

• While this works, any time that a user needs to add a new non-data dependency they need to define a new input on the downstream op.
• Every implementation of an op with a non-data dependency possibly has a unique input name.
• Changing from a single non-data dependency to multiple means another signature change in the downstream op.

These all can lead to confusion and make it harder to reason about graphs.

A possible solution

How about setting non-data dependencies using a method on ops (and I guess graphs) similar to how .configured() works?

Current implementation with Nothing dependencies (from docs)

from dagster import In, Nothing, graph, op


@op
def create_table_1():
    get_database_connection().execute(
        "create table_1 as select * from some_source_table"
    )


@op(ins={"start": In(Nothing)})
def create_table_2():
    get_database_connection().execute("create table_2 as select * from table_1")


@graph
def nothing_dependency():
    create_table_2(start=create_table_1())

Using a method (in this case .wait_until(), but it doesn't need to be that specific name) to specify non-data dependencies.

from dagster import graph, op


@op
def create_table_1():
    get_database_connection().execute(
        "create table_1 as select * from some_source_table"
    )


@op
def create_table_2():
    get_database_connection().execute("create table_2 as select * from table_1")


@graph
def nothing_dependency():
    table_created = create_table_1()
    create_table_2.wait_until(table_created)()

# or

@graph
def nothing_dependency_inline()
    create_table_2.wait_until(create_table_1())()

By using a method, much of the ambiguity (what did I call my nothing in again?) and need to alter the signature of downstream ops can be avoided.

It could also be used to depend on a variable number of upstream ops by taking SolidOutputHandle as *args

class OpDefinition(SolidDefinition):
    ...
    def wait_until(self, *ops: list[SolidOutputHandle]) -> Self:
        ...

Implementation

An initial implementation could work similarly to how .configured() works now, in that it creates a new copy of the op, except in this case instead of baking in the config it would add Nothing to ins.

It could also work by adding a default In(Array(Nothing)) input to all ops, then the method connects those.

Later if the method is useful, then better visualization of non-data dependencies can be more easily shown as they are standardized, as would any optimizations around non-data dependencies.

[sryza]

Hey @abkfenris - a solution we've discussed recently for this is for every op to have a default Nothing input called "after" or something. You could use "fan-in" if you wanted to be downstream of multiple ops.

e.g.

@op
def op1():
    ...

@op
def op2():
    ...

@op
def op3():
    ...


@job
def job1():
    op3(after=[op1(), op2()])

Thoughts?

[ShahBinoy]

so if it carries an after, shouldnt there be a before too or is it just going to be called in one direction?

[sryza]

@ShahBinoy are there cases you have in mind where before would be helpful?

[abkfenris]

I totally forgot about replying to this until I needed it again.

Adding a default Nothing input could work, but maybe use _after or similar to help make it clear that it is something other than what a user has defined.

One reason that I was thinking about using a method is it would allow the API to remain the same, even if the implementation under the hood changes. It could still be _after: In(Nothing) initially, but then if there is a better way it gives you the flexibility to change it. It also would help separate concerns a little between data and non-data dependency flow and what is user defined vs dagster defined.

[eromoe]

@sryza Is the after on planing? Not find on https://docs.dagster.io/_apidocs/ops

[sryza]

@eromoe alas this has not yet been implemented and isn't part of our near-term plan.

[eromoe]

I have similar request
#17169