-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[dagster-airlift][rfc] Multi code locations working #25343
base: master
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is pretty great. I thought this would be harder.
e20b6b8
to
2c055e5
Compare
a80f724
to
67ee24b
Compare
ExpectedMat(AssetKey("dag_first_code_location__asset"), runs_in_dagster=False) | ||
], | ||
"dag_second_code_location": [ | ||
ExpectedMat(AssetKey("dag_first_code_location__asset"), runs_in_dagster=False) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The asset key for the second DAG appears to be incorrect. It should be AssetKey("dag_second_code_location__asset")
instead of AssetKey("dag_first_code_location__asset")
. This change will ensure that the asset key correctly corresponds to the DAG it represents.
Spotted by Graphite Reviewer
Is this helpful? React 👍 or 👎 to let us know.
2c055e5
to
5d3e709
Compare
67ee24b
to
0c97247
Compare
86a320e
to
af598c3
Compare
0d2ddbd
to
f2ecb1d
Compare
af598c3
to
510cb79
Compare
f2ecb1d
to
b0e654c
Compare
510cb79
to
902ae76
Compare
b0e654c
to
336df64
Compare
315cf37
to
ed75413
Compare
ed75413
to
ca7b2e7
Compare
Summary & Motivation
Working example of splitting an airflow instance into multiple code locations.
The main interesting bit here is the surface area we expose to filter down the dags. I think it makes sense, but my question is; do we envision needing to split further? IE, what if you have a really complicated dag where you want to split individual tasks out? We might need another pluggability layer for that.
How I Tested These Changes
Added a new kitchen sink full example of splitting an instance into two code locations.
Changelog
NOCHANGELOG