[dagster-airlift][rfc] Multi code locations working

[dpeng817]

Summary & Motivation

Working example of splitting an airflow instance into multiple code locations.

The main interesting bit here is the surface area we expose to filter down the dags. I think it makes sense, but my question is; do we envision needing to split further? IE, what if you have a really complicated dag where you want to split individual tasks out? We might need another pluggability layer for that.

How I Tested These Changes

Added a new kitchen sink full example of splitting an instance into two code locations.

Changelog

NOCHANGELOG

[dpeng817]

• [dagster-airlift] [federation-demo] dbt cloud side #25536
• [dagster-airlift] [federated-demo] dbt cloud defs #25535
• [dagster-airlift][federation-demo] Federated instance stage #25533
• [dagster-airlift][federation-demo] Add tests for completion stage #25532
• [dagster-airlift] [federation-demo] separate airflow home #25506
• [dagster-airlift][federation-demo] setup script changes #25505
• [dagster-dlift] runtime event translation #25487
• [dagster-dlift] job + run methods #25486
• [dagster-dlift] project-level asset spec methods #25485
• [dagster-dlift] defs loader #25484
• [dagster-dlift] factor out state fetching #25483
• [dagster-dlift] implement translator #25482
• [dagster-dlift] rename DbtCloudInstance to DbtCloudClient #25481
• [dagster-dlift] get_tests #25458
• [dagster-dlift] get_sources method #25457
• [dagster-dlift] get_models #25446
• [dagster-dlift] scaffold testing environment #25444
• [dagster-dlift] project scaffold #25443
• [dagster-airlift][rfc] Multi code locations working #25343 👈
• [dagster-airlift][rfc] Proxy operator launches partitioned runs #25324
• [dagster-airlift][rfc] Constrain proxy operator to one run #25301
• [dagster-airlift][tutorial] Add section on time-window partition handling #25297
• [dagster-airlift] Add test for partitioned observation #25293
• [dagster-airlift] Remove airflow variable parsing #25289
• [dagster-airlift] Ensure cli requirements are installed #25284
• [dagster-airlift] better errors when [in-airflow] is not installed, or apache-airflow is not installed #25283
• [dagster-airlift][test] fixtures tear down even after test failure #25282
• [dagster-airlift] [tutorial] add tutorial section for dag-level proxying #25280
• [dagster-airlift] [tutorial] Improve code snippet parsing in tutorial #25268
• [dagster-airlift] Allow dag-level custom callback #25247
• [dagster-airlift] Make proxy operator pluggable #25205
• [dagster-airlift][tutorial] Assets forward tutorial #25193
• [dagster-airlift][cli] dagster-airlift proxy scaffold cli #25168
• [dagster-airlift][dag] in-airflow dag proxying behavior #25160
• [dagster-airlift][dag] Dag-level override proxied state #25159
• [dagster-airlift][dag] Dag overrides top level API #25158
• [dagster-airlift] Support dag-level overrides in builder #25243
• [dagster-airlift] Introduce dag handle, use in sensor #25242
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @dpeng817 and the rest of your teammates on Graphite

[schrockn]

This is pretty great. I thought this would be harder.

[graphite-app]

The asset key for the second DAG appears to be incorrect. It should be AssetKey("dag_second_code_location__asset") instead of AssetKey("dag_first_code_location__asset"). This change will ensure that the asset key correctly corresponds to the DAG it represents.

Spotted by Graphite Reviewer

Is this helpful? React 👍 or 👎 to let us know.