Replies: 1 comment
-
I have found the answer here: #17718 |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Context
I’m working on a pipeline where I need to load an external
CSV
file and process each row as apd.Series
through a series of operations in a graph. Each node in the graph may have external dependencies, such as reading geospatial data from files stored onS3
. I want to represent these dependencies as external assets in Dagster, but without materializing them within Dagster itself (e.g., usingxarray
to read the geospatial data).As each node processes its respective inputs, the external assets (like the geospatial data) will be passed between nodes as needed. Finally, the last node in the pipeline will handle the materialization of the final output
How would you approach structuring this in Dagster? Is this even possible?
Approach
Based on: https://www.youtube.com/watch?v=KVqyarPbCeU&t=795s
I want to use the
burnable_4km
asset as a reference to an S3 file path, without materializing it. Sinceburnable_4km
is an external resource defined withAssetSpec
and only want to reference the metadata (like the S3 file path) in myop
.Issue
In Dagster UI I'm seeing the below error. What am I missing?
Beta Was this translation helpful? Give feedback.
All reactions