Removing the need to interact with Output in Assets #15392
Replies: 7 comments 8 replies
-
I've been leaning |
Beta Was this translation helpful? Give feedback.
-
I'm generally wary of these kinds of shenanigans, because they make it difficult for people to understand what's going on when they scan the code. And there are situations that become extra weird / friction-y. E.g. def my_helper_function(...) -> AssetMaterialization:
return AssetMaterialization(metadata=...)
@asset
def my_asset():
return my_helper_function(...)
def test_my_helper_function():
result = my_helper_function(...)
assert result == ??? |
Beta Was this translation helpful? Give feedback.
-
I'd personally prefer option 1. I think for some users the concept of
I think |
Beta Was this translation helpful? Give feedback.
-
Option 1Another thing that isn't great about the status-quo Option 2/3I like the general direction of where this is going. Thinking from the docs perspective, eventually the asset concept page could be something like:
|
Beta Was this translation helpful? Give feedback.
-
Just a small point against Context: Context feels overloaded already for me as a user. I never know what is and isn't in Context (or what the correct type of Context should be at a given point (OpExecutionContext, AssetExecutionContext?) If I don't give context the right type (or an explicit type), then the IDE doesn't surface the methods. |
Beta Was this translation helpful? Give feedback.
-
What's wrong with returning an |
Beta Was this translation helpful? Give feedback.
-
The latest update based on design progress on the data quality work is to support |
Beta Was this translation helpful? Give feedback.
-
Along the path to to shift IO management to opt-in, a road bump in the learning curve for users is that in order to communicate metadata about an asset materialization, they need to interact with the concepts of "outputs".
This is done via
Output
or via the context
This is a bit awkward as we need to engage with the potentially new concept of "outputs" and fill out the value with
None
.How do we improve this?
Option 1
context.add_asset_metadata
A slight tweak to the status quo - we add a similar method to
add_output_metadata
that avoids the concept of outputs.Option 2A
return AssetMaterialization
(implicit key)Explored in #14931 we could allow returning materialization events directly.
The
AssetMaterialization
returned would get merged with system provided information to create the finalAssetMaterialization
event that gets reported.For convenience We make
asset_key
an optional argument in the single asset case by indirectly inferring it from the current context.Option 2B
return AssetMaterialization
(explicit key)A variant of 2A removing the shenanigans
Option 3
return MaterializationMetadata
A variant of 2, we could create a new type that users return for this purpose
Beta Was this translation helpful? Give feedback.
All reactions