extract catalog from dagster

[dridk]

In all our Dagster assets, we use Table Schema to document our assets.
I'd like to to extract this information to build a data catalog (a simple parquet file).
Do you know where I can get this information ? I guess it is in a sqlite database somewhere.

A good feature would be to have an endpoint API to display all the metadata and use it as a catalog.

Exemples:

curl http://server_dagster/schema/ 

[
{
    "assets": "stocks",
     "columns": [
      {"name", "string", "description" }
    ]

}
]

[jamiedemaria]

hey @dridk how are you setting the Table Schema so that it shows up in the UI? That will affect how you can access it later

[dridk]

Yes I m using dagster table shema

[jamiedemaria]

ok - but are you returning it as metadata? something else?

[dridk]

Sorry, I misread your answer.
I m using it as metadata like this. Is there another way?

@asset
def test(): 
      df = pd.DataFrame(..)

     return Output(df, metadata={ "schema": TableShema .... })

[jamiedemaria]

No problem. I think you should be able to use the code in this discussion to get the metadata for all of the assets in your project