Column/field description

[simonaubertbd]

Describe the enhancement requested

Hello,

I may have misunderstood or misread the documentation but I didn't find it. I would love to have a field/column description just like we have in databases :

Best regards,

Simon

Component(s)

No response

[gaborcsardi]

For every column, you can store arbitrary key-value metadata (string -> string), see at

parquet-format/src/main/thrift/parquet.thrift

Line 809 in 5a5c894

8: optional list<KeyValue> key_value_metadata

[simonaubertbd]

Hello @gaborcsardi Yes, thanks for the tip but it can't be exploited by a software such as dataviz, etl or dataviz.

[liquidcarbon]

Hi! I'm confused by different types of metadata.

1. there's key-value metadata of type BLOB that you can write via schema metadata (in python pyarrow.table(data, metadata=...)) and read similarly, or through https://duckdb.org/docs/data/parquet/metadata.html#parquet-key-value-metadata
2. there's separate place, with these fields, that you can read with https://duckdb.org/docs/data/parquet/metadata.html#parquet-metadata

['file_name', 'row_group_id', 'row_group_num_rows',
       'row_group_num_columns', 'row_group_bytes', 'column_id', 'file_offset',
       'num_values', 'path_in_schema', 'type', 'stats_min', 'stats_max',
       'stats_null_count', 'stats_distinct_count', 'stats_min_value',
       'stats_max_value', 'compression', 'encodings', 'index_page_offset',
       'dictionary_page_offset', 'data_page_offset', 'total_compressed_size',
       'total_uncompressed_size', 'key_value_metadata']

column-level information and data types are recorded here. But how to you write something into THAT key_level_metadata ?

[simonaubertbd]

Since Parquet is now meant to be a standard file format, use for intermediate storage between tools, it really seems necessary to have this field/column description added. It exists in a lot of format such as Tableau Hyper, Alteryx yxdb, etc

As for now, it means loosing informations when storing in Parquet.

Best regards,

Simon

[gaborcsardi]

Hello @gaborcsardi Yes, thanks for the tip but it can't be exploited by a software such as dataviz, etl or dataviz.

Why can't dataviz or etl software add the description as metadata? E.g. as a key called description?

[simonaubertbd]

@gaborcsardi Because it means asking the user of ETL a key name and the user of the dataviz the key name. This means another place (a file, a mail, an issue...) where the users have to share the information "hey guys, I put the field description in the key label". But since in most ETL or dataviz tool, you don't have the ability to retrieve metadata by key, users of each tools have to ask the tool editor a way to retrieve a value by key description. The whole process isn't user friendly.

Since it's a very common metadata and a lot of file format already have it, I think this is way better to have it in the file format specification. As of today, this is probably my main pain point with Parquet.

Best regards,

Simon