Is there a way to write custom key-value metadata to the output parquet files?
Parquet - Writing custom key-value metadata
+1
Userlevel 1
+10
- Safer
- 371 replies
-
16 August 2022
FME does not currently have a way to write custom key-value metadata to Parquet. FME supports User Attribute key-values that map to Parquet fields. However, neither hard coded nor user defined metadata are supported at this point. I've added this to our Ideas Forum. Our team would love to gain a better understanding of your use case and intent, and understand why the user attributes that are currently supported aren't sufficient. If you can add these and an other context or examples to the Idea, that would be appreciated.
+1
For my use case, I'm trying to read & write GeoParquet files.
I've managed to read them without issues (at least some sample ones, and ones generated with GDAL).
And I can write GeoParquet files using the python caller in combination with fastparquet, and they work in GDAL and other libraries. However this method sets lengths on the columns, which for some reason then cannot be read in FME using the Parquet reader, giving a column length mismatch error.
I noticed when using the FME Parquet writer it doesn't write lengths to the schema, but I also couldn't add the required key-value metadata. If I try and update the file output by the Parquet writer with Python it removes all the data.
I've attached a reader and writer example workspaces.
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.