Skip to main content

The Swedish DOT offers extensive road data as Open Data. It is packaged in an ESRI Personal Database (mdb) with 120+ tables. I want to dynamically load all of the data into Postgis for local storage.

However, quite many fields, scattered in the database structure, are of the type CHAR with a width of 2147483647 (sic!). Of course, no post contains that many characters, and FME tries to convert these fields to VARCHAR with the same width. This breaks the translation with the message that the type VARCHAR cannot exceed 10485760 characters.

So in short, I need a way to filter and catch all tables with fields of the type CHAR, with a width of 2147483647 and alter the width to either the actual content of the field, or a static value that does not truncate what's in there.

Regards, Mats.E

You can use the FeatureReader which will return a schema feature in addition to all the data features. You can the modify the schema feature to set specific datatypes before sending it along with the data features to the FeatureWriter.

Let us know if you need more specific details.


Hi @david_r 

I would like to know how to change the schema, meaning datatypes and such. 

I am using FME 2022, but I can test it as well in FME 2023

Br

Felipe Verdú


you could use the “schema scanner” to dynamically recreate the schema in the workflow, and add possible grouping by “fme_feature_type”. how ever the schema may alter every time you run it based on the max string lengths in the source.

 

As mentioned above you could use the “feature reader” and alter the outputted schema files with the “schema mapper” transformer. Alternatively you could expose the “fme_data_type” from the schema, identify those with a length longer than “2147483647” and replace it with a number below your threshold.


Hi @david_r 

I would like to know how to change the schema, meaning datatypes and such. 

I am using FME 2022, but I can test it as well in FME 2023

Br

Felipe Verdú

The schema is defined as a list on the feature output by the FeatureReader, so you can simply modify the schema there before passing it onto e.g. the FeatureWriter. For example, you could look through the list for a specific field based on attribute{}.name, then modify the data type. Just be aware that you’ll have to modify the data type both in attribute{}.fme_data_type and attribute{}.native_data_type simultaneously.

 


Reply