Skip to main content

Hi everyone,

 

I'm working on an FME workflow that needs to process multiple Shapefiles. The challenge is that these files come with different schemas (different attributes, attribute names, etc.).

 

I’ve noticed that transformers like DuplicateFilter, AttributeExposer, DatetimeConverter, SchemaMapper... don’t handle these schema changes dynamically. That means every time a new Shapefile with a different schema is added, I have to manually update these transformers to accommodate the changes.

 

Is there a way to make this process more dynamic, so the workflow can adapt automatically to different input schemas without manual intervention each time?

 

Any tips or suggestions would be greatly appreciated!

 

Thanks in advance!

Are there any rules on the schema of source datasets - i.e. attribute naming convention according to data type, required operations, etc.?

If there are clear and strict rules, it might be possible to create a workspace to handle only datasets conforming to the rules. Otherwise, however, I think “manually update these transformers to accommodate the changes”  would be your job for now.

In the future, AI could do that - and we could lose our job 😆


If the destination schema is common regardless of the source dataset schema, it could be possible to create a "semi-automated" workspace.

  1. Firstly create a mapping file - describing source attribute names and corresponding destination attributes names for the source dataset.
  2. Shepefile Reader: Read a Shapefile dataset 
  3. SchemaMapper (Attribute Mapping): Use the mapping file to rename source attributes to destination attributes. You can create a published user parameter linked to the Dataset (mapping file path) parameter in the SchemaMapper.
  4. AttributeExposer: Expose destination attributes.
  5. Subsequent transformers: Perform required transformation using the destination attribute names.

You will have to create the mapping file and set it to the user parameter linked to the Dataset parameter in the ShemaMapper whenever source schema has changed, but it would not be necessary to edit the workspace.

 


If the destination schema is common regardless of the source dataset schema, it could be possible to create a "semi-automated" workspace.

  1. Firstly create a mapping file - describing source attribute names and corresponding destination attributes names for the source dataset.
  2. Shepefile Reader: Read a Shapefile dataset 
  3. SchemaMapper (Attribute Mapping): Use the mapping file to rename source attributes to destination attributes. You can create a published user parameter linked to the Dataset (mapping file path) parameter in the SchemaMapper.
  4. AttributeExposer: Expose destination attributes.
  5. Subsequent transformers: Perform required transformation using the destination attribute names.

You will have to create the mapping file and set it to the user parameter linked to the Dataset parameter in the ShemaMapper whenever source schema has changed, but it would not be necessary to edit the workspace.

 

Understood. But every time a shapefile passes through the flow, I'll have to manually trigger transformers like DuplicateFilter, AttributeExposer, and DatetimeConverter, right? It's not possible to define a SchemaMapper for multiple shapefiles at the same time...


Reply