Skip to main content

Hello. Fairly new to FME, so might be a bit over my head.

I’m trying to create a validation workflow that compares 50+ files against 3 main ones. They all have the same structure, except the geography field name that is to be used for the comparison.

The 50+ files are all named with a similar structure > final_DOMAIN_GEOG_Distance or final_DOMAIN_GEOG_Driving. Pretty much the files are divided by domain and 3 geographies.

The 3 main files are a compilation of all domains and GEOG. 

My thought process was to have 2 csv readers (1 for distance and 1 for driving) and a few parameters that would read the user choice for domain and geography and read the csv accordingly. 

I have two issues:

  1. the parameters for file path seem to work but the csv schema remains the same (eg, it’s reading the original reader before adding the parameters), so it doesn’t update the field name with the geography, when it’s different form the original one
  2. is it possible to create parameters that can be used in the transformers, so it always reads the right geography field name?

Hope it makes sense!

Thanks!

Maria

Hi ​@mraposeira ,

Are geography field names completely unpredictable, or is there any naming convention to determine if a field name is geography one?

I think there could be a solution if there were a naming convention.

Hi takashi. 

 

The field names are very similar: MB_CODE_2021, SA1_CODE_2021 and SA2_CODE_2021. They are all the same type as well. I’m getting stuck with the reader not updating the schema to read the right filed name if the user picks a different geography

Looks like there is a common pattern *_CODE_yyyy (*: any uppercase alphabetical or numeric characters, yyyy: four digits; i.e. matches a regex ^[A-Z\d]+_CODE_\d{4}$) for the field names. Does this pattern match only the geograpy field names?

If so, I think you can rename the geography field name to a common and unique name e.g. "geography" with BulkAttributeRenamer (Action: Regular Expression Replace), expose the common name with AttributeExposer, and then perform any operations with the common name in the subsequent processes, 

Hi Takashi.

Yes the geography field name is the only that follows the pattern. All others are the same for all files, so no worries there.

I’ll try that and let you know how I go!

Thanks for the help.

Hi ​@takashi.

 

The BulkAttributeRenamer did the trick, didn’t need the AttributeExposer, as I defined the CSV reader to pull all attributes and that seemed to help.

Thank you so much for the guidance!

Community Stats

32,026
Posts
121,631
Replies
39,648
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.