Question

SchemaMapper: source field does not import when matches destination field name

  • 29 September 2016
  • 5 replies
  • 0 views
D
Badge

Hi, I need to import lots of different datasets into a database. The field names of the source datasets (i.e. reader) don't always match the field names of the destination database (i.e. writer). Therefore I'm using the SchemaMapper transformer. My external lookup table that SchemaMapper uses has all of the different field names of my source datasets, and the corresponding names for my destination database (see example below).

Source DatasetDestination DatabaseDateRECORDDATEDate FromRECORDDATEDate_valueRECORDDATELatLATITUDEYLATITUDELatitudeLATITUDELongLONGITUDEXLONGITUDELongitudeLONGITUDE

However, if my source dataset has a field name that already matches the field name of my destination database, then that field does not import into my destination database when I run my FME script using SchemaMapper. So using the example table above, if my source dataset has a field name "LATITUDE', then that field name will not import. Is there a setting in SchemaMapper to get around this? Or alternative easy fix? Or is it something I'm doing wrong?

Cheers, Damian

5 replies

mark2atsafe
Rank
Userlevel 4
Badge +25

Hi @damian6105

Sadly I don't think this is going to work at all, not just when the attribute exists.

What happens is that FME runs through the entire list mapping the data. It doesn't stop when it finds a match that is applicable.

For example, if you have an attribute called Date that is set to "Sept-2016" then firstly it is renamed to RECORDDATE, so RECORDDATE has a value of "Sept-2016"

Then Date From is renamed to RECORDDATE, so the original value is overwritten and RECORDDATE is now an empty string.

Then Date_value is renamed to RECORDDATE, again overwritting it with an empty string.

The same thing is happening to LATITUDE - it is being overwritten with Lat, Y, and Latitude.

So if everything has worked up until now, then I'm guessing you are running it on data called Date_Value, Latitude, and Longitude. That's because the last attribute in the list wins.

This is already filed as a problem with our developers - PR#46006. I'll add this thread to the information and increase the priority, since it does seem to me to be an important issue.

I don't really have a simple workaround right now, but I will see what I can come up with.

takashi
Rank
Userlevel 2
Badge +17

Hi @damian6105, as @Mark2AtSafe mentioned, there doesn't seem to be a workaround with the SchemaMapper unfortunately. However, if all the possible attribute names in the source datasets are known when creating the workspace and they will not change at run-time, the AttributeRenamer may be used instead. e.g.

Note: This parameters setting example is for FME 2016. In FME 2015 and earlier, the old attribute names have to be exposed beforehand, and you will have to set the <No Action> flag explicitly to the Default Value column for each row in order to prevent that the empty string will be assigned to output attribute when the corresponding input attribute is missing.
sigtill
Rank
Badge +21

Workaround to rename to something unique - like:

Date -> RECORDDATE_AHORSEGOT4FEET

MyDate -> RECORDDATE_AHORSEGOT4FEET

RECORDDATE -> RECORDDATE_AHORSEGOT4FEET

And then at the end:

RECORDDATE_AHORSEGOT4FEET-> RECORDDATE

The real SigTill
mark2atsafe
Rank
Userlevel 4
Badge +25

Workaround to rename to something unique - like:

Date -> RECORDDATE_AHORSEGOT4FEET

MyDate -> RECORDDATE_AHORSEGOT4FEET

RECORDDATE -> RECORDDATE_AHORSEGOT4FEET

And then at the end:

RECORDDATE_AHORSEGOT4FEET-> RECORDDATE

I still think that doesn't work. If Date is valid and gets copied to RECORDDATE_AHORSEGOT4FEET then MyDate will overwrite it with an empty value.

 

 

They would need to be different names. All I could think of was:

 

 

Date -> RECORDDATE1

 

MyDate -> RECORDDATE2

 

RECORDDATE -> RECORDDATE3

 

 

...and then try and figure out which of those has a value. Or maybe concatenate RECORDDATE1,2,3 into RECORDDATE, since two of the three will be empty values.

 

takashi
Rank
Userlevel 2
Badge +17

Seeing @Mark2AtSafe's comment, an idea flashed in my mind.

1. In the mapping table for the SchemaMapper, add a suffix to every destination attribute name to differentiate them uniquely. The suffix format is e.g. <specific character string><one or more digits>. The table looks like this.

Source Dataset,Destination Database
Date,RECORDDATE_blabla1
Date From,RECORDDATE_blabla2
Date_value,RECORDDATE_blabla3
Lat,LATITUDE_blabla1
Y,LATITUDE_blabla2
Latitude,LATITUDE_blabla3
Long,LONGITUDE_blabla1
X,LONGITUDE_blabla2
Longitude,LONGITUDE_blabla3

2. Remove every empty attribute from the resulting features, with the NullAttributeMapper.

  • Map: All Attributes
  • If Attribute Value Is: Empty
  • Map To: Missing.
3. Remove the suffix from all attribute names, with the BulkAttributeRenamer.
  • Rename: All Attributes
  • Action: Regular Expression Replace
  • Text to Find: ^(.+)_blabla\d+$
  • String: \1

Reply


Community Stats

28,654
Posts
109,285
Replies
36,527
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.