Skip to main content
Question

How to speed up the AttributeValueMapper?

M
Forum|alt.badge.img

Hi all,

We've just upgraded to 2018.0.0.2 from 2017.1.x and a workbench that used to run in about 5 minutes is now running at 1 record per second and I have 62,000 records to process. The bottleneck appears to an AttributeValueMapper with 140 lookup values. The FileGDB and AttributeRemover read all 62,000 records in next to no time and then they hit the AVM and everything slows right down.

We've tried a FeatureWriter (to CSV with DuplicateRemover) to get a file of unique values and updated this with the new mapped values and then used a DatabaseJoiner but that is still slow at 2 features per second.

Any suggestions on how to speed things up?

The workbench takes a FileGDB table and uses DatabaseJoiners to join two more FileGDB tables (with a common key field in each one) to build one output table in PostgreSQL with selected fields from each table.

Thanks

Ross

3 replies

fmelizard
Safer
Forum|alt.badge.img+18
  • fmelizardSafer
  • Safer
  • May 10, 2018

Hi @mixedbredie, Really sorry to hear that you're running into this! Can you please get in touch with at in Support. https://www.safe.com/support/report-a-problem/

 

 

If we could get a copy of your source data and workflow that would help us work much quicker on this matter.

 

M
Forum|alt.badge.img
fmelizard wrote:

Hi @mixedbredie, Really sorry to hear that you're running into this! Can you please get in touch with at in Support. https://www.safe.com/support/report-a-problem/

 

 

If we could get a copy of your source data and workflow that would help us work much quicker on this matter.

 

Hi @MattAtSafe

 

 

I've raised the issue with our FME support, 1Spatial. I'll get the data and workbench cleared for sharing.

 

 

The logs were filling up with lines like this and made me wonder what was happening.

 

 

2018-05-11 09:24:14| 18.6| 0.6|INFORM|FILEGDB reader: Splitting feature table

 

2018-05-11 09:24:16| 20.0| 1.5|INFORM|... Last line repeated 3 times ...

 

2018-05-11 09:24:16| 20.0| 0.0|INFORM|AttributeValueMapper(@Lookup): Processed 4 of 61754 features

 

2018-05-11 09:24:17| 20.5| 0.5|INFORM|FILEGDB reader: Splitting feature table

 

2018-05-11 09:24:21| 24.8| 4.3|INFORM|... Last line repeated 9 times ...

 

2018-05-11 09:24:21| 24.8| 0.0|INFORM|AttributeValueMapper(@Lookup): Processed 9 of 61754 features

 

2018-05-11 09:24:22| 25.3| 0.5|INFORM|FILEGDB reader: Splitting feature table

 

2018-05-11 09:24:26| 29.5| 4.2|INFORM|... Last line repeated 9 times ...

 

2018-05-11 09:24:26| 29.5| 0.0|INFORM|AttributeValueMapper(@Lookup): Processed 14 of 61754 features

 

2018-05-11 09:24:27| 30.0| 0.5|INFORM|FILEGDB reader: Splitting feature table

 

2018-05-11 09:24:31| 34.4| 4.4|INFORM|... Last line repeated 9 times ...

 

2018-05-11 09:24:31| 34.4| 0.0|INFORM|AttributeValueMapper(@Lookup): Processed 19 of 61754 features

 

2018-05-11 09:24:32| 34.9| 0.5|INFORM|FILEGDB reader: Splitting feature table

 

2018-05-11 09:24:36| 38.3| 3.5|INFORM|... Last line repeated 7 times ...

 

2018-05-11 09:24:36| 38.3| 0.0|INFORM|AttributeValueMapper(@Lookup): Processed 23 of 61754 features

 

2018-05-11 09:24:36| 38.8| 0.5|INFORM|FILEGDB reader: Splitting feature table

 

2018-05-11 09:24:41| 43.3| 4.5|INFORM|... Last line repeated 9 times ...
M
Forum|alt.badge.img
fmelizard wrote:

Hi @mixedbredie, Really sorry to hear that you're running into this! Can you please get in touch with at in Support. https://www.safe.com/support/report-a-problem/

 

 

If we could get a copy of your source data and workflow that would help us work much quicker on this matter.

 

Turns out it was the Joiner following the AttributeValueMapper that was slowing things down. 1Spatial were able to replicate and have sent the data and workbench to Safe to check.

 

 

I ended up importing the data into PostgreSQL and then doing everything in the database and then using FME to write the final output to the various places. Processing time went from ~18 hours to < 4 minutes. Which was nice.

 

 

Reply


Helpful Members This Week

Community Stats

31,353
Posts
118,924
Replies
38,967
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.