Solved

AreaonAreaOverlayer very slow with many features

  • 17 March 2021
  • 1 reply
  • 15 views
paul_m
Rank
Badge +1

Hello! I'm iterating through 58 file geodatabases and grabbing a feature class of a common name from each (though I'm only using 3 of the feature classes for testing right now). Then I'm trying to use the AreaOnAreaOverlayer transformer to effectively clip millions of features by state boundaries and give them a field with a value for which state they are in. There seems to be a bottleneck at the AreaOnAreaOverlayer transformer (see attached screenshot) that is on track to take hundreds of hours with just 5% of the data coming through.

 

FME 2021.0 is using ~20GB of memory and I'm getting the following message in my log:

ResourceManager: Optimizing Memory Usage. Please wait...

AreaOnAreaOverlayer (OverlayFactory): Searching for topologically significant nodes... 0.79% done

 

Does anyone know of a faster way to do this? Any help would be much appreciated!

icon

Best answer by markatsafe 18 March 2021, 00:03

View original

1 reply

M
Badge +2

@paul_m​ Both the FeatureMerger and the AreaOnAreaOverlayer are acting as blockers in your workflow.

Try switching your FeatureMerger to FeatureJoiner OR order your readers in the navigator so that the SBC_MU_National reader is the first reader, then use Suppliers First in FeatureMerger. You can do the same with Clipper (Clippers FIrst) which can also improve performance.

But mainly, don't read all your 58 geodatabases in on fell swoop. Create a parent workspace with WorkspaceRunner. Then call this workspace, with the name of each Feil Geodb passed in. And run that 58 times. Batch Processing Using the WorkspaceRunner

Reply


Community Stats

28,692
Posts
109,432
Replies
36,542
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.