Question

Clipper versus ArcGIS Erase tool performance (and attribute preservation)

  • 16 March 2015
  • 6 replies
  • 10 views
olivier
Rank
Badge +6
Hi!

 

 

I would need to prepare a FME-based flow doing more or less what the "Erase" tool of ArcGIS is doing. As there is no equivalent transformer (correct me if I am wrong), I use a Clipper transformer as shown in the attached workspace.

 

How could the process I came up with be run in an acceptable amount of time (close to the 25 minutes of ArcGIS Erase tool) while preserving the attributes of the layer being erased (which ArcGIS does by default)? Indeed for many projects we need to recuperate a lot of information from those specific roads in order to create them with all the required attributes. Preserving these attributes in the process will have a significant impact on performance (I stopped the process when it was already running more than an 1 hour). Even preserving only the Id and then using later on a FeatureMerger in order to join the original attributes to the remaining roads (after clipping) results in a lengthy processing time (1 hour 19 minutes with FME 2014 sp3 64-bit). Do you see a way to keep attributes without impacting the performance too much?

 

 

 

The workspace shows the logic adopted so far. The actual data is too large to attach here.

 

 

 

Side question: do you know if there is any possibility for FME to call an ArcGIS tool from the Toolbox, and delegate that process step to the ArcGIS client before performing the rest of the actions? That would also be an option.

 

 

Thanks in advance for your suggestions!

 

 

Best regards,

 

 

Olivier

 

 

6 replies

ebygomm
Rank
Userlevel 1
Badge +21
I take it you have a group by on the clipper?
olivier
Rank
Badge +6
Hi Elizabeth,

 

 

Yes. I am actually using the row and column attributes created by the Tiler. This helps the Clipper treat each tile individually. Not using them in the Group By would lead to much longer processing time. 

 

 

Olivier
G
Badge +3
You can preserve attributes by setting merge attributes and set a merge attribute prefix.

 

 

If you have it conpletely tiled, you can use this fact to set paralell prcessing. Maybe safe some time.

 

 

 

 

I prefer to use a AreaOnAreaOverlayer followed by one or more testers, mostly faster than the Clipper.

 

 

 

 

 
olivier
Rank
Badge +6
Thank you for your suggestions. In the meantime I installed FME 2015 build 15453 for 64-bit but no performance improvement is observed. I have also tried using a LineOnAreaOverlayer (my roads are lines) with different Parallel Processing Levels. No matter which processing level I choose I always get around 1h55min processing time which is way too long compared to ArcGIS 25 minutes. The best timing I get with FME is obtained when removing all attributes upfront in the process. It then goes down to 31 minutes. However the problem is that I then don't have any attributes in my output. Any additional suggestions are welcome.
H

Although this is ages ago - if ArcGIS is available on the same computer as FME, its geoprocessing tools can be used to extend FME's functionality through the use of the PythonCaller. You would then code a feature processor that employs this tool. Make sure to set your FME Python environment to the ArcGIS one before running your workspace.

 

See this article for more details:

https://community.safe.com/s/article/using-arcpy-for-fme-feature-processing

 

olivier
Rank
Badge +6

Although this is ages ago - if ArcGIS is available on the same computer as FME, its geoprocessing tools can be used to extend FME's functionality through the use of the PythonCaller. You would then code a feature processor that employs this tool. Make sure to set your FME Python environment to the ArcGIS one before running your workspace.

 

See this article for more details:

https://community.safe.com/s/article/using-arcpy-for-fme-feature-processing

 

Thank you very much @huubzwart​ ! This is an interesting solution indeed.

Reply


Community Stats

28,696
Posts
109,442
Replies
36,546
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.