Skip to main content
Solved

What is the best way to clip a lot of data ?

  • June 17, 2020
  • 10 replies
  • 325 views
P

Hello guys,

I have a files in a Esri geodatabase with the county of New York and an other files with "the constraint" its an huge group off small polygon (8 millions) who represent a mix between the slope + forest +swamp, what ever.

 

 

I would like to clip the county by the constraint and get the outside flow, it will form at the end the polygon of the county with some holes where was the constraints.

So i use the clipper with this two files i think as many of you know it take long time and at the end fme say that i didn't have enough place (i had 20 gb free).

 

After that i try to create different workbench with 1 million feature to run in each workspace (workspace 1 = 0 to 1 million , workspace 2 = 1 to 2 million, workspace 3 = 2 to 3 million, etc), but im not sur its work to so.

 

 

So well my question is for you what is the best way ?

Best answer by kd

Take a look at the 'clippers first' option. If you can guarantee that the clipper features enter the transformer first you can avoid that the clipper behaves as a blocking transformer, eating all of your pc's memory.

I agree with @erik_jan as well. If you only need one county it is best to feed this county's polygon to the featurereader that reads the other data to filter the data (spatial filter) before you start reading it into the FME software.

You can/should still clip the geometries afterwards.

Generally speaking, the more you can filter your data before you read it into FME the better your workflow will be performance and memory wise.

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

10 replies

erik_jan
Contributor
Forum|alt.badge.img+22
  • erik_janContributor
  • Contributor
  • 2179 replies
  • June 17, 2020

The best and fastest way would be:

Read the county polygon, followed by a FeatureReader transformer to read only those small polygons that fall in the county.

Then feed both into the Clipper transformer to do the final clipping.

The FeatureReader will be able to filter out all features that are not inside or partially inside the county and will do so in a more optimized way.

If the county polygon contains a large number of vertices, then replacing it by a bounding box (BoundingBoxReplacer transformer) before using the FeatureReader transformer can improve performance too.

Hope this helps.

K
Forum|alt.badge.img+1
  • kd
  • 52 replies
  • Best Answer
  • June 17, 2020

Take a look at the 'clippers first' option. If you can guarantee that the clipper features enter the transformer first you can avoid that the clipper behaves as a blocking transformer, eating all of your pc's memory.

I agree with @erik_jan as well. If you only need one county it is best to feed this county's polygon to the featurereader that reads the other data to filter the data (spatial filter) before you start reading it into the FME software.

You can/should still clip the geometries afterwards.

Generally speaking, the more you can filter your data before you read it into FME the better your workflow will be performance and memory wise.

J
Forum|alt.badge.img+2
  • jlbaker2779
  • 194 replies
  • June 17, 2020

Convert the boundary polygon to a line using the TopologyBuilder, use a SpatialRelator to identify the polygons that intersect, clip, and then bring all the data back together.

 

That will only clip what needs to be clipped and ignore the rest.

P
  • pascalfilippi
  • Author
  • 13 replies
  • June 17, 2020

Take a look at the 'clippers first' option. If you can guarantee that the clipper features enter the transformer first you can avoid that the clipper behaves as a blocking transformer, eating all of your pc's memory.

I agree with @erik_jan as well. If you only need one county it is best to feed this county's polygon to the featurereader that reads the other data to filter the data (spatial filter) before you start reading it into the FME software.

You can/should still clip the geometries afterwards.

Generally speaking, the more you can filter your data before you read it into FME the better your workflow will be performance and memory wise.

im sorry i didnt precise that i would like to clipper all my county actually, thank you i will try your advice

P
  • pascalfilippi
  • Author
  • 13 replies
  • June 17, 2020

Im sorry i didnt precise i want to clip every county actually :)

P
  • pascalfilippi
  • Author
  • 13 replies
  • June 17, 2020

Convert the boundary polygon to a line using the TopologyBuilder, use a SpatialRelator to identify the polygons that intersect, clip, and then bring all the data back together.

 

That will only clip what needs to be clipped and ignore the rest.

im sorry i didnt precise that i would like to clipper all my county actually, thank you for your answer

P
  • pascalfilippi
  • Author
  • 13 replies
  • June 17, 2020

The best and fastest way would be:

Read the county polygon, followed by a FeatureReader transformer to read only those small polygons that fall in the county.

Then feed both into the Clipper transformer to do the final clipping.

The FeatureReader will be able to filter out all features that are not inside or partially inside the county and will do so in a more optimized way.

If the county polygon contains a large number of vertices, then replacing it by a bounding box (BoundingBoxReplacer transformer) before using the FeatureReader transformer can improve performance too.

Hope this helps.

im sorry i didnt precise that i would like to clipper all my county actually, thank you very much

P
  • pascalfilippi
  • Author
  • 13 replies
  • June 17, 2020

Actually at the beginning i wanted to clip all the state of newyork than i was thinking may be it will be easier if y clip county by county

P
  • pascalfilippi
  • Author
  • 13 replies
  • June 17, 2020

I think i can use your way if can create a loop and run the clipper on each county with your technique to select the smalls polygones in each county. but i don t know how to create a loop

erik_jan
Contributor
Forum|alt.badge.img+22
  • erik_janContributor
  • Contributor
  • 2179 replies
  • June 18, 2020

im sorry i didnt precise that i would like to clipper all my county actually, thank you very much

This process will do all counties, but one at a time.

And the use of the FeatureReader will limit the number of features to be processed by the Clipper.

Could be faster than all at once.

Community Stats

32,073
Posts
121,788
Replies
39,700
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.