Skip to main content
Solved

better strategy for pointOnAreaOverlayer

  • February 6, 2015
  • 7 replies
  • 24 views
B
Forum|alt.badge.img
Hello,

 

 

I have 3 millions points that I have to find in what districts they are localized. There is almost 200 districts to test. I am using the pointOnAreaOverlayer because it is faster than the spatialRelator.

 

 

However, it still takes a lot of time. I wonder if using the bounding box of districts in a first POAO and then a listExploder on all matchs and finally a second POAO with a GroupBy on the districts that BBOX matched, to test in what district the point is really localized could be faster?

 

 

This strategy would be similar to how spatial query are done in database when using the spatial index. Doest the POAO already use such strategy internally?

 

 

Any info about that? For now my tests are not really promising...

Best answer by fmelizard

HIi,

 

Anything going into FME gets translated to the internal format and therefore is also spatially indexed. I would try the Clipper as Takashi suggested, but also the POAO with areas going in first (drag the areas reader to the top) is worth trying.

 

Doc states: ' This can reduce memory use of the PointOnAreaOverlayer if you have many Point features to overlay.'
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

7 replies

takashi
Celebrity
  • takashiCelebrity
  • February 6, 2015
Hi,

 

 

Have you tried the Clipper? In my experiences, the Clipper is faster than any other transformers, to perform spatial querying for points against areas.

 

 

Takashi
Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?
gio
Contributor
Forum|alt.badge.img+15
  • gioContributor
  • Contributor
  • February 6, 2015
I think the numbers you need to test simply is a lot.

 

So any spatial strategy wil take a lot of time.

 

 

You are referring to quad-tree indexing i assume.

 

If you have all in a spatial database  you can use it, though it is depricated. "users are encouraged to use R-tree indexing")

 

 

Maybe you should tile the proces and run it per tile using a workspacecaller.

 

 

I tried oredering them first adn using grid. FIrst bit took 1m28sec for 65407points.

 

Second bit samish. That still would be more than 1,5 hours of processing (if i scale it linearily..).

 

 
fmelizard
Safer
Forum|alt.badge.img+21
  • fmelizardSafer
  • Safer
  • Best Answer
  • February 6, 2015
HIi,

 

Anything going into FME gets translated to the internal format and therefore is also spatially indexed. I would try the Clipper as Takashi suggested, but also the POAO with areas going in first (drag the areas reader to the top) is worth trying.

 

Doc states: ' This can reduce memory use of the PointOnAreaOverlayer if you have many Point features to overlay.'
B
Forum|alt.badge.img
  • burton449
  • Author
  • February 8, 2015
Thank you for your answers!

 

 

It seems the deceleration is mostly caused by too much use of RAM when manipulating a large amount of features in a complex workspace containing sorters or aggregators wich must accumulate all features before processing.

 

 

Takashi: thank you for the Clipper advice. Its true that he is slightly faster than the POAO. However, you cant merge attributes in a list when the points intersect more than one polygon. I had to use the many Clippers in cascade to achieve my goal and then the speed was reduced.

 

 

Gio: Quad-tree depracted? Why and where is it depracated? In my opinion, both have their advantages depending in what goal they are created .

 

 

 
takashi
Celebrity
  • takashiCelebrity
  • February 9, 2015
You are right. The Clipper cannot detect intersections between a point and multiple areas.  As you mentioned, the PoinOnLineOverlayer is suitable if there may be overlapped districts.
Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?
gio
Contributor
Forum|alt.badge.img+15
  • gioContributor
  • Contributor
  • February 9, 2015
"deprecated" ....ask Oracle...it is a quote from their site.

 

 

I think it is since version 11.
takashi
Celebrity
  • takashiCelebrity
  • February 9, 2015
Flashed. If you overlayed districts with the AreaOnAreaOverlayer beforehand, the Clipper can also be used. But, it's unknown whether the performance is better than POAO unless measuring. Possibly worse...

 

Jusa an idea.
Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?

Community Stats

32,154
Posts
122,043
Replies
39,812
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.