Skip to main content
Solved

Spatial filter performance

  • June 16, 2015
  • 6 replies
  • 72 views
K
Forum|alt.badge.img
Dear All , 

 

 

We face a performance issue in spatial filter when both candiate and filter having more than 150K , as shown below , it is taken more than 2 days and still running . below is part to the log files

 

 

2015-06-16 11:11:48|168167.0|  7.0|INFORM|SpatialFilter_13(SpatialFilterFactory): Processed 'Candidate' 118233 / 150608

 

2015-06-16 11:11:56|168174.1|  7.2|INFORM|SpatialFilter_13(SpatialFilterFactory): Processed 'Candidate' 118236 / 150608

 

2015-06-16 11:12:01|168179.0|  4.9|INFORM|SpatialFilter_13(SpatialFilterFactory): Processed 'Candidate' 118242 / 150608

 

2015-06-16 11:12:06|168183.9|  4.9|INFORM|SpatialFilter_13(SpatialFilterFactory): Processed 'Candidate' 118243 / 150608

 

2015-06-16 11:12:11|168188.2|  4.3|INFORM|SpatialFilter_13(SpatialFilterFactory): Processed 'Candidate' 118255 / 150608

Best answer by takashi

Hi @annachukicheva, Regarding the original question, I assumed that the filter and candidates geometries are areas. In your case, both geometries are points, so I think that you can omit the step 1,2 and 5-7.

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

6 replies

takashi
Celebrity
  • takashiCelebrity
  • June 16, 2015
Hi,

 

 

How about tiling the features so that the processing would be divided into groups?

 

 

(1) Add a unique ID attribute to the candidates.

 

(2) Extract geometry as an attribute for each candidate (GeometryExtractor).

 

(3) Send both filters and candidates to a Tiler.

 

(4) Separate the tiled filters and candidates, send them to the SpatialFilter. Set tile column and row attributes to the "Group By" parameter.

 

(5) Select unique Passed features using the DuplicateRemover (Key Attribute = ID attribute of candidate).

 

(7) Restore the original geometry (GeometryReplacer).

 

 

The optimal number of tiles is unknown but I expect 10x10 - 20x20 would be effective.

 

 

Takashi
Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?
gio
Contributor
Forum|alt.badge.img+15
  • gioContributor
  • Contributor
  • June 17, 2015
The tiling strategy has as perk that you can use the parallelprocessing option (because of the group by for one)

 

Should yield (albeit maybe slight) performence gain.
K
Forum|alt.badge.img
  • khalid_omer
  • Author
  • June 18, 2015
thanks for the supports , I will check it and let you know . 
A
Forum|alt.badge.img

Hello, @takashi,

first of all, thank you a lot for all your good advices you gave in this share point.

have a question to your process, you wrote here.

First, to my process: I need to check: points objects with another point objects (if they are equal or not). So i made as in your list written until the step Nr. 4.

With step 5 and 7 i m not so sure if i need them to complete. Can you please explain me what for you do these steps? Are they needed in my process, u think?

Thanks,

Anna

takashi
Celebrity
  • takashiCelebrity
  • Best Answer
  • January 27, 2016

Hi @annachukicheva, Regarding the original question, I assumed that the filter and candidates geometries are areas. In your case, both geometries are points, so I think that you can omit the step 1,2 and 5-7.

Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?
A
Forum|alt.badge.img

Hi @annachukicheva, Regarding the original question, I assumed that the filter and candidates geometries are areas. In your case, both geometries are points, so I think that you can omit the step 1,2 and 5-7.

Thank you @takashi a lot!

Community Stats

32,154
Posts
122,043
Replies
39,812
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.