Skip to main content
Solved

Spatial filter performance

K
Forum|alt.badge.img
Dear All , 

 

 

We face a performance issue in spatial filter when both candiate and filter having more than 150K , as shown below , it is taken more than 2 days and still running . below is part to the log files

 

 

2015-06-16 11:11:48|168167.0|  7.0|INFORM|SpatialFilter_13(SpatialFilterFactory): Processed 'Candidate' 118233 / 150608

 

2015-06-16 11:11:56|168174.1|  7.2|INFORM|SpatialFilter_13(SpatialFilterFactory): Processed 'Candidate' 118236 / 150608

 

2015-06-16 11:12:01|168179.0|  4.9|INFORM|SpatialFilter_13(SpatialFilterFactory): Processed 'Candidate' 118242 / 150608

 

2015-06-16 11:12:06|168183.9|  4.9|INFORM|SpatialFilter_13(SpatialFilterFactory): Processed 'Candidate' 118243 / 150608

 

2015-06-16 11:12:11|168188.2|  4.3|INFORM|SpatialFilter_13(SpatialFilterFactory): Processed 'Candidate' 118255 / 150608

Best answer by takashi

Hi @annachukicheva, Regarding the original question, I assumed that the filter and candidates geometries are areas. In your case, both geometries are points, so I think that you can omit the step 1,2 and 5-7.

View original
Did this help you find an answer to your question?
This post is closed to further activity.
It may be a question with a best answer, an implemented idea, or just a post needing no comment.
If you have a follow-up or related question, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

6 replies

takashi
Evangelist
  • takashiEvangelist
  • June 16, 2015
Hi,

 

 

How about tiling the features so that the processing would be divided into groups?

 

 

(1) Add a unique ID attribute to the candidates.

 

(2) Extract geometry as an attribute for each candidate (GeometryExtractor).

 

(3) Send both filters and candidates to a Tiler.

 

(4) Separate the tiled filters and candidates, send them to the SpatialFilter. Set tile column and row attributes to the "Group By" parameter.

 

(5) Select unique Passed features using the DuplicateRemover (Key Attribute = ID attribute of candidate).

 

(7) Restore the original geometry (GeometryReplacer).

 

 

The optimal number of tiles is unknown but I expect 10x10 - 20x20 would be effective.

 

 

Takashi
gio
Contributor
Forum|alt.badge.img+15
  • gioContributor
  • Contributor
  • June 17, 2015
The tiling strategy has as perk that you can use the parallelprocessing option (because of the group by for one)

 

Should yield (albeit maybe slight) performence gain.
K
Forum|alt.badge.img
thanks for the supports , I will check it and let you know . 
A
Forum|alt.badge.img

Hello, @takashi,

first of all, thank you a lot for all your good advices you gave in this share point.

have a question to your process, you wrote here.

First, to my process: I need to check: points objects with another point objects (if they are equal or not). So i made as in your list written until the step Nr. 4.

With step 5 and 7 i m not so sure if i need them to complete. Can you please explain me what for you do these steps? Are they needed in my process, u think?

Thanks,

Anna

takashi
Evangelist
  • takashiEvangelist
  • Best Answer
  • January 27, 2016

Hi @annachukicheva, Regarding the original question, I assumed that the filter and candidates geometries are areas. In your case, both geometries are points, so I think that you can omit the step 1,2 and 5-7.

A
Forum|alt.badge.img
takashi wrote:

Hi @annachukicheva, Regarding the original question, I assumed that the filter and candidates geometries are areas. In your case, both geometries are points, so I think that you can omit the step 1,2 and 5-7.

Thank you @takashi a lot!

Helpful Members This Week

Community Stats

31,392
Posts
119,053
Replies
39,014
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.