Solved

analyzing points on routes


Badge +6

I have for example 100 routes and about 5000 points......these 100 routes are identically until certain points (mean different lengths)...so i want as a result every point with every route with measure extract in an attribute...I#ve done that and the result is right...My Question now is if there is any possibility to have a better calculation performance....now i have for this workbench (screenshot) about 40 mins calculation time. Can i place the transformers in another order or can change some options?

Thank you and greetz

 

Franco

 

 

 

icon

Best answer by takashi 22 May 2015, 14:42

View original

33 replies

Userlevel 2
Badge +17
Hi Franco,

 

 

A point could be located at multiple lines? In other words, a point could be the crossing of multiple lines?

 

If so, I don't think your workspace produces expected result, since the PointOnLineOverlayer will transfer only one measure value from a line to a point.

 

I think you will have to think of a way to get correct result first, setting aside the performance issue.

 

 

I would use the SpatialRelator and the ListExploder to make copies of points for every line, and then apply the PointOnLineOverlayer, grouping by unique identifier of the lines. If "NAME" value is unique for each line, it can be used as line ID.

 

 

Example:

 

 

 

Takashi
Badge +6

Hi Takashi,

I've set it up that way but it is still low performance in SpatialRelator ( i have now 100 routes and about 360000 points) because FME makes here singular comparisons.....maybe i can construct a loop where he only take one route after another? would that be faster? and how i build this up?

 

 

Greetz

 

Franco
Userlevel 2
Badge +17
Perhaps the NeighborFinder might be faster.

 

 

Send the point to the Base port, send the lines to the Candidate port.

 

- Number of Neighbors to Find: <blank>

 

- Maximum Distance: <specify 0 or very small value>

 

- Close Candidate List Name: <specify a list name. e.g. "_list">

 

 

Then, apply the ListExploder to the Matched points.

 

In addition, it's better to remove unnecessary attributes (including parts of the list) before exploding.

 

 

Not tested. I'm not sure whether it's really faster...

 

Userlevel 2
Badge +17
don't forget to set "Use Incoming List" to the "Conflict Resolution" parameter in the ListExploder.

 

 

Anyway, FME has to execute the spatial processing too many times if you process the entire data.

 

360000 x 100 = 36 million

 

It may not be avoid to consume considerable time.

 

 

An idea for a workaround.

 

If you clip the points and lines by the same grid cells and add cell ID to them, you can perform the processing with grouping by the cell ID. And also the parallel processing could also be effective.
Badge +6

Hi Takashi,

i've set it up with the NeighborFinder now.....FME process and write the first 35000 objetcts from the ListExploder very quick and then he stops and process about 0.2 % for every Log entry in the Translation Log...so very slow....

I wil also try your other solution...

Thx and Greetz

 

Franco
Badge +6

Hi Takashi,

Neighborfinder Solution is faster than Spatial Relator, but i need it a little bit faster.....

For the clipping solution i have to take the GridChopper?....right?

how about a solution with looping the routes? Ho do you think about that?

Greetz

Franco

Userlevel 2
Badge +17
Maybe this is easier.

 

 

BoundingBoxAccumulator:

 

Create a bounding box covering entire geometries based on the lines.

 

Don't use the points, since the number of points is too large and also it's enough that the bounding box will cover all the road line.

 

 

Tiler:

 

Create grid cells within the bounding box.

 

The Tiler adds "_column" and "_rows" attributes to each cell; those can be used as identifier of the cells. Optionally you can also create one ID attribute with concatenating them.

 

e.g. _cell_id = @Value(_column)_@Value(_row)

 

 

Clipper:

 

Clip the points and lines by the grid cells.

 

Use "Merge Attributes" option so that the cell ID will be added to the points and lines.
Userlevel 2
Badge +17
P.S. For the POINTS, the SpatialFilter with bounding box mode could be faster.

 

Use Bounding Box: Yes
Badge +6

Hi Takashi,

I'm not sure about this workspace becaus enow i have a memory problem and FME stops when the objects go to clipper....I've never worked befor with tiler or boundingbox....maybe some params are wrong....

and Points only in spatialfilter nad lines with the gridcells in clipper...thats right or? and the results then in overlaying and measureextractor?...thats the way?

by adding clipper i have the memory problem

Greetz

 

Franco
Userlevel 2
Badge +17
Memory issue depends on the environment such as OS, machine spec. I cannot specify what is the bottle neck in your environment.

 

 

Anywy, my intention looks like this.

 

Userlevel 2
Badge +17
Additional comment.

 

The SpatialFilter and the Clipper in the screenshot will add cell ID attribute ("_column" and "_row") to the points and lines.

 

You can then set/add the cell ID to "Group By" parameter of the following NeighborFinder and PointOnLineOverlayer.

 

 

My intention is to divide the processing into several blocks, so that the performance could be increased and also the Parallel Processing can also be applied to the NeighborFinder and PointOnLineOverlayer.
Userlevel 2
Badge +17
Ah, I noticed a defect, sorry.

 

If you clipped the lines before measure generating, the measure values cannot be calculated as expected.

 

You will have to insert the MeasureGenerator before the Clipper.
Badge +6

Takashi,

 

yes i understand it all but it looks like boundingboxacc and tiler takes too much memory....params on Boundingboxacc are default and i have tle size on 1 and 1

FME reads whole points in the Spatialfilter and whole original routes into clipper but he stops by Boundingboxacc and tiler...hmmmm

 

Greetz

 

Franco

 

 
Userlevel 2
Badge +17
Maybe the tile size is too small. Too many blocks will be meaningless.

 

 

Set "Number of Tiles" to the "Tiling Type", and set a number to the "Number of Horizontal/Vertical Tiles".

 

I guess 5 or less is effective for the number of tiles, but the optimal number is unknown unless you run the workspace several times with different numbers. 
Userlevel 2
Badge +17
Full shot. Hope you accomplish the goal.

 

Badge +6

Hi Takashi,

without bounding box and clipping i had about 5 hrs 30mins for 366900 points with 100 routes

now i have 52mins!!!!

so it works ....i have to check the result with my colleague on tuesday but on my first look the result is right!!

Thank you very much

this was very important because we have very very many routes to analyze!!!

 

Greetz

 

Franco
Userlevel 2
Badge +17
Good to hear that the dividing approach was effective.

 

That's much greater improving than I thought!
Badge +6

Hi guys, Hi Takashi;

I was a little bit too optimistic....evrything works good and FME also make the multiple points (one point each route) right...BUT....the measuring is wrong...he calculates the measure from the longest route and write only this measure to all points with the same X,Y (but the points can be on other routes which have the same geometry but not the same length......have add a screenshot to explain

 

 

the blue point is on many routes (like 336236-1, 336237-1.....) but alll points get the same measure (i think its the longest route) BUT:

the routes which come from the south have other measuring than the routes which come from the east...

so where is the fault?

 

Thx

Franco

 

 

Userlevel 2
Badge +17
Hi Franco,

 

 

A possible reason I can think of is wrong parameter setting for the "Conflict Resolution" parameter of the List Exploder.

 

Check whether you have set "Use Incoming List" to the parameter.

 

 

Takashi
Badge +6

Hi Takashi,

 

list exploder:

List Attribute _list {}

acc Mode: Merge Incoming List

Conflict Resolution: Use Incoming List

 

 

is Input is ordered by group "NO" right in PointONLineOverlayer?

 

 

Cheers

Franco

 

Userlevel 2
Badge +17
"No" is correct for the parameter.

 

How did you set "Group By" of the PointOnLineOverlayer?
Badge +6
group by: _column _row
Userlevel 2
Badge +17
You have forgotten to add the line ID attribute to the "Group By" parameter.
Badge +6

Hi Takashi,

now i have feature classes where more routes are included.....until to the transformer pointonlineoverlayer everything is going alright...then FME aborts because of memory problems....i have tried certain kinds of parallelism modes...(most of all FME said that the parallelism mode is too high) .nothing really works....i have run this workbench on the best machine in our company.....where can i get some more memory...to set up the number of tiles? or less tiles?....or have i do the worst case.......to divide my 10 feature classes again and again in little portions?

 

 

Thanks and Greetz

 

Franco
Userlevel 2
Badge +17
Hi Franco,

 

 

How about this?

 

Use two Sorters to sort the points and lines by the same grouping order (i.e. "_column", "_row", and line ID attribute) before the PointOnLineOverlayer.

 

Then, set "Yes" to "Input is Ordered by" parameter of the PointOnLineOverlayer.

 

 

The sorting might increase the processing time, but I expect that this reduces the memory usage in the PointOnLineOverlayer. I don't think parallel processing is effective against the memory issue.

 

 

In addition, if you are using FME 32bit version, switch to 64bit version. Usually 64bit version can use larger memory space than 32bit version. 

 

 

Hope this helps.

 

 

Takashi

Reply