Hi Franco,
A point could be located at multiple lines? In other words, a point could be the crossing of multiple lines?
If so, I don't think your workspace produces expected result, since the PointOnLineOverlayer will transfer only one measure value from a line to a point.
I think you will have to think of a way to get correct result first, setting aside the performance issue.
I would use the SpatialRelator and the ListExploder to make copies of points for every line, and then apply the PointOnLineOverlayer, grouping by unique identifier of the lines. If "NAME" value is unique for each line, it can be used as line ID.
Example:
Takashi
Hi Takashi,
I've set it up that way but it is still low performance in SpatialRelator ( i have now 100 routes and about 360000 points) because FME makes here singular comparisons.....maybe i can construct a loop where he only take one route after another? would that be faster? and how i build this up?
Greetz
Franco
Perhaps the NeighborFinder might be faster.
Send the point to the Base port, send the lines to the Candidate port.
- Number of Neighbors to Find: <blank>
- Maximum Distance: <specify 0 or very small value>
- Close Candidate List Name: <specify a list name. e.g. "_list">
Then, apply the ListExploder to the Matched points.
In addition, it's better to remove unnecessary attributes (including parts of the list) before exploding.
Not tested. I'm not sure whether it's really faster...
don't forget to set "Use Incoming List" to the "Conflict Resolution" parameter in the ListExploder.
Anyway, FME has to execute the spatial processing too many times if you process the entire data.
360000 x 100 = 36 million
It may not be avoid to consume considerable time.
An idea for a workaround.
If you clip the points and lines by the same grid cells and add cell ID to them, you can perform the processing with grouping by the cell ID. And also the parallel processing could also be effective.
Hi Takashi,
i've set it up with the NeighborFinder now.....FME process and write the first 35000 objetcts from the ListExploder very quick and then he stops and process about 0.2 % for every Log entry in the Translation Log...so very slow....
I wil also try your other solution...
Thx and Greetz
Franco
Hi Takashi,
Neighborfinder Solution is faster than Spatial Relator, but i need it a little bit faster.....
For the clipping solution i have to take the GridChopper?....right?
how about a solution with looping the routes? Ho do you think about that?
Greetz
Franco
Maybe this is easier.
BoundingBoxAccumulator:
Create a bounding box covering entire geometries based on the lines.
Don't use the points, since the number of points is too large and also it's enough that the bounding box will cover all the road line.
Tiler:
Create grid cells within the bounding box.
The Tiler adds "_column" and "_rows" attributes to each cell; those can be used as identifier of the cells. Optionally you can also create one ID attribute with concatenating them.
e.g. _cell_id = @Value(_column)_@Value(_row)
Clipper:
Clip the points and lines by the grid cells.
Use "Merge Attributes" option so that the cell ID will be added to the points and lines.
P.S. For the POINTS, the SpatialFilter with
bounding box mode could be faster.
Use Bounding Box: Yes
Hi Takashi,
I'm not sure about this workspace becaus enow i have a memory problem and FME stops when the objects go to clipper....I've never worked befor with tiler or boundingbox....maybe some params are wrong....
and Points only in spatialfilter nad lines with the gridcells in clipper...thats right or? and the results then in overlaying and measureextractor?...thats the way?
by adding clipper i have the memory problem
Greetz
Franco
Memory issue depends on the environment such as OS, machine spec. I cannot specify what is the bottle neck in your environment.
Anywy, my intention looks like this.
Additional comment.
The SpatialFilter and the Clipper in the screenshot will add cell ID attribute ("_column" and "_row") to the points and lines.
You can then set/add the cell ID to "Group By" parameter of the following NeighborFinder and PointOnLineOverlayer.
My intention is to divide the processing into several blocks, so that the performance could be increased and also the Parallel Processing can also be applied to the NeighborFinder and PointOnLineOverlayer.
Ah, I noticed a defect, sorry.
If you clipped the lines before measure generating, the measure values cannot be calculated as expected.
You will have to insert the MeasureGenerator before the Clipper.
Takashi,
yes i understand it all but it looks like boundingboxacc and tiler takes too much memory....params on Boundingboxacc are default and i have tle size on 1 and 1
FME reads whole points in the Spatialfilter and whole original routes into clipper but he stops by Boundingboxacc and tiler...hmmmm
Greetz
Franco
Maybe the tile size is too small. Too many blocks will be meaningless.
Set "Number of Tiles" to the "Tiling Type", and set a number to the "Number of Horizontal/Vertical Tiles".
I guess 5 or less is effective for the number of tiles, but the optimal number is unknown unless you run the workspace several times with different numbers.
Full shot. Hope you accomplish the goal.
Hi Takashi,
without bounding box and clipping i had about 5 hrs 30mins for 366900 points with 100 routes
now i have 52mins!!!!
so it works ....i have to check the result with my colleague on tuesday but on my first look the result is right!!
Thank you very much
this was very important because we have very very many routes to analyze!!!
Greetz
Franco
Good to hear that the dividing approach was effective.
That's much greater improving than I thought!
Hi guys, Hi Takashi;
I was a little bit too optimistic....evrything works good and FME also make the multiple points (one point each route) right...BUT....the measuring is wrong...he calculates the measure from the longest route and write only this measure to all points with the same X,Y (but the points can be on other routes which have the same geometry but not the same length......have add a screenshot to explain
the blue point is on many routes (like 336236-1, 336237-1.....) but alll points get the same measure (i think its the longest route) BUT:
the routes which come from the south have other measuring than the routes which come from the east...
so where is the fault?
Thx
Franco
Hi Franco,
A possible reason I can think of is wrong parameter setting for the "Conflict Resolution" parameter of the List Exploder.
Check whether you have set "Use Incoming List" to the parameter.
Takashi
Hi Takashi,
list exploder:
List Attribute _list {}
acc Mode: Merge Incoming List
Conflict Resolution: Use Incoming List
is Input is ordered by group "NO" right in PointONLineOverlayer?
Cheers
Franco
"No" is correct for the parameter.
How did you set "Group By" of the PointOnLineOverlayer?
group by: _column _row
You have forgotten to add the line ID attribute to the "Group By" parameter.
Hi Takashi,
now i have feature classes where more routes are included.....until to the transformer pointonlineoverlayer everything is going alright...then FME aborts because of memory problems....i have tried certain kinds of parallelism modes...(most of all FME said that the parallelism mode is too high) .nothing really works....i have run this workbench on the best machine in our company.....where can i get some more memory...to set up the number of tiles? or less tiles?....or have i do the worst case.......to divide my 10 feature classes again and again in little portions?
Thanks and Greetz
Franco
Hi Franco,
How about this?
Use two Sorters to sort the points and lines by the same grouping order (i.e. "_column", "_row", and line ID attribute) before the PointOnLineOverlayer.
Then, set "Yes" to "Input is Ordered by" parameter of the PointOnLineOverlayer.
The sorting might increase the processing time, but I expect that this reduces the memory usage in the PointOnLineOverlayer. I don't think parallel processing is effective against the memory issue.
In addition, if you are using FME 32bit version, switch to 64bit version. Usually 64bit version can use larger memory space than 32bit version.
Hope this helps.
Takashi
