Performance Issues with extracting a large number of points from 1m LIDAR

So, if I'm getting this right, you're trying to extract the values from the Lidar Raster (which is 11TeraBytes in size!) at any given point (of which you have a few million)?

 

 

With those data quantities you're going to have to do a *lot* of optimisation.

 

 

You're kind of on the right track, but only just starting.

 

 

Suggestions:

 

1) Your attempt to get a subset of the lidar data is suboptimal. Instead, take the BBoxAccumulator, and then follow with a FeatureReader, setting it to only select features within the bbox. This reduces the data read in the first place, and removes the need for a clipper too, further benefiting.

 

 

2) Use FME 2015; historically Safe are good at ensuring each version is faster than the last, so it may help.

 

 

3) Then it gets a little more tricky. It depends on the distribution of the points and the data structure. If the points are in groups (ideally spatial, but potentially attribute based), I would suggest splitting them up somehow, and using WorkspaceRunner to process the subsets. This can massively speed things up, even if you only run one child process at a time.

 

 

Those should be enough to get you started; but the core is: split the dataset as much as possible and only process small subsets at a time.

 

See how far you get and update here if it's still slow after some more fiddling.

 

 

Cheers,

 

Jonathan

 

 

 

(P.s. Note: I'm not sure you ever needed the "Dissolver" there, as the output for the BBoxAccumulator should be a single box).