Skip to main content
Solved

Optimising use of RasterCellCoercer

J
Forum|alt.badge.img
Hi All,

 

    I've been tasked with optimising a workbench that's quite slow.

 

    Basically I take in a very large DEM raster 2m granularity, and need to clip it against a vector dataset of 50m grid squares. The vector dataset then pulls the underlying raster data (RasterCellCoercer), and calculates statistics (StatisticsCalculator) using a group-by from the clipping.

 

    

 

    The clipping part I've got down to about 30minutes.

 

    

 

    The problem is the RasterCellCoercer. That part takes 15hrs! It creates about 300,000,000 points (yes, 300 million!) from the raster (there are about 4.5 Billion points in the source raster, so the other 4.2billion are no "noData" or were culled completely). I can't think of a way to get the number lower because that's how many the features require.

 

    

 

    Any suggestions for a faster way to get the DEM values in a way that they can be grouped for a statisticsCalculator pass?

 

    Would somehow trying to coerce it into a PointCloud be faster? (And if so, how?).

 

    

 

    Thanks,

 

    Jonathan

Best answer by fmelizard

The RasterCellCoercer supports feature tables in FME 2018 builds 18256+, meaning that its performance has been improved and it processes faster now.

View original
Did this help you find an answer to your question?

10 replies

owen
Forum|alt.badge.img+1
  • owen
  • November 26, 2014
Hi, me again. cant you use the RasterBandMinMaxExtractor and/or RasterPropertiesExtractor? Would be speedy if it tells you what you need.

 

 

Or might the Tiler be quicker than the Clipper?
J
Forum|alt.badge.img
  • jonathan
  • Author
  • November 27, 2014

Hi Owen,

 

Thanks for the thoughts. Alas the two extractors names don't tell me what I need. I need things like count and stand deviation as well as max/min. Cunning plan though.

 

 

I did consider tiler, but in theory Clipper should be quicker; that part of the process is only about 30mins. It's the RasterCellCoercer that's slow, and if I were to use the tiler, that would have many more cells to coerce!

 

Ah well, I guess this is just a "big data" problem and thus inherently going to be slow.

Thanks though!

Jonathan

david_r
Evangelist
  • david_rEvangelist
  • November 27, 2014
Hi,

 

 

I've been in the exact same situation. Unfortunately, the RasterCellCoercer is rather slow because it has to create all those features. Feature creation is rather expensive in FME (case in point: ListExploder), performance wise.

 

 

In the end, I ended up reading the raster using the GDAL libraries in a Python script. There you can access all the cell values numerically in using arrays and it's really extremely fast compared to the RasterCellCoercer. Unfortunately, it does means quite a bit of Python coding...

 

 

You should write Safe and tell them you want to be added to my feature request for better raster support in the fmeobjects API, that would make it a lot easier :-)

 

 

David
sunsilk11
Contributor
Forum|alt.badge.img+6
  • sunsilk11Contributor
  • Contributor
  • February 4, 2016

Hi I am also looking to generate points from raster using RasterCellCoercer. It takes about 8 hours to generate point for 1 tile, and I have about 1500 tiles. Is there any alternative option available in FME 2016.

Regarding the GDAL libraries, are there any sample code I could start with?

Thanks

david_r
Evangelist
  • david_rEvangelist
  • February 4, 2016
sunsilk11 wrote:

Hi I am also looking to generate points from raster using RasterCellCoercer. It takes about 8 hours to generate point for 1 tile, and I have about 1500 tiles. Is there any alternative option available in FME 2016.

Regarding the GDAL libraries, are there any sample code I could start with?

Thanks

Remember to upvote my suggestion to give it more visibility: https://knowledge.safe.com/content/idea/19483/raster-cell-support-in-fmeobjects.html

Regarding GDAL, it will depend entirely on which format you're reading. Google is your friend, there is quite a lot of info out there.

jdh
Contributor
Forum|alt.badge.img+28
  • jdhContributor
  • Contributor
  • February 4, 2016

Coercing to point cloud is significantly faster.

See http://knowledge.safe.com/content/idea/22245/spatial-statistics-on-raster-data.html for a workflow.

fmelizard
Contributor
Forum|alt.badge.img+17
  • fmelizardContributor
  • Contributor
  • February 4, 2016
jdh wrote:

Coercing to point cloud is significantly faster.

See http://knowledge.safe.com/content/idea/22245/spatial-statistics-on-raster-data.html for a workflow.

Definitely try going to point cloud. Then manipulate the points using the PointCloud transformers. Coerce to points only at the end. Drastic speed improvements for large amounts of data.

sunsilk11
Contributor
Forum|alt.badge.img+6
  • sunsilk11Contributor
  • Contributor
  • February 4, 2016

Hi, I am reading 2 ASCI files convert these to rasters and clip each to a polygon feature class, then generate points using RasterCellCoercer so that I end up with a point feature class with DTM and DSM height values for each point. This process is taking far too long. See attached workbench.

Any suggestion how the performance can be improved?

jdh
Contributor
Forum|alt.badge.img+28
  • jdhContributor
  • Contributor
  • February 4, 2016
sunsilk11 wrote:

Hi, I am reading 2 ASCI files convert these to rasters and clip each to a polygon feature class, then generate points using RasterCellCoercer so that I end up with a point feature class with DTM and DSM height values for each point. This process is taking far too long. See attached workbench.

Any suggestion how the performance can be improved?

I suggest you start a new thread for this question

fmelizard
Contributor
Forum|alt.badge.img+17
  • fmelizardContributor
  • Contributor
  • Best Answer
  • June 4, 2018

The RasterCellCoercer supports feature tables in FME 2018 builds 18256+, meaning that its performance has been improved and it processes faster now.

Reply


Helpful Members This Week

Community Stats

30,995
Posts
117,478
Replies
38,775
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.