Solved

Tips to improve performance on RasterResampler with Clipper 1.6gb jpeg200 file

  • 22 February 2013
  • 6 replies
  • 5 views
P
Hi all,

 

 

We're currently processing a 1.6gb jpeg2000 file. We're applying a clipper to break it down to a 4x4 grid and then resampling using bilinear interpolation within the RasterResampler transformer.

 

 

Its setup to clip first the resample.

 

 

All files are running locally. The issue is at current estimates, per grid cell will take 40 hours.

 

 

Strangely enough, we processed a 1.2gb jpeg2000 file a while ago with same setting presumadly, and took 19 hours for the entirety of the project. Not sure whats happened.

 

 

Wanted to ask if any has tips/tricks to improve speed or recommend different transformer combinations.

 

 

Cheers,
icon

Best answer by mark2atsafe 25 February 2013, 21:32

View original

6 replies

fmelizard
Rank
Userlevel 4
Badge +13
Hi,

 

 

Maybe the RasterTiler would prove to be a better choice to clip the raster?

 

Also using the parallel processing option on the clipper can improve the preformance. see http://fmepedia.safe.com/articles/Samples_and_Demos/Parallel-Processing

 

 

fmelizard
Rank
Userlevel 4
Badge +13
If the image quality is not much of an issue, the NN option of resampling is usually faster than bilinear.

 

Hope this hepls
fmelizard
Rank
Userlevel 4
Badge +13
NN sampling  is usually faster than bilinear, but if image quality is important to you stick to bilinear.

 

Hope this helps
mark2atsafe
Rank
Userlevel 4
Badge +25
It's uncertain, but the RasterCheckpointer transformer may help. I'm fairly unclear about how this works (and in what scenarios) but it does sometimes help to include it between two raster operations.

 

 

So, put it between the Clipper and the RasterResampler and see if it helps. Run it on a smaller sample dataset if you have one, just to see if it helps.

 

 

Hope this helps

 

 

Mark

 

 

[PS: What I *think* it *might* do: the RasterResampler may be working on the entire 1.6gb dataset, not a tile, because raster writers work by calculating what data will be produced, and then processing the entire source. They wouldn't necessarily work on the mid-translation tiles as a vector writer would. So, by adding a CheckPointer you are forcing the tiled data to be treated as a new "source", which may be quicker]
P
I feel I should do a follow up to my question. I left it to run over the weekend and it took approximately 2 days to process 1 cell of 16. This used clipper -> rasterresampler(Bilinear).

 

 

I've now included the rasterCheckPointer recommended by Manitoba Mark, and we're seeing huge decrease in processing time for each cell. Like 15 MINUTES!. OMG WHAT I hear you say. Believe it.

 

 

Solution at the moment was clipper -> rasterCheckPointers  -> rasterResampler(Bilinear) -> save.

 

 

Thanks a lot!

 

 

N
Thanks for the tip for adding a RasterCheckpointer to workspaces.

 

 

We have a workspace that generates GeoTIFF files from a TIFF and DGN that took 3 hours to run on average, using bilinear interpolation. With the RasterCheckpointer added between the RasterResampler and RasterRotationApplier the workspace running time reduced to 1 min 20 sec.

Reply


Community Stats

28,670
Posts
109,366
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.