Skip to main content
Solved

Tips to improve performance on RasterResampler with Clipper 1.6gb jpeg200 file

  • February 22, 2013
  • 6 replies
  • 17 views
P
Hi all,

 

 

We're currently processing a 1.6gb jpeg2000 file. We're applying a clipper to break it down to a 4x4 grid and then resampling using bilinear interpolation within the RasterResampler transformer.

 

 

Its setup to clip first the resample.

 

 

All files are running locally. The issue is at current estimates, per grid cell will take 40 hours.

 

 

Strangely enough, we processed a 1.2gb jpeg2000 file a while ago with same setting presumadly, and took 19 hours for the entirety of the project. Not sure whats happened.

 

 

Wanted to ask if any has tips/tricks to improve speed or recommend different transformer combinations.

 

 

Cheers,

Best answer by mark2atsafe

It's uncertain, but the RasterCheckpointer transformer may help. I'm fairly unclear about how this works (and in what scenarios) but it does sometimes help to include it between two raster operations.

 

 

So, put it between the Clipper and the RasterResampler and see if it helps. Run it on a smaller sample dataset if you have one, just to see if it helps.

 

 

Hope this helps

 

 

Mark

 

 

[PS: What I *think* it *might* do: the RasterResampler may be working on the entire 1.6gb dataset, not a tile, because raster writers work by calculating what data will be produced, and then processing the entire source. They wouldn't necessarily work on the mid-translation tiles as a vector writer would. So, by adding a CheckPointer you are forcing the tiled data to be treated as a new "source", which may be quicker]
View original
Did this help you find an answer to your question?
This post is closed to further activity.
It may be a question with a best answer, an implemented idea, or just a post needing no comment.
If you have a follow-up or related question, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

6 replies

fmelizard
Contributor
Forum|alt.badge.img+17
  • fmelizardContributor
  • Contributor
  • February 22, 2013
Hi,

 

 

Maybe the RasterTiler would prove to be a better choice to clip the raster?

 

Also using the parallel processing option on the clipper can improve the preformance. see http://fmepedia.safe.com/articles/Samples_and_Demos/Parallel-Processing

 

 

fmelizard
Contributor
Forum|alt.badge.img+17
  • fmelizardContributor
  • Contributor
  • February 22, 2013
If the image quality is not much of an issue, the NN option of resampling is usually faster than bilinear.

 

Hope this hepls
fmelizard
Contributor
Forum|alt.badge.img+17
  • fmelizardContributor
  • Contributor
  • February 22, 2013
NN sampling  is usually faster than bilinear, but if image quality is important to you stick to bilinear.

 

Hope this helps
mark2atsafe
Safer
Forum|alt.badge.img+43
  • mark2atsafeSafer
  • Safer
  • Best Answer
  • February 25, 2013
It's uncertain, but the RasterCheckpointer transformer may help. I'm fairly unclear about how this works (and in what scenarios) but it does sometimes help to include it between two raster operations.

 

 

So, put it between the Clipper and the RasterResampler and see if it helps. Run it on a smaller sample dataset if you have one, just to see if it helps.

 

 

Hope this helps

 

 

Mark

 

 

[PS: What I *think* it *might* do: the RasterResampler may be working on the entire 1.6gb dataset, not a tile, because raster writers work by calculating what data will be produced, and then processing the entire source. They wouldn't necessarily work on the mid-translation tiles as a vector writer would. So, by adding a CheckPointer you are forcing the tiled data to be treated as a new "source", which may be quicker]
P
  • pat8037
  • Author
  • February 26, 2013
I feel I should do a follow up to my question. I left it to run over the weekend and it took approximately 2 days to process 1 cell of 16. This used clipper -> rasterresampler(Bilinear).

 

 

I've now included the rasterCheckPointer recommended by Manitoba Mark, and we're seeing huge decrease in processing time for each cell. Like 15 MINUTES!. OMG WHAT I hear you say. Believe it.

 

 

Solution at the moment was clipper -> rasterCheckPointers  -> rasterResampler(Bilinear) -> save.

 

 

Thanks a lot!

 

 

N
  • nicola
  • June 1, 2015
Thanks for the tip for adding a RasterCheckpointer to workspaces.

 

 

We have a workspace that generates GeoTIFF files from a TIFF and DGN that took 3 hours to run on average, using bilinear interpolation. With the RasterCheckpointer added between the RasterResampler and RasterRotationApplier the workspace running time reduced to 1 min 20 sec.

Helpful Members This Week

Community Stats

30,922
Posts
117,200
Replies
38,727
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.