Question

How to reproject .tif images correctly?


Badge

Hi, can anyone assist?

I have thousands of .tif images and reprojected them (esriReprojector (custom transformation)), however they now all have black lines around them and that comes up in the mosaicked image.

I found a couple suggestions that I tried; first I tried using the already mosaicked image and then reproject - while the blank mosaic is projected, once I get to the writer and add the rasters back, the data reverts to the original coordinates.

 

Next I tried using the RasterBandNoDataSetter option but that gave me large white blotches all over my images...

 

I am out of options now BUT I need to get this solved as the on-the-fly projection is not an option for our current project.


17 replies

Badge

@daleatsafe - question posted; thanks!

Badge +22

As a general rule of thumb it's much better to mosiac the tiles, reproject them and then retile them if necessary, rather then reproject and mosaic - due to the black lines you mentioned.

 

 

If that is not possible, you can try setting the RasterBandNoDataSetter (0) to the reprojected images, mosaic, and then a RasterBandNoDataRemover.

 

 

Also, if your original data contains 0 value pixels (178,142,0) - not just pure black (0,0,0) it's a good idea to change that value to 1 before reprojecting, to distinguish them from 0 - NoData values.
Badge +2

Few hints that would maybe help you.

Add RasterCellValueReplacer before your RasterBandNoDataSetter. Replace value <= 15 and New Value = 0.

Add a Sorter before your RasterMosaicker. Sort them by files name, if they are named correctly and logically.

Try reprojecting after the RasterMosaicker.

Try writing into a ECW file.

 

If that doesn't work, I would consider trying to do those steps first directly into ArcGIS if you're able to do it.

Badge

@jdh thanks for your reply...so I tried using the mosaicked image, however, when I use the writer to add the images back, even tho it has the correct coordinates listed, the data is offset - basically reverted to the original coordinates...I assume that they would read their respective .tfw files and use the information there to project the images.

 

Maybe I am not using the RasterMosaicker correctly, but once the Mosaic is created, I use a writer to addRasters; otherwise the mosaic is blank. Am I missing something? I am a relative new user to FME, so forgive my ignorance...

Badge +22

@jdh thanks for your reply...so I tried using the mosaicked image, however, when I use the writer to add the images back, even tho it has the correct coordinates listed, the data is offset - basically reverted to the original coordinates...I assume that they would read their respective .tfw files and use the information there to project the images.

 

Maybe I am not using the RasterMosaicker correctly, but once the Mosaic is created, I use a writer to addRasters; otherwise the mosaic is blank. Am I missing something? I am a relative new user to FME, so forgive my ignorance...

I'm not understanding what you mean by 'when I use the writer to add the images back,'

 

Can you post a screenshot of your workspace, with summary annotations (select the transformers/writer and press ctrl-shift-k, or right click-> show summary annotation)

 

 

 

 

Badge

I'm not understanding what you mean by 'when I use the writer to add the images back,'

 

Can you post a screenshot of your workspace, with summary annotations (select the transformers/writer and press ctrl-shift-k, or right click-> show summary annotation)

 

 

 

 

See image - so if I do not add the source (path) for the images in the writer, the mosaic comes out empty...this is when I project from an already created mosaic...

Badge

Few hints that would maybe help you.

Add RasterCellValueReplacer before your RasterBandNoDataSetter. Replace value <= 15 and New Value = 0.

Add a Sorter before your RasterMosaicker. Sort them by files name, if they are named correctly and logically.

Try reprojecting after the RasterMosaicker.

Try writing into a ECW file.

 

If that doesn't work, I would consider trying to do those steps first directly into ArcGIS if you're able to do it.

I can't seem to find the ECW_Mosaic option...maybe my license level doesn't allow?

Badge +2

I can't seem to find the ECW_Mosaic option...maybe my license level doesn't allow?

I just named the writer like that. It is the ECW writer that you need.

Badge

I just named the writer like that. It is the ECW writer that you need.

Ah ok; I will try your solution and see if that resolves the issue...thanks for the info...

Badge

Few hints that would maybe help you.

Add RasterCellValueReplacer before your RasterBandNoDataSetter. Replace value <= 15 and New Value = 0.

Add a Sorter before your RasterMosaicker. Sort them by files name, if they are named correctly and logically.

Try reprojecting after the RasterMosaicker.

Try writing into a ECW file.

 

If that doesn't work, I would consider trying to do those steps first directly into ArcGIS if you're able to do it.

Tested this on a small subset of my data and seems to work...I am now running on a larger area to be sure (a single island of 284 images) but seems it is processing at about 1% per hour, so it will be over a week before I know how it looks...is there was a way to set parallel processing on the out-of-the-box transformers to speed things up...my next island is over 1600 images!

Badge

Tested this on a small subset of my data and seems to work...I am now running on a larger area to be sure (a single island of 284 images) but seems it is processing at about 1% per hour, so it will be over a week before I know how it looks...is there was a way to set parallel processing on the out-of-the-box transformers to speed things up...my next island is over 1600 images!

Hmmmm, I may have solved the parallel processing issue. FME documentation is not 'beginner friendly' but I tried something and it seems more processors are working now! *fingers crossed*

Badge +2

Hmmmm, I may have solved the parallel processing issue. FME documentation is not 'beginner friendly' but I tried something and it seems more processors are working now! *fingers crossed*

Glad for you it is working! yes, it can take a while, that's normal.

Maybe you could speed up your process with some modifications though :

- Add parallel processing with RasterMosaicker (http://docs.safe.com/fme/2017.0/html/FME_Desktop_Documentation/FME_Transformers_HelpPane/Transformers/rastermosaicker.htm)

- Try to use the Esri Geodatabase (File Geodb Mosaic Dataset) writer (I never used it, don't know what it worth)

But it looks like you're doing fine now! Good luck!

 

Badge

Few hints that would maybe help you.

Add RasterCellValueReplacer before your RasterBandNoDataSetter. Replace value <= 15 and New Value = 0.

Add a Sorter before your RasterMosaicker. Sort them by files name, if they are named correctly and logically.

Try reprojecting after the RasterMosaicker.

Try writing into a ECW file.

 

If that doesn't work, I would consider trying to do those steps first directly into ArcGIS if you're able to do it.

After 5 days, processing the first 284 images, this has worked great! Thank you!! :)

 

Now I will set the other few thousands that will maybe take a few months, because the parallel processing does not work! :(

Badge +2

After 5 days, processing the first 284 images, this has worked great! Thank you!! :)

 

Now I will set the other few thousands that will maybe take a few months, because the parallel processing does not work! :(

Have you considered using FME 2019 64 bits? Do you have a fast machine?

Maybe to put more time to make parallel processing working would worth it! What's few hours when you can save few months? ;) You could create a new topic about that.

Badge

Have you considered using FME 2019 64 bits? Do you have a fast machine?

Maybe to put more time to make parallel processing working would worth it! What's few hours when you can save few months? ;) You could create a new topic about that.

I am using FME 2019 64 bits but it seems the writer is what is taking most of the time and the writer cannot do parallel processing? As I said, pretty new to FME and that is my understanding of how the parallel processing works. My machine is pretty fast, and resources would not be an issue if that was the case. My plan is to run multiple workbench items at the same time and hopefully that will use the additional processors up...but careful not to end up slowing everything else...

 

it is a work in progress I guess, just happy that the output is good and no more black lines in my images! :)

Badge

I am using FME 2019 64 bits but it seems the writer is what is taking most of the time and the writer cannot do parallel processing? As I said, pretty new to FME and that is my understanding of how the parallel processing works. My machine is pretty fast, and resources would not be an issue if that was the case. My plan is to run multiple workbench items at the same time and hopefully that will use the additional processors up...but careful not to end up slowing everything else...

 

it is a work in progress I guess, just happy that the output is good and no more black lines in my images! :)

Just sharing what my workbench looks like fyi...

 

 

Badge

Just sharing what my workbench looks like fyi...

 

 

Custom Transformer with the parallel processing bit...

Reply