Skip to main content
Question

How to reproject .tif images correctly?


Forum|alt.badge.img

Hi, can anyone assist?

I have thousands of .tif images and reprojected them (esriReprojector (custom transformation)), however they now all have black lines around them and that comes up in the mosaicked image.

I found a couple suggestions that I tried; first I tried using the already mosaicked image and then reproject - while the blank mosaic is projected, once I get to the writer and add the rasters back, the data reverts to the original coordinates.

 

Next I tried using the RasterBandNoDataSetter option but that gave me large white blotches all over my images...

 

I am out of options now BUT I need to get this solved as the on-the-fly projection is not an option for our current project.

17 replies

Forum|alt.badge.img
  • Author
  • August 16, 2019

@daleatsafe - question posted; thanks!


jdh
Contributor
Forum|alt.badge.img+28
  • Contributor
  • August 16, 2019

As a general rule of thumb it's much better to mosiac the tiles, reproject them and then retile them if necessary, rather then reproject and mosaic - due to the black lines you mentioned.

 

 

If that is not possible, you can try setting the RasterBandNoDataSetter (0) to the reprojected images, mosaic, and then a RasterBandNoDataRemover.

 

 

Also, if your original data contains 0 value pixels (178,142,0) - not just pure black (0,0,0) it's a good idea to change that value to 1 before reprojecting, to distinguish them from 0 - NoData values.

philippeb
Enthusiast
Forum|alt.badge.img+20
  • Enthusiast
  • August 16, 2019

Few hints that would maybe help you.

Add RasterCellValueReplacer before your RasterBandNoDataSetter. Replace value <= 15 and New Value = 0.

Add a Sorter before your RasterMosaicker. Sort them by files name, if they are named correctly and logically.

Try reprojecting after the RasterMosaicker.

Try writing into a ECW file.

 

If that doesn't work, I would consider trying to do those steps first directly into ArcGIS if you're able to do it.


Forum|alt.badge.img
  • Author
  • August 16, 2019

@jdh thanks for your reply...so I tried using the mosaicked image, however, when I use the writer to add the images back, even tho it has the correct coordinates listed, the data is offset - basically reverted to the original coordinates...I assume that they would read their respective .tfw files and use the information there to project the images.

 

Maybe I am not using the RasterMosaicker correctly, but once the Mosaic is created, I use a writer to addRasters; otherwise the mosaic is blank. Am I missing something? I am a relative new user to FME, so forgive my ignorance...


jdh
Contributor
Forum|alt.badge.img+28
  • Contributor
  • August 16, 2019
awd wrote:

@jdh thanks for your reply...so I tried using the mosaicked image, however, when I use the writer to add the images back, even tho it has the correct coordinates listed, the data is offset - basically reverted to the original coordinates...I assume that they would read their respective .tfw files and use the information there to project the images.

 

Maybe I am not using the RasterMosaicker correctly, but once the Mosaic is created, I use a writer to addRasters; otherwise the mosaic is blank. Am I missing something? I am a relative new user to FME, so forgive my ignorance...

I'm not understanding what you mean by 'when I use the writer to add the images back,'

 

Can you post a screenshot of your workspace, with summary annotations (select the transformers/writer and press ctrl-shift-k, or right click-> show summary annotation)

 

 

 

 


Forum|alt.badge.img
  • Author
  • August 16, 2019
jdh wrote:

I'm not understanding what you mean by 'when I use the writer to add the images back,'

 

Can you post a screenshot of your workspace, with summary annotations (select the transformers/writer and press ctrl-shift-k, or right click-> show summary annotation)

 

 

 

 

See image - so if I do not add the source (path) for the images in the writer, the mosaic comes out empty...this is when I project from an already created mosaic...


Forum|alt.badge.img
  • Author
  • August 16, 2019
philippeb wrote:

Few hints that would maybe help you.

Add RasterCellValueReplacer before your RasterBandNoDataSetter. Replace value <= 15 and New Value = 0.

Add a Sorter before your RasterMosaicker. Sort them by files name, if they are named correctly and logically.

Try reprojecting after the RasterMosaicker.

Try writing into a ECW file.

 

If that doesn't work, I would consider trying to do those steps first directly into ArcGIS if you're able to do it.

I can't seem to find the ECW_Mosaic option...maybe my license level doesn't allow?


philippeb
Enthusiast
Forum|alt.badge.img+20
  • Enthusiast
  • August 19, 2019
awd wrote:

I can't seem to find the ECW_Mosaic option...maybe my license level doesn't allow?

I just named the writer like that. It is the ECW writer that you need.


Forum|alt.badge.img
  • Author
  • August 19, 2019
philippeb wrote:

I just named the writer like that. It is the ECW writer that you need.

Ah ok; I will try your solution and see if that resolves the issue...thanks for the info...


Forum|alt.badge.img
  • Author
  • August 20, 2019
philippeb wrote:

Few hints that would maybe help you.

Add RasterCellValueReplacer before your RasterBandNoDataSetter. Replace value <= 15 and New Value = 0.

Add a Sorter before your RasterMosaicker. Sort them by files name, if they are named correctly and logically.

Try reprojecting after the RasterMosaicker.

Try writing into a ECW file.

 

If that doesn't work, I would consider trying to do those steps first directly into ArcGIS if you're able to do it.

Tested this on a small subset of my data and seems to work...I am now running on a larger area to be sure (a single island of 284 images) but seems it is processing at about 1% per hour, so it will be over a week before I know how it looks...is there was a way to set parallel processing on the out-of-the-box transformers to speed things up...my next island is over 1600 images!


Forum|alt.badge.img
  • Author
  • August 20, 2019
awd wrote:

Tested this on a small subset of my data and seems to work...I am now running on a larger area to be sure (a single island of 284 images) but seems it is processing at about 1% per hour, so it will be over a week before I know how it looks...is there was a way to set parallel processing on the out-of-the-box transformers to speed things up...my next island is over 1600 images!

Hmmmm, I may have solved the parallel processing issue. FME documentation is not 'beginner friendly' but I tried something and it seems more processors are working now! *fingers crossed*


philippeb
Enthusiast
Forum|alt.badge.img+20
  • Enthusiast
  • August 20, 2019
awd wrote:

Hmmmm, I may have solved the parallel processing issue. FME documentation is not 'beginner friendly' but I tried something and it seems more processors are working now! *fingers crossed*

Glad for you it is working! yes, it can take a while, that's normal.

Maybe you could speed up your process with some modifications though :

- Add parallel processing with RasterMosaicker (http://docs.safe.com/fme/2017.0/html/FME_Desktop_Documentation/FME_Transformers_HelpPane/Transformers/rastermosaicker.htm)

- Try to use the Esri Geodatabase (File Geodb Mosaic Dataset) writer (I never used it, don't know what it worth)

But it looks like you're doing fine now! Good luck!

 


Forum|alt.badge.img
  • Author
  • August 23, 2019
philippeb wrote:

Few hints that would maybe help you.

Add RasterCellValueReplacer before your RasterBandNoDataSetter. Replace value <= 15 and New Value = 0.

Add a Sorter before your RasterMosaicker. Sort them by files name, if they are named correctly and logically.

Try reprojecting after the RasterMosaicker.

Try writing into a ECW file.

 

If that doesn't work, I would consider trying to do those steps first directly into ArcGIS if you're able to do it.

After 5 days, processing the first 284 images, this has worked great! Thank you!! :)

 

Now I will set the other few thousands that will maybe take a few months, because the parallel processing does not work! :(


philippeb
Enthusiast
Forum|alt.badge.img+20
  • Enthusiast
  • August 23, 2019
awd wrote:

After 5 days, processing the first 284 images, this has worked great! Thank you!! :)

 

Now I will set the other few thousands that will maybe take a few months, because the parallel processing does not work! :(

Have you considered using FME 2019 64 bits? Do you have a fast machine?

Maybe to put more time to make parallel processing working would worth it! What's few hours when you can save few months? ;) You could create a new topic about that.


Forum|alt.badge.img
  • Author
  • August 23, 2019
philippeb wrote:

Have you considered using FME 2019 64 bits? Do you have a fast machine?

Maybe to put more time to make parallel processing working would worth it! What's few hours when you can save few months? ;) You could create a new topic about that.

I am using FME 2019 64 bits but it seems the writer is what is taking most of the time and the writer cannot do parallel processing? As I said, pretty new to FME and that is my understanding of how the parallel processing works. My machine is pretty fast, and resources would not be an issue if that was the case. My plan is to run multiple workbench items at the same time and hopefully that will use the additional processors up...but careful not to end up slowing everything else...

 

it is a work in progress I guess, just happy that the output is good and no more black lines in my images! :)


Forum|alt.badge.img
  • Author
  • August 23, 2019
awd wrote:

I am using FME 2019 64 bits but it seems the writer is what is taking most of the time and the writer cannot do parallel processing? As I said, pretty new to FME and that is my understanding of how the parallel processing works. My machine is pretty fast, and resources would not be an issue if that was the case. My plan is to run multiple workbench items at the same time and hopefully that will use the additional processors up...but careful not to end up slowing everything else...

 

it is a work in progress I guess, just happy that the output is good and no more black lines in my images! :)

Just sharing what my workbench looks like fyi...

 

 


Forum|alt.badge.img
  • Author
  • August 23, 2019
awd wrote:

Just sharing what my workbench looks like fyi...

 

 

Custom Transformer with the parallel processing bit...


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings