Skip to main content
Question

Convert mix of JPEG and b/w TIFF to File Geodatabase raster

tim_wood
Contributor
Forum|alt.badge.img+8

I'm finally posting here after much experimenting with various raster transformers and not really getting anywhere. There are too many permutations of things I've tried to list here, so I'll just outline my requirements.

I have a set of scanned images of map sheets comprised of JPEGs which are 3 band RED8, GREEN8, BLUE8, no palette and TIFFs which are UINT8, 1 palette. I also have a CSV file with georeferencing and clipping data. I can successfully apply the georeferencing data and clip the images.

I want to output the results in 2 ways. Firstly, a set of full colour (3 band RED8, GREEN8, BLUE8 in FME parlance?) TIFFs with TFW world files. Secondly, as a mosaicked image in ESRI format - File Geodatabase Raster and/or ArcSDE raster.

When I load the output TIFFs that came from JPEG source images into FGDB rasters and display them in ArcGIS, dark pixels (they aren't pure black i.e. no NoData values so they should display) are shown white. If I convert the JPEG to a TIFF in IrfanView then use FME to load that into an FGDB raster it displays fine.

Another thing I think I might need to do is convert the source TIFFs to 3 band RED8, GREEN8, BLUE8 so that they go into the FGDB raster like this which enables better looking display in ArcMap. I think I need to use the RasterBandAdder for this but need a bit of help...

8 replies

tim_wood
Contributor
Forum|alt.badge.img+8
  • tim_woodContributor
  • Author
  • Contributor
  • August 16, 2017
OK. I've found out why the image isn't displaying in ArcGIS correctly. It's because the Blue band has "NoData" as it's value for those pixels. I've only done a quick spot check, but I haven't found any other bands with "NoData" values, just the Blue one. I have no idea how this has happened.

 

To expand on my need to increase the source TIFFs to 3 band RED8, GREEN8, BLUE8... This is because some of them will be merged with JPEGs with a higher colour depth in the mosaicked raster.

 

"NoData" seems to be a key issue here which I think I've now fixed for the source TIFFs. I use the RasterBandNoDataRemover, then expand the source TIFFs to 3 band RED8, GREEN8, BLUE8 with the RasterPaletterResolver, then use the RasterExpressionEvaluator to subtract 1 from the pixel value. So 255 becomes 254 but 0 stays as 0. Then I can use the RasterBandNoDataSetter to set the NoData value to 255. However, after the Clipper, I now get "2017-08-16 19:08:28| 17.1| 3.1|WARN |TIFF writer: One or more pixels clipped to fit 1 bit domain." To be continued...

 

redgeographics
Celebrity
Forum|alt.badge.img+49

Maybe if you use the RasterCoercer to set everything to RGB24?

tim_wood
Contributor
Forum|alt.badge.img+8
  • tim_woodContributor
  • Author
  • Contributor
  • August 17, 2017
redgeographics wrote:

Maybe if you use the RasterCoercer to set everything to RGB24?

Thanks. I'm sure I tried that early on but I've learnt so much about rasters and processing them in FME since that it's probably worth looking again.

 

 

tim_wood
Contributor
Forum|alt.badge.img+8
  • tim_woodContributor
  • Author
  • Contributor
  • August 17, 2017

I've attached my current process for TIFFs which isn't working. I get a corrupt TIFF out at the end and FME logs this warning: "TIFF writer: One or more pixels clipped to fit 1 bit domain".

The source TIFFs are 1 bit CCITT T.6 compressed. I have to use the RasterPaletteResolver to remove the palette otherwise I can't use the RasterInterpretationCoercer to convert it to RGB24.

The FeatureMerger joins georeferencing data from a CSV which is then used by the RasterGeoreferencer. I'm using the RasterBandNoDataSetter because the pixel values of the data are now 0 and 255, and I want to keep the white background.

I have previously used RasterExpressionEvaluator to adjust the pixel values (e.g. A[0]-1) so that there's no conflict between data and NoData. I've also tried doing the conversion to RGB24 after the Clipper but haven't had any success with that either.

tim_wood
Contributor
Forum|alt.badge.img+8
  • tim_woodContributor
  • Author
  • Contributor
  • August 17, 2017
tim_wood wrote:

I've attached my current process for TIFFs which isn't working. I get a corrupt TIFF out at the end and FME logs this warning: "TIFF writer: One or more pixels clipped to fit 1 bit domain".

The source TIFFs are 1 bit CCITT T.6 compressed. I have to use the RasterPaletteResolver to remove the palette otherwise I can't use the RasterInterpretationCoercer to convert it to RGB24.

The FeatureMerger joins georeferencing data from a CSV which is then used by the RasterGeoreferencer. I'm using the RasterBandNoDataSetter because the pixel values of the data are now 0 and 255, and I want to keep the white background.

I have previously used RasterExpressionEvaluator to adjust the pixel values (e.g. A[0]-1) so that there's no conflict between data and NoData. I've also tried doing the conversion to RGB24 after the Clipper but haven't had any success with that either.

I've read these two articles but am still not sure what to do. Should I be using an Alpha band?

 

https://knowledge.safe.com/articles/19620/raster-clipping-1.html

 

https://knowledge.safe.com/articles/18764/setting-alpha-vs-nodata-color-and-numeric-data.html

 

tim_wood
Contributor
Forum|alt.badge.img+8
  • tim_woodContributor
  • Author
  • Contributor
  • August 17, 2017
tim_wood wrote:

I've attached my current process for TIFFs which isn't working. I get a corrupt TIFF out at the end and FME logs this warning: "TIFF writer: One or more pixels clipped to fit 1 bit domain".

The source TIFFs are 1 bit CCITT T.6 compressed. I have to use the RasterPaletteResolver to remove the palette otherwise I can't use the RasterInterpretationCoercer to convert it to RGB24.

The FeatureMerger joins georeferencing data from a CSV which is then used by the RasterGeoreferencer. I'm using the RasterBandNoDataSetter because the pixel values of the data are now 0 and 255, and I want to keep the white background.

I have previously used RasterExpressionEvaluator to adjust the pixel values (e.g. A[0]-1) so that there's no conflict between data and NoData. I've also tried doing the conversion to RGB24 after the Clipper but haven't had any success with that either.

RasterPaletteResolver gives me 3 bands so I'm not sure the RasterInterpretationCoercer is necessary.

 

 

tim_wood
Contributor
Forum|alt.badge.img+8
  • tim_woodContributor
  • Author
  • Contributor
  • August 18, 2017

Thanks to Dave Campanas at Safe (@DaveAtSafe?) for getting me to the solution for TIFFs:

"This is a bit tricky. The settings from the incoming Tiff data are being propagated to the output through the format attributes on the source image. These attributes override the bits per cell and compression set on the output, leading to the 1 bit output. These format attributes are designed to make TIFF to TIFF conversions simpler, but do assume that the output structure matches the input.

I have added a BulkAttributeRemover to your workspace to remove these attributes, and have also unset the bits per pixel setting in the output feature type so that it will default to the standard 8 bits per band. The output now appears correct."

(I had been trying the bits per pixels with and without values).

tim_wood
Contributor
Forum|alt.badge.img+8
  • tim_woodContributor
  • Author
  • Contributor
  • August 18, 2017

JPEG issue solved :-)

Some pixels have a value of zero in one band (often the blue band it seems), which is the NoData value. It seems that even though other bands have non-zero values in them, the NoData value in the blue band overrides them. So an RGB value of 40,37,0 will get displayed (or rather not displayed!) as NoData, rather than the dark grey it is meant to be. I assume this is expected behaviour...?

Anyway, my fix for this is the same as for the 1-bit TIFFs. I use the RasterExpressionEvaluator to increment the values in each band by 1. Then I can use the RasterBandNoDataSetter to set the NoData value to zero. It would be nice to only do this for pixels with a value of zero but it's not crucial.

Reply


Helpful Members This Week

Community Stats

31,281
Posts
118,651
Replies
38,870
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.