Skip to main content
  • EN

Hi,

 

This question is linked to a previous question I posted on the community, about the number accuracy of the Esri ASCII GRID writer of FME.

 

Please see that question for more details and screenshot, but a quick recap of it; when I created a dummy raster with Real32 interpretation, in which an original pixel value would be '0.0250313',

the rounded pixel value would become '0.025',

but the Esri GRID File seems to convert to Real64 interpretation and stores the value as '0.02503128908574581',

whereas I would like the rounded value '0.025' to be written to the file. I noticed that this could save up to 50% in the file size. I suppose that this item is related to the Floating Point Precision and String Formatting in FME.

 

Now I have been investigating some options for an efficient workaround to this question. I.e. to be able to create Esri ASCII Grid files that don't write out pixel values with 16 decimal values. I came up with two methods, but I think they are both not optimal;

  1. Write the Esri ASCII GRID file as regular, read in the result, update the data lines, and create a new smaller version of the file
  2. Fetch the relevant Raster properties, coerce the raster to a PointCloud and then to a MultiPoint, Fetch the (Z) coordinates as a list, and manipulate the list into rows of pixel values.

 

I added a screenshot of the steps and the workspace itself to this case for who is interested.

imageUpon doing so I came across a couple of questions.

1. Do people maybe have some other tips/suggestions for me on other more efficient implementations?

 

That is because both of my methods have some limitations. The first option ends up creating individual features for each pixel, which results in a lot of features and thereby poor performance for large rasters.

The second method does seem to be already a bit more efficient, but it seems to use quite much RAM/memory, and FME also warns me about this when it's creating the list of coordinates for the MultiPoints (Z value representing the pixel values).

 

Another sideremark in my implementation of method 2.

2. It seemed to me that the RasterBandPropertyExtractor returns a list attribute in which the list attribute '_band{0}.band_num_tile_rows' seems incorrect to me (It seems to always return a value of '1').

Luckily the RasterPropertyExtractor does seem to return the correct number of rows and columns for the raster. So could it be that this fixed value of 1 for the number of rows is a bug in the RasterBandPropertyExtractor transformer?

 

Then at last I think that the most of my 2nd method could be simplified by using Python, using the Python FME Objects API for Raster Manipulation.

Specifically I think that the getData() function of one of the FMETile* classes (in particular the FMEReal32Tile class), seems to return the raster data almost directly in the structure I would need it to create the updated Esri ASCII Grid.

 

Unfortunately I'm not so experienced with Python. I had a look online and I found two items with examples of how the Python FME Objects API for Raster Manipulation was used. The example from takashi, and another one from a blogpost on another forum.

While I was able to replicate these examples (I also embedded these in my attached workspace), unfortunately I'm a bit too new with Python and classes to succesfully invoke the getData() Python function for my specific raster.

3. Is there anyone in the community that can maybe help me out with a working PythonCaller script to succesfully invoke the getData() Python function to get the Real32 bit value Raster data values as a nested list?

 

Any feedback/help would be greatly appreciated!

 

Kind regards,

 

Thijs

 

 

Hello @thijsknapen​ , I am commenting on the RasterPropertyExtractor bit of this post (Question 2)! I talked to our development team to see if these are expected results or if this indeed is a bug. This does seem to be expected.

 

I think there is some confusion here between the num_rows and num_cols, stored at the raster level and the band_num_tile_rows and band_num_tile_cols stored at the band level. I am considering filing a ticket to have our documentation be a bit clearer.

 

I would recommend using the distinct raster properties for the number of raster/band rows and columns (eg. num_rows, num_cols), and basically ignore the band tile sizes unless you have a specific reason to care about tile size! Hope this helps, Kailin.

Hello @thijsknapen​ , I am commenting on the RasterPropertyExtractor bit of this post (Question 2)! I talked to our development team to see if these are expected results or if this indeed is a bug. This does seem to be expected.

 

I think there is some confusion here between the num_rows and num_cols, stored at the raster level and the band_num_tile_rows and band_num_tile_cols stored at the band level. I am considering filing a ticket to have our documentation be a bit clearer.

 

I would recommend using the distinct raster properties for the number of raster/band rows and columns (eg. num_rows, num_cols), and basically ignore the band tile sizes unless you have a specific reason to care about tile size! Hope this helps, Kailin.

Hi @kailinatsafe​ ,

 

Thanks for the answer. I must say that I am a bit confused about this answer. A couple of days ago I spoke with Dan M over Live Chat and he mentioned:

"Ahh okay. Just found out that this is a recent bug that we're looking into. I've been able to reproduce this myself as well already. I was a bit suspicious that this looked like a bug too"

 

Also, if this is the intended behaviour, I'm really curious to find out what the values of band_num_tile_rows and band_num_tile_cols represent. In particular, I don't understand how come band_num_tile_cols does seem to equal num_cols, but whereas band_num_tile_rows does not equal num_rows.

 

Thanks for the suggestion of using the RasterPropertyExtractor's for the num_cols and num_rows. That's exactly what I did now. However, for me it's not only about getting to a working solution, but also looking for the how and why of things, and thinking about the efficiency of solutions.

Also, if transformers don't behave in an intended way, I hope that by mentioning it here, it may hopefully help to get recognized as an issue, so that it can be solved for some future release so that others don't also have to go look at workarounds to methods that should work the normal way.

Hi @kailinatsafe​ ,

 

Thanks for the answer. I must say that I am a bit confused about this answer. A couple of days ago I spoke with Dan M over Live Chat and he mentioned:

"Ahh okay. Just found out that this is a recent bug that we're looking into. I've been able to reproduce this myself as well already. I was a bit suspicious that this looked like a bug too"

 

Also, if this is the intended behaviour, I'm really curious to find out what the values of band_num_tile_rows and band_num_tile_cols represent. In particular, I don't understand how come band_num_tile_cols does seem to equal num_cols, but whereas band_num_tile_rows does not equal num_rows.

 

Thanks for the suggestion of using the RasterPropertyExtractor's for the num_cols and num_rows. That's exactly what I did now. However, for me it's not only about getting to a working solution, but also looking for the how and why of things, and thinking about the efficiency of solutions.

Also, if transformers don't behave in an intended way, I hope that by mentioning it here, it may hopefully help to get recognized as an issue, so that it can be solved for some future release so that others don't also have to go look at workarounds to methods that should work the normal way.

Hello @thijsknapen​ , I think Dan and I may have spoken about this after your Live Chat, the issue we initially thought was related ended up being slightly different (caused by something else) than the one you proposed. Apologies for the confusion there, so sorry!

 

I filed TECHPUBS-7638 today to have both the RasterPropertyExtractor and RasterBandPropertyExtractor documentation updated, to improve the attribute/property explanations.

 

The tile size (band_num_tile_rows/cols) used by the raster (stored on each band) really only indicates how FME might access and transport data values for that particular raster. Hence the comment “ignore the band tile sizes unless you have a specific reason to care about them”.

 

Reasons to care about tile size… Advanced users may want to manipulate tile size to structure their data in a more specific way (eg. they want smaller or larger tiles to match a tiling scheme). In this case, you could use a Geotiff Writer to produce a COG, or WebMapTiler for example. (but even then, these attributes would be more of a way to make sure the tiling operation was successful).

 

Also, to address your comment “In particular, I don't understand how come band_num_tile_cols does seem to equal num_cols, but whereas band_num_tile_rows does not equal num_rows.”. Ultimately, this really shouldn’t matter (again its more for FME, as explained above^). Please let me know if you have any other questions about this, I hope this is clearer. Happy to help, Kailin.

Hi @kailinatsafe​ ,

 

Thanks for the answer. I must say that I am a bit confused about this answer. A couple of days ago I spoke with Dan M over Live Chat and he mentioned:

"Ahh okay. Just found out that this is a recent bug that we're looking into. I've been able to reproduce this myself as well already. I was a bit suspicious that this looked like a bug too"

 

Also, if this is the intended behaviour, I'm really curious to find out what the values of band_num_tile_rows and band_num_tile_cols represent. In particular, I don't understand how come band_num_tile_cols does seem to equal num_cols, but whereas band_num_tile_rows does not equal num_rows.

 

Thanks for the suggestion of using the RasterPropertyExtractor's for the num_cols and num_rows. That's exactly what I did now. However, for me it's not only about getting to a working solution, but also looking for the how and why of things, and thinking about the efficiency of solutions.

Also, if transformers don't behave in an intended way, I hope that by mentioning it here, it may hopefully help to get recognized as an issue, so that it can be solved for some future release so that others don't also have to go look at workarounds to methods that should work the normal way.

@thijsknapen​ The Raster documentation has now been updated. Please let us know if anything else is unclear.

Hi @kailinatsafe​ ,

 

Thanks for the answer. I must say that I am a bit confused about this answer. A couple of days ago I spoke with Dan M over Live Chat and he mentioned:

"Ahh okay. Just found out that this is a recent bug that we're looking into. I've been able to reproduce this myself as well already. I was a bit suspicious that this looked like a bug too"

 

Also, if this is the intended behaviour, I'm really curious to find out what the values of band_num_tile_rows and band_num_tile_cols represent. In particular, I don't understand how come band_num_tile_cols does seem to equal num_cols, but whereas band_num_tile_rows does not equal num_rows.

 

Thanks for the suggestion of using the RasterPropertyExtractor's for the num_cols and num_rows. That's exactly what I did now. However, for me it's not only about getting to a working solution, but also looking for the how and why of things, and thinking about the efficiency of solutions.

Also, if transformers don't behave in an intended way, I hope that by mentioning it here, it may hopefully help to get recognized as an issue, so that it can be solved for some future release so that others don't also have to go look at workarounds to methods that should work the normal way.

Hi Liz,

Thanks for the heads up. Looks good to me. Keep up the good work!

Reply


Community Stats

30,126
Posts
114,375
Replies
37,767
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.