Solved

Generate points with raster constraints

jdh
Rank
Badge +22
  • jdh Contributor
  • Contributor
  • 1959 replies

I have a coarse raster where the pixel value indicates the number of points that should be generated within the confines of the pixel.

 

 

I have a fine raster (1 to 2 orders of magnitude higher resolution than coarse raster) where the pixel value indicates whether a point can be placed there. For simplicity purposes this can be a binary mask.

 

 

I would like to generate points according to these two rasters with the additional criteria of a minimum separation between points.

 

 

Thoughts on how to do this in an efficient manner? I'm hoping to avoid the brute force approach of generating a random point, checking the constraints and repeating until the desired number of points is reached.
icon

Best answer by jdh 24 February 2020, 16:31

View original

14 replies

david_r
Rank
Userlevel 4

Depending on your constraints, I would perhaps try something like:

  • Coerce the binary mask to points, but only where the pixel value = "True". Nodata may be useful for this.
  • Coerce the coarse raster to polygons and overlay the points to check for number of overlaps, i.e. if you need to create or remove points.

Negative side is that coercing rasters to points/polygons is fairly slow in FME, so if the rasters are large and you need to be quick, you may want to look for an alternative.

Otherwise I've had great success so far using the fmeobjects raster support to do stuff in Python. If you convert the rasters to numpy arrays you get great performance.

jdh
Rank
Badge +22

Depending on your constraints, I would perhaps try something like:

  • Coerce the binary mask to points, but only where the pixel value = "True". Nodata may be useful for this.
  • Coerce the coarse raster to polygons and overlay the points to check for number of overlaps, i.e. if you need to create or remove points.

Negative side is that coercing rasters to points/polygons is fairly slow in FME, so if the rasters are large and you need to be quick, you may want to look for an alternative.

Otherwise I've had great success so far using the fmeobjects raster support to do stuff in Python. If you convert the rasters to numpy arrays you get great performance.

Yeah, I'm hoping to avoid coercing the raster to polygons as the expected size will be 37000x35000 pixels. I can chop it up into pieces to avoid memory issues, but vectorizing the raster is painfully slow no matter what.

david_r
Rank
Userlevel 4

Yeah, I'm hoping to avoid coercing the raster to polygons as the expected size will be 37000x35000 pixels. I can chop it up into pieces to avoid memory issues, but vectorizing the raster is painfully slow no matter what.

Ooof, agree that's way too big for vector conversion.

jdh
Rank
Badge +22

Ooof, agree that's way too big for vector conversion.

I'm thinking numpy as well, and thensome sort of random population extraction from the mask. But I have no idea how to subset the pixels based on the coarser raster (other than the aforementioned convert to polygons)

jdh
Rank
Badge +22

I ended up solving this by taking advantage of the fact that the python raster API requires you to specify a tile in order to retrieve the rater data.

 

 

Looping through each pixel in the coarse raster I created a tile in the fine raster corresponding to it's boundaries. Converted the tile to numpy flat non zero array, then used random.choice to select the number of elements based on the coarse raster value, then determined the coordinates of the center of the pixel of those elements and exported them as points.
david_r
Rank
Userlevel 4

I ended up solving this by taking advantage of the fact that the python raster API requires you to specify a tile in order to retrieve the rater data.

 

 

Looping through each pixel in the coarse raster I created a tile in the fine raster corresponding to it's boundaries. Converted the tile to numpy flat non zero array, then used random.choice to select the number of elements based on the coarse raster value, then determined the coordinates of the center of the pixel of those elements and exported them as points.

Nice! But will it be on the exam? ;-)

But seriously, I'd go watch that presentation.
jdh
Rank
Badge +22

Nice! But will it be on the exam? ;-)

But seriously, I'd go watch that presentation.

This ran in 30s as opposed to the coerce to polygons approach which I killed after 6 hours.

david_r
Rank
Userlevel 4

This ran in 30s as opposed to the coerce to polygons approach which I killed after 6 hours.

That is really impressive performance.

jdh
Rank
Badge +22

That is really impressive performance.

Numpy for the win.

jdh
Rank
Badge +22

I ended up solving this by taking advantage of the fact that the python raster API requires you to specify a tile in order to retrieve the rater data.

 

 

Looping through each pixel in the coarse raster I created a tile in the fine raster corresponding to it's boundaries. Converted the tile to numpy flat non zero array, then used random.choice to select the number of elements based on the coarse raster value, then determined the coordinates of the center of the pixel of those elements and exported them as points.

I did need to write a couple of helper function to convert between pixel space (rows/cols) and real space (lat/long) as that's not part of the api.

david_r
Rank
Userlevel 4

I did need to write a couple of helper function to convert between pixel space (rows/cols) and real space (lat/long) as that's not part of the api.

That sounds very familiar. I've been doing quite a bit of FMERaster stuff lately as well. And yes, numpy arrays are amazing.

jdh
Rank
Badge +22

That sounds very familiar. I've been doing quite a bit of FMERaster stuff lately as well. And yes, numpy arrays are amazing.

Are you as aggravated by the tile concrete classes vs abstract classes as I am?

david_r
Rank
Userlevel 4

Are you as aggravated by the tile concrete classes vs abstract classes as I am?

I am, but my biggest issues are cryptic or non-existing error messages when you do something wrong (not that we're spoiled with examples), difficulties in debugging Python code in FME (I'm looking at you, FMEBandTilePopulator!) and this weird thing where you're not allowed to assign integer values to bands containing floats (not very ducky-typey). Also it bugs me that Safe haven't fixed the small documentation errors that were flagged by Takashi more than a year ago.

jdh
Rank
Badge +22

I am, but my biggest issues are cryptic or non-existing error messages when you do something wrong (not that we're spoiled with examples), difficulties in debugging Python code in FME (I'm looking at you, FMEBandTilePopulator!) and this weird thing where you're not allowed to assign integer values to bands containing floats (not very ducky-typey). Also it bugs me that Safe haven't fixed the small documentation errors that were flagged by Takashi more than a year ago.

Ah yes, the non-existent error message, I know it well.

Reply


Community Stats

28,661
Posts
109,324
Replies
36,528
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.