Skip to main content
Solved

Generate points with raster constraints

jdh
Contributor
Forum|alt.badge.img+28
  • jdhContributor
  • Contributor

I have a coarse raster where the pixel value indicates the number of points that should be generated within the confines of the pixel.

 

 

I have a fine raster (1 to 2 orders of magnitude higher resolution than coarse raster) where the pixel value indicates whether a point can be placed there. For simplicity purposes this can be a binary mask.

 

 

I would like to generate points according to these two rasters with the additional criteria of a minimum separation between points.

 

 

Thoughts on how to do this in an efficient manner? I'm hoping to avoid the brute force approach of generating a random point, checking the constraints and repeating until the desired number of points is reached.

Best answer by jdh

I ended up solving this by taking advantage of the fact that the python raster API requires you to specify a tile in order to retrieve the rater data.

 

 

Looping through each pixel in the coarse raster I created a tile in the fine raster corresponding to it's boundaries. Converted the tile to numpy flat non zero array, then used random.choice to select the number of elements based on the coarse raster value, then determined the coordinates of the center of the pixel of those elements and exported them as points.
View original
Did this help you find an answer to your question?

14 replies

david_r
Evangelist
  • david_rEvangelist
  • February 18, 2020

Depending on your constraints, I would perhaps try something like:

  • Coerce the binary mask to points, but only where the pixel value = "True". Nodata may be useful for this.
  • Coerce the coarse raster to polygons and overlay the points to check for number of overlaps, i.e. if you need to create or remove points.

Negative side is that coercing rasters to points/polygons is fairly slow in FME, so if the rasters are large and you need to be quick, you may want to look for an alternative.

Otherwise I've had great success so far using the fmeobjects raster support to do stuff in Python. If you convert the rasters to numpy arrays you get great performance.

jdh
Contributor
Forum|alt.badge.img+28
  • jdhContributor
  • Author
  • Contributor
  • February 18, 2020
david_r wrote:

Depending on your constraints, I would perhaps try something like:

  • Coerce the binary mask to points, but only where the pixel value = "True". Nodata may be useful for this.
  • Coerce the coarse raster to polygons and overlay the points to check for number of overlaps, i.e. if you need to create or remove points.

Negative side is that coercing rasters to points/polygons is fairly slow in FME, so if the rasters are large and you need to be quick, you may want to look for an alternative.

Otherwise I've had great success so far using the fmeobjects raster support to do stuff in Python. If you convert the rasters to numpy arrays you get great performance.

Yeah, I'm hoping to avoid coercing the raster to polygons as the expected size will be 37000x35000 pixels. I can chop it up into pieces to avoid memory issues, but vectorizing the raster is painfully slow no matter what.

david_r
Evangelist
  • david_rEvangelist
  • February 18, 2020
jdh wrote:

Yeah, I'm hoping to avoid coercing the raster to polygons as the expected size will be 37000x35000 pixels. I can chop it up into pieces to avoid memory issues, but vectorizing the raster is painfully slow no matter what.

Ooof, agree that's way too big for vector conversion.

jdh
Contributor
Forum|alt.badge.img+28
  • jdhContributor
  • Author
  • Contributor
  • February 18, 2020
david_r wrote:

Ooof, agree that's way too big for vector conversion.

I'm thinking numpy as well, and thensome sort of random population extraction from the mask. But I have no idea how to subset the pixels based on the coarser raster (other than the aforementioned convert to polygons)

jdh
Contributor
Forum|alt.badge.img+28
  • jdhContributor
  • Author
  • Contributor
  • Best Answer
  • February 24, 2020

I ended up solving this by taking advantage of the fact that the python raster API requires you to specify a tile in order to retrieve the rater data.

 

 

Looping through each pixel in the coarse raster I created a tile in the fine raster corresponding to it's boundaries. Converted the tile to numpy flat non zero array, then used random.choice to select the number of elements based on the coarse raster value, then determined the coordinates of the center of the pixel of those elements and exported them as points.
david_r
Evangelist
  • david_rEvangelist
  • February 24, 2020
jdh wrote:

I ended up solving this by taking advantage of the fact that the python raster API requires you to specify a tile in order to retrieve the rater data.

 

 

Looping through each pixel in the coarse raster I created a tile in the fine raster corresponding to it's boundaries. Converted the tile to numpy flat non zero array, then used random.choice to select the number of elements based on the coarse raster value, then determined the coordinates of the center of the pixel of those elements and exported them as points.

Nice! But will it be on the exam? ;-)

But seriously, I'd go watch that presentation.
jdh
Contributor
Forum|alt.badge.img+28
  • jdhContributor
  • Author
  • Contributor
  • February 24, 2020
david_r wrote:

Nice! But will it be on the exam? ;-)

But seriously, I'd go watch that presentation.

This ran in 30s as opposed to the coerce to polygons approach which I killed after 6 hours.

david_r
Evangelist
  • david_rEvangelist
  • February 24, 2020
jdh wrote:

This ran in 30s as opposed to the coerce to polygons approach which I killed after 6 hours.

That is really impressive performance.

jdh
Contributor
Forum|alt.badge.img+28
  • jdhContributor
  • Author
  • Contributor
  • February 24, 2020
david_r wrote:

That is really impressive performance.

Numpy for the win.

jdh
Contributor
Forum|alt.badge.img+28
  • jdhContributor
  • Author
  • Contributor
  • February 24, 2020
jdh wrote:

I ended up solving this by taking advantage of the fact that the python raster API requires you to specify a tile in order to retrieve the rater data.

 

 

Looping through each pixel in the coarse raster I created a tile in the fine raster corresponding to it's boundaries. Converted the tile to numpy flat non zero array, then used random.choice to select the number of elements based on the coarse raster value, then determined the coordinates of the center of the pixel of those elements and exported them as points.

I did need to write a couple of helper function to convert between pixel space (rows/cols) and real space (lat/long) as that's not part of the api.

david_r
Evangelist
  • david_rEvangelist
  • February 24, 2020
jdh wrote:

I did need to write a couple of helper function to convert between pixel space (rows/cols) and real space (lat/long) as that's not part of the api.

That sounds very familiar. I've been doing quite a bit of FMERaster stuff lately as well. And yes, numpy arrays are amazing.

jdh
Contributor
Forum|alt.badge.img+28
  • jdhContributor
  • Author
  • Contributor
  • February 24, 2020
david_r wrote:

That sounds very familiar. I've been doing quite a bit of FMERaster stuff lately as well. And yes, numpy arrays are amazing.

Are you as aggravated by the tile concrete classes vs abstract classes as I am?

david_r
Evangelist
  • david_rEvangelist
  • February 25, 2020
jdh wrote:

Are you as aggravated by the tile concrete classes vs abstract classes as I am?

I am, but my biggest issues are cryptic or non-existing error messages when you do something wrong (not that we're spoiled with examples), difficulties in debugging Python code in FME (I'm looking at you, FMEBandTilePopulator!) and this weird thing where you're not allowed to assign integer values to bands containing floats (not very ducky-typey). Also it bugs me that Safe haven't fixed the small documentation errors that were flagged by Takashi more than a year ago.

jdh
Contributor
Forum|alt.badge.img+28
  • jdhContributor
  • Author
  • Contributor
  • February 25, 2020
david_r wrote:

I am, but my biggest issues are cryptic or non-existing error messages when you do something wrong (not that we're spoiled with examples), difficulties in debugging Python code in FME (I'm looking at you, FMEBandTilePopulator!) and this weird thing where you're not allowed to assign integer values to bands containing floats (not very ducky-typey). Also it bugs me that Safe haven't fixed the small documentation errors that were flagged by Takashi more than a year ago.

Ah yes, the non-existent error message, I know it well.

Reply


Helpful Members This Week

Community Stats

30,958
Posts
117,332
Replies
38,746
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.