Solved

Line density vector to raster

K

I have a question similar to this.

However, following and reading the links, it is not addressing my task, so far I can tell.

I have a million lines, which I need to analyze to a line density raster, usually done in ArcMap Spatial Analyst Line Density function. The function needs to: 1) take in input lines, 2) adjust cell size, 3) search radius, 4) area unit. I can do it ArcMap, no problem, but as the analysis task is a part of a longer data extraction and modifying process in FME Workbench, I try to keep it there.

icon

Best answer by dmitribagh 13 February 2020, 00:50

View original

11 replies

dmitribagh
Rank
Userlevel 2
Badge +11

Hi @karl_zinglersen,

I am not sure whether what I tried is exactly what you are looking for, nevertheless, the approach might work after some modifications. RasterConvolver allows applying filters called kernels (or lenses) to each pixel and calculate new pixel values. It comes with a set of predefined kernels in a few sizes, but we can create kernels of bigger sizes and with different properties. Check the attached template - it generates a kernel, which looks like a circle of 1's, its size is a parameter in the workspace (search diameter), and applies it to the input dataset, some road centerlines in this case. You also can set the cell size for rasterization. The more lines go through the kernel, the bigger the value of the pixels inside it. I added some coloring just so that the results look more prominent (thin lines are the source data):

I am not a big expert in using RasterConvolver - maybe this operation could be done with just one transformer, I used two to remove the bright pixels representing original line pixels.

 

 

I hope this helps.

Dmitri

 

 

roaddensity.fmwt

 

K

Hi @karl_zinglersen,

I am not sure whether what I tried is exactly what you are looking for, nevertheless, the approach might work after some modifications. RasterConvolver allows applying filters called kernels (or lenses) to each pixel and calculate new pixel values. It comes with a set of predefined kernels in a few sizes, but we can create kernels of bigger sizes and with different properties. Check the attached template - it generates a kernel, which looks like a circle of 1's, its size is a parameter in the workspace (search diameter), and applies it to the input dataset, some road centerlines in this case. You also can set the cell size for rasterization. The more lines go through the kernel, the bigger the value of the pixels inside it. I added some coloring just so that the results look more prominent (thin lines are the source data):

I am not a big expert in using RasterConvolver - maybe this operation could be done with just one transformer, I used two to remove the bright pixels representing original line pixels.

 

 

I hope this helps.

Dmitri

 

 

roaddensity.fmwt

 

Hi Dmitri @dmitribagh

Looks promising, than you! I'll test the next days back in office and give a feedback on the results.

Karl

K

Hi @karl_zinglersen,

I am not sure whether what I tried is exactly what you are looking for, nevertheless, the approach might work after some modifications. RasterConvolver allows applying filters called kernels (or lenses) to each pixel and calculate new pixel values. It comes with a set of predefined kernels in a few sizes, but we can create kernels of bigger sizes and with different properties. Check the attached template - it generates a kernel, which looks like a circle of 1's, its size is a parameter in the workspace (search diameter), and applies it to the input dataset, some road centerlines in this case. You also can set the cell size for rasterization. The more lines go through the kernel, the bigger the value of the pixels inside it. I added some coloring just so that the results look more prominent (thin lines are the source data):

I am not a big expert in using RasterConvolver - maybe this operation could be done with just one transformer, I used two to remove the bright pixels representing original line pixels.

 

 

I hope this helps.

Dmitri

 

 

roaddensity.fmwt

 

Hi Dmitri,

Today I've tested the workspace, and a nice output is produced with great focus on the concentration of lines. I got quite a shock when I unfolded the "create simple round kernel"-group. Quite some work there, I realize. Good job.

Unlike roads, my task is a bit more complicated, as I will have multiple lines not only intersecting, but lying more or less on top. Consequently, I need information in the raster on how many kilometers or meters of lines there are within each raster cell. So far I read the output from the workspace, the result c. 1 is highest value = there is a road, and below 1 = there is a road nearby.

How would it be possible to print the number or lines - or even better - the line length of a trawl section into each cell?

Karl

dmitribagh
Rank
Userlevel 2
Badge +11

Hi Dmitri,

Today I've tested the workspace, and a nice output is produced with great focus on the concentration of lines. I got quite a shock when I unfolded the "create simple round kernel"-group. Quite some work there, I realize. Good job.

Unlike roads, my task is a bit more complicated, as I will have multiple lines not only intersecting, but lying more or less on top. Consequently, I need information in the raster on how many kilometers or meters of lines there are within each raster cell. So far I read the output from the workspace, the result c. 1 is highest value = there is a road, and below 1 = there is a road nearby.

How would it be possible to print the number or lines - or even better - the line length of a trawl section into each cell?

Karl

Hi Karl @karl_zinglersen,

I'll experiment a bit more and let you know. Do you have a small sample of your data that you could share?

Dmitri

 

dmitribagh
Rank
Userlevel 2
Badge +11

Hi Karl @karl_zinglersen,

How about this? Have a look at this screenshot made in Data Inspector

I used a totally different approach - I converted the network (I used rivers this time) to a point cloud, overlaid it with a raster with row and column numbers as pixel values so that each point in the point cloud knows to which raster cell it will eventually belong. Then I calculated how many points I have per cell (grouping by row and column) and rasterized the point cloud.

Check the feature information on the right - the first three bands are RGB, scaled for a nice visual representation of the length, the fourth band is the original number, which is a relatively accurate length estimate (982 meters of streams inside a 250 by 250 square).

 

 

The length estimate can be more accurate and less accurate depending on whether it is needed - the price we pay for a more accurate number is performance. Going from lines to a point cloud is fast, but this conversion creates points only from vertices. In order to have a point per, for example, each meter, we have to densify the lines, and that will be slowest part of the process - with millions of lines it can take hours.

 

 

In this example, with ~35000 lines, it took 8 seconds to generate this raster with no densification (and less accurate length estimate), and 40 seconds with densification (the length number is quite accurate).

 

 

So what do you think?

Dmitri

K

Hi Karl @karl_zinglersen,

How about this? Have a look at this screenshot made in Data Inspector

I used a totally different approach - I converted the network (I used rivers this time) to a point cloud, overlaid it with a raster with row and column numbers as pixel values so that each point in the point cloud knows to which raster cell it will eventually belong. Then I calculated how many points I have per cell (grouping by row and column) and rasterized the point cloud.

Check the feature information on the right - the first three bands are RGB, scaled for a nice visual representation of the length, the fourth band is the original number, which is a relatively accurate length estimate (982 meters of streams inside a 250 by 250 square).

 

 

The length estimate can be more accurate and less accurate depending on whether it is needed - the price we pay for a more accurate number is performance. Going from lines to a point cloud is fast, but this conversion creates points only from vertices. In order to have a point per, for example, each meter, we have to densify the lines, and that will be slowest part of the process - with millions of lines it can take hours.

 

 

In this example, with ~35000 lines, it took 8 seconds to generate this raster with no densification (and less accurate length estimate), and 40 seconds with densification (the length number is quite accurate).

 

 

So what do you think?

Dmitri

Hi Dmitri,

Thank you very much for your effort. The lines to be processed are consisting of only 2 vertices - a start and an end vertex (typed in by the captains), but span across several kilometers and counting 3 million objects, so densifying into multiple vertices, and then count these would take some processing time.

In the GDAL library we have the gdal_rasterizer and with the -burn flag you can insert multiple vector values into a cell. Maybe we can call GDAL from FME? I know GDAL readers and writers is a part of FME, but I am not aware if the GDAL programs are.

 

In QGIS3, also utilizing GDAL, we have the function Sum Line Length, which calculates line lengths and counts within polygons, e.g. a polygon grid of 1000x1000m. The vector file size is multiplied with c. 3. That could be another approach to follow in FME and still being within the automated workflow. Sample data is in EPSG:32622

Sample data attached: trawl_1999-2019_PRA.zip

 

dmitribagh
Rank
Userlevel 2
Badge +11

Hi Karl @karl_zinglersen,

FME actually also can tile and calculate length per tile/cell very easily - even without creating any polygons. Tiler will do the splitting of all the lines. Aggregator by _row and _column will combine all the lines in each single (imaginary) tile, and then LengthCalculator will count the total length per aggregate. Or, you can calculate the length after Tiler and aggregate with summing the length attribute. Then you can rasterize this dataset using length in NumericRasterizer (through 3DForcer - set height = length).

 

The problem is performance - for millions of lines stretching across dozens of kilometers will take a lot of time, my guess is it will be many hours if not days.

 

On the other hand, a point cloud approach requires only a single vector operation - densification, the rest is happening in the point cloud/raster land, fast and efficient. I combined 16 copies of your original dataset (~390,000 features) and ran them through densification to 10 meters and cell size 1000 m. I got a 400,000,000 points point cloud, from which I generated the following raster, all together in less than 13 minutes (just for convenience, I reprojected it to UTM), and a single copy produces the output in 47 seconds.

 

Dmitri

densityraster.fmw

K

Hi Karl @karl_zinglersen,

FME actually also can tile and calculate length per tile/cell very easily - even without creating any polygons. Tiler will do the splitting of all the lines. Aggregator by _row and _column will combine all the lines in each single (imaginary) tile, and then LengthCalculator will count the total length per aggregate. Or, you can calculate the length after Tiler and aggregate with summing the length attribute. Then you can rasterize this dataset using length in NumericRasterizer (through 3DForcer - set height = length).

 

The problem is performance - for millions of lines stretching across dozens of kilometers will take a lot of time, my guess is it will be many hours if not days.

 

On the other hand, a point cloud approach requires only a single vector operation - densification, the rest is happening in the point cloud/raster land, fast and efficient. I combined 16 copies of your original dataset (~390,000 features) and ran them through densification to 10 meters and cell size 1000 m. I got a 400,000,000 points point cloud, from which I generated the following raster, all together in less than 13 minutes (just for convenience, I reprojected it to UTM), and a single copy produces the output in 47 seconds.

 

Dmitri

densityraster.fmw

Hi Dmitri,

Looks very promising, thank you very much. I will test Friday and get back.

Karl

K

Hi Karl @karl_zinglersen,

FME actually also can tile and calculate length per tile/cell very easily - even without creating any polygons. Tiler will do the splitting of all the lines. Aggregator by _row and _column will combine all the lines in each single (imaginary) tile, and then LengthCalculator will count the total length per aggregate. Or, you can calculate the length after Tiler and aggregate with summing the length attribute. Then you can rasterize this dataset using length in NumericRasterizer (through 3DForcer - set height = length).

 

The problem is performance - for millions of lines stretching across dozens of kilometers will take a lot of time, my guess is it will be many hours if not days.

 

On the other hand, a point cloud approach requires only a single vector operation - densification, the rest is happening in the point cloud/raster land, fast and efficient. I combined 16 copies of your original dataset (~390,000 features) and ran them through densification to 10 meters and cell size 1000 m. I got a 400,000,000 points point cloud, from which I generated the following raster, all together in less than 13 minutes (just for convenience, I reprojected it to UTM), and a single copy produces the output in 47 seconds.

 

Dmitri

densityraster.fmw

Hi Dmitri, I am testing the new workspace.

The first run on the full data set took 48 minutes, which is OK. However, the raster produced had a 6000 meter cell size, but I have an idea that it is due to the distribution of data across multiple UTM zones - some lines are near New Foundland and others in Russia, so I made a bounding box within the Greenland EEZ area for new run.

The first run produced c. 720.000.000 points.

Karl

K

Hi Karl @karl_zinglersen,

FME actually also can tile and calculate length per tile/cell very easily - even without creating any polygons. Tiler will do the splitting of all the lines. Aggregator by _row and _column will combine all the lines in each single (imaginary) tile, and then LengthCalculator will count the total length per aggregate. Or, you can calculate the length after Tiler and aggregate with summing the length attribute. Then you can rasterize this dataset using length in NumericRasterizer (through 3DForcer - set height = length).

 

The problem is performance - for millions of lines stretching across dozens of kilometers will take a lot of time, my guess is it will be many hours if not days.

 

On the other hand, a point cloud approach requires only a single vector operation - densification, the rest is happening in the point cloud/raster land, fast and efficient. I combined 16 copies of your original dataset (~390,000 features) and ran them through densification to 10 meters and cell size 1000 m. I got a 400,000,000 points point cloud, from which I generated the following raster, all together in less than 13 minutes (just for convenience, I reprojected it to UTM), and a single copy produces the output in 47 seconds.

 

Dmitri

densityraster.fmw

Hi Dmitri,

I've now run the workspace with a clipped data set in UTM (data from UTM zones 18-27) and with good success, thank you very much.

For 658.000 line objects 708013186 points were created, and the process took 1 hour 39 minutes 29.3 seconds. A density raster of c. 1000x1000 m (+/- 5 m) cell size in UTM Zone 24 N WGS84 was produced looking very satisfactory.

I skipped the RGB coloring part.

Raster values are between 0 and 586281.

A last question: How can I explain the raster values? Is it number of points or line lengths in meters?

Karl

 

dmitribagh
Rank
Userlevel 2
Badge +11

Hi Dmitri,

I've now run the workspace with a clipped data set in UTM (data from UTM zones 18-27) and with good success, thank you very much.

For 658.000 line objects 708013186 points were created, and the process took 1 hour 39 minutes 29.3 seconds. A density raster of c. 1000x1000 m (+/- 5 m) cell size in UTM Zone 24 N WGS84 was produced looking very satisfactory.

I skipped the RGB coloring part.

Raster values are between 0 and 586281.

A last question: How can I explain the raster values? Is it number of points or line lengths in meters?

Karl

 

@karl_zinglersen it's length - we densify a line by placing vertices every N meters. This means, that each point represents N meters (with some errors along the cell edges, but they shouldn't be too big on those volumes). At the end of the transformation, I multiply the number of points by the densification parameter, and that gives the units (meters). But please check that my math is correct, I make errors easily, sorry about that.

Dmitri

Reply


Community Stats

28,729
Posts
109,527
Replies
36,565
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.