Skip to main content
  • EN

I have a huge road network dataset and a 1km x 1km grid dataset of the US and I'd like to assign each road segment a grid base on line on area overlay.

I could use one of:

  • LineOnAreaOverlay
  • SpatialFilter
  • SpatialRelator

But all of these are blocking transformers.

I'm reading in my grid dataset first and have it waiting for the road network...but when the road network arrives at these transformers...the road segments are held in memory, for a small dataset this is fine, but for due to the size of my road network, this will cause me to run out of memory.

Given I'm reading and writing to a postgres database...I've developed an SQL script which will do exactly what I want to do...but for any future users of this workbench, it would be much more friendly if there were a bunch of transformers rather than a load of SQL.

Is there a better combination of transforms that could help make this process more 'run through'.

Cheers.

In this case I would use the SQLCreator and let the database do the spatial join.

That is usually the best performing way.

Could you have your grid as a normal reader that points to a FeatureReader with a Spatial Filter Intersects for the Roads. If your Accumaltion Mode is Merge Initiator and Result, then you have your grid attributes on each road segment.

 

 

Caveat if a single road segment intersects several grids then it will be read in multiple times.

Could you have your grid as a normal reader that points to a FeatureReader with a Spatial Filter Intersects for the Roads. If your Accumaltion Mode is Merge Initiator and Result, then you have your grid attributes on each road segment.

 

 

Caveat if a single road segment intersects several grids then it will be read in multiple times.

Hey @jdh - this won't get me past the blocking nature of the Spatial Filter. I guess i'll stick with calling a db function at the end of my translation.

Cheers

Hey @jdh - this won't get me past the blocking nature of the Spatial Filter. I guess i'll stick with calling a db function at the end of my translation.

Cheers

At this point you shouldn't need the SpatialFilter. The Road features read in by the FeatureReader will only be those that intersect the trigger polygon, and will have all it's attributes.

If you know what are the coordinates of your grid in advance and you have a way to identify a grid cell in the database by its row and column values, you can simply take the middle point of a line, compute the grid row and column and then associate the corresponding grid cell with your line.

@hughloughrey

The key to a solution is to find a transformer that has a parameter to turn the transformer from group-based (blocking) to feature-based. In this scenario I would go with the Clipper, which will tag the roads with the grid it is clipped by. Make sure the grid data is read first and then set the Clipper Type parameter to Clippers First. That stops it from blocking the roads.

It can make a massive difference to a workspace. See this blog post for an example of how it works.

Obviously the roads will get clipped at the boundaries. To get them back to their original status is not hard. Use a FeatureMerger where the original roads are the Requestor and the clipped roads are the Supplier. Set a list and you will get a list of the grid squares a road intersects with.

But the answer from @jdh is also good. Again you would need to merge the roads together - this time probably with an Aggregator transformer.

Hope that helps.

Could you have your grid as a normal reader that points to a FeatureReader with a Spatial Filter Intersects for the Roads. If your Accumaltion Mode is Merge Initiator and Result, then you have your grid attributes on each road segment.

 

 

Caveat if a single road segment intersects several grids then it will be read in multiple times.

Hey @jdh - sorry i mis-understood your answer - i've now gone back and relooked at it. I hadn't used the FeatureReader before - awesome transformer. Appreciate it! :)

Reply


Community Stats

30,114
Posts
114,307
Replies
37,746
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.