Skip to main content
Solved

Parallel processing with PointOnAreaOverlayer fails to add polygon attributes to point data


tim_wood
Contributor
Forum|alt.badge.img+8

FME 2017.0 (17271) on Windows Server 2008 R2 Standard.

I'm using PointOnAreaOverlayer to add 3 attributes from a polygon layer (non-overlapping polygons) to a point layer. The polygons are coming from ArcSDE (SQL Server) and the point data comes from SQL Server with geometry in the Geometry data type. The 3 attribute columns already exist in the point layer and I'm using fme_db_operation to UPDATE them in the Writer. Between the PointOnAreaOverlayer and Writer I'm using AttributeManager to assign the polygon attribute values to the appropriate SQL Server table attributes.

If I don't use any parallel processing in the PointOnAreaOverlayer, the translation works and takes about an hour per million points. However, if I group by one of the point layer fields (4 distinct values) and use Aggressive parallel processing, it cuts that time to 40 minutes per million points, but the polygon attributes do not get added to the point data. When I inspect the output from AttributeManager, the attributes that should have been added from the polygon layer are <missing>.

Any ideas?

I'm going to try not exposing the point data attributes to be updated and instead rename the polygon attributes in AttributeManager. However I would expect both methods to work.

Best answer by mark_f

Hi @tim_wood Depending on your data you might be able to use some of the techniques in @Mark2AtSafe blog article https://blog.safe.com/2016/12/parallel-processing-tips-evangelist159/

View original
Did this help you find an answer to your question?

6 replies

jdh
Contributor
Forum|alt.badge.img+28
  • Contributor
  • May 11, 2017

For the parallel processing to work the group by attribute on the point layer, also has to be on the polygon layer.


tim_wood
Contributor
Forum|alt.badge.img+8
  • Author
  • Contributor
  • May 11, 2017

Why? That doesn't make sense to me.


itay
Supporter
Forum|alt.badge.img+16
  • Supporter
  • May 11, 2017

because that way the spatial processing can take place on the features belonging to the same group.

So all point features and polygons need to have the same group by value to be processed together, that is how the group by setting works.


jdh
Contributor
Forum|alt.badge.img+28
  • Contributor
  • May 11, 2017

When parallel processing your features, each group is processed independently of one another, so your points in Group A will only be processed with your polygons in Group A, if your polygons don't share the same attribute, then there will be no polygons in Group A and thus the missing attributes.


Forum|alt.badge.img+2
  • Best Answer
  • May 11, 2017

Hi @tim_wood Depending on your data you might be able to use some of the techniques in @Mark2AtSafe blog article https://blog.safe.com/2016/12/parallel-processing-tips-evangelist159/


tim_wood
Contributor
Forum|alt.badge.img+8
  • Author
  • Contributor
  • May 12, 2017
mark_f wrote:

Hi @tim_wood Depending on your data you might be able to use some of the techniques in @Mark2AtSafe blog article https://blog.safe.com/2016/12/parallel-processing-tips-evangelist159/

Brilliant! Cloner sounds like it's just what I need (and I get to use a new transformer!).

 

I've got 7 polygons which are an area + buffer zones. The points are addresses. I need to assign the buffer zone info to each address. Hopefully with only 7 polygons, Cloner won't impact that much on the processing time and I'll still feel the benefit.

 

I'm really struggling to think of a situation where the polygons would have the same group by attribute as the points. If they did, surely you could just do a straight join...?

 


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings