Skip to main content
Solved

Explanation of 'Hit Data-shattering' Terminology

  • November 16, 2018
  • 4 replies
  • 76 views

Forum|alt.badge.img

Hi,

 

I am using FME Desktop 2015.1.

I am running a process which includes a LineOnAreaOverlayer which I suspect is the source of the message, within the log window I get a message' Hit data-shattering case as 6463 new nodes were created after phase 2, compared to 1 nodes created after phase 1.'

 

What does Data-Shattering mean within the context of the calculation factory process?

 

In essence I am interested to know if this is something I need to worry about and attempt to mitigate somehow? If any given feature which has experienced data shattering within a transformer, is the output result from that tramsformer correct? Or has some sort of calculation collapse occurred and the result may be incorrect?

I realise that data shattering may occur at a feature level and not necessarily to all feature which enter a transformer.

 

Thanks in advance,

 

Rob

 

Best answer by mark2atsafe

So I believe you are correct that the LineOnAreaOverlayer is the source of the message. If I understand correctly, it means that there is a very complicated geometry operation going on that cannot be resolved, and so we output this - unfortunately vague - message.

This blog post will explain in general what is happening. The section that starts off "Tolerance and Ignorance" in particular highlights the problem. Basically lines are being intersected at a microscopic level, in an iterative process, and suddenly you have got 6463 intersections whereas the previous iteration only had 1 intersection.

Unfortunately I can't really say what state the data will be in. I don't know if it reverts back to the one intersection, or keeps the 6463 intersections; and I can't say if it carried on to completion or just decided to stop at that point. You are correct it probably won't apply to all features, probably just to the overlay of 2 of them. But, again, I don't know what the state of those other features would be (whether they got intersected properly or whether the process gave up before getting that far).

But, fortunately, in FME 2018 we now have new settings for tolerance in transformers like this. As that blog post explains, the solution to that sort of problems is an automated tolerance setting (we call it Anchored Vertex Adjustment) that resolves all of the issues around that massive number of intersections.

So, if you can upgrade to 2018, then set tolerance to Automatic in the LineOnAreaOverlayer and you will not get that error message (I'm 99.999% sure) and the data will be processed correctly.

If you can't upgrade to FME2018, then it's trickier to suggest a solution. One thing to try would be a CoordinateRounder (round all your coordinates to whatever you know your data accuracy to be, say 3 or 4 decimal places) and try the overlay again.

Another potential solution would be to use a Snapper to snap tiny gaps between features. Again use a snapping value of what you know your data accuracy to be.

But as that blog post says, these are not very elegant solutions. They can solve one problem but create another, unless you iterate through the data again and again. The best way to resolve this is to upgrade to FME2018. It's got newer capabilities and will produce a much better result - probably quicker too!

View original
Did this help you find an answer to your question?

4 replies

mark2atsafe
Safer
Forum|alt.badge.img+44
  • Safer
  • Best Answer
  • November 16, 2018

So I believe you are correct that the LineOnAreaOverlayer is the source of the message. If I understand correctly, it means that there is a very complicated geometry operation going on that cannot be resolved, and so we output this - unfortunately vague - message.

This blog post will explain in general what is happening. The section that starts off "Tolerance and Ignorance" in particular highlights the problem. Basically lines are being intersected at a microscopic level, in an iterative process, and suddenly you have got 6463 intersections whereas the previous iteration only had 1 intersection.

Unfortunately I can't really say what state the data will be in. I don't know if it reverts back to the one intersection, or keeps the 6463 intersections; and I can't say if it carried on to completion or just decided to stop at that point. You are correct it probably won't apply to all features, probably just to the overlay of 2 of them. But, again, I don't know what the state of those other features would be (whether they got intersected properly or whether the process gave up before getting that far).

But, fortunately, in FME 2018 we now have new settings for tolerance in transformers like this. As that blog post explains, the solution to that sort of problems is an automated tolerance setting (we call it Anchored Vertex Adjustment) that resolves all of the issues around that massive number of intersections.

So, if you can upgrade to 2018, then set tolerance to Automatic in the LineOnAreaOverlayer and you will not get that error message (I'm 99.999% sure) and the data will be processed correctly.

If you can't upgrade to FME2018, then it's trickier to suggest a solution. One thing to try would be a CoordinateRounder (round all your coordinates to whatever you know your data accuracy to be, say 3 or 4 decimal places) and try the overlay again.

Another potential solution would be to use a Snapper to snap tiny gaps between features. Again use a snapping value of what you know your data accuracy to be.

But as that blog post says, these are not very elegant solutions. They can solve one problem but create another, unless you iterate through the data again and again. The best way to resolve this is to upgrade to FME2018. It's got newer capabilities and will produce a much better result - probably quicker too!


ebygomm
Influencer
Forum|alt.badge.img+38
  • Influencer
  • November 16, 2018

I used to encounter this fairly frequently with an areaonarea overlayer and ignored it without any obvious consequence for a couple of years, however i was never actually keeping the geometries produced by that transformer, just calculating overlapping areas - ymmw


Forum|alt.badge.img
  • Author
  • November 20, 2018

Hi @Mark2AtSafe and @egomm,

 

Thanks for the replies; I suspected that asking about this would only serve to increase the fuzziness of the state of the calculation process; and consequently my decision about whether to rely on the results.

 

Mark, I will look at the blog post and some on the mitigation strategies that you have suggested. (Due to corporate software repackaging and deploy I may be with 2015 for some time, it is a shame as the new vertex adjustment tolerance settings sound great.)

 

Egomm, I had also previousy seen this behaviour with AreaOnArea as well, I think I ignored it or downsampled the data to mitigate the matter, in this case however I can't do the same.

 

 

Thanks for the advice,

 

Rob


olivier
Contributor
Forum|alt.badge.img+7
  • Contributor
  • July 31, 2019

I frequently face this issue with the AreaAmalgamator even in FME 2019.0. Generally I get it solved by adding a CoordinateRounder in front. When I am in meters I use just 1 decimal and it is good enough for my data. I wonder though why Safe didn't add a tolerance option to the AreaAmalgamator.


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings