Skip to main content
Solved

How to limit the decimals in the result of the lengthcalculator ?

  • October 9, 2020
  • 3 replies
  • 77 views
C

Hello,

I have a shp layer of lines for which I need to calculate the length of each line then round it to suppress the decimals.

 

I use the lenghtcalculator to get the lenght of each line but for some lines I get results such as 7.566995918750763e-10 or 1.7276990785794134e-9

With these results, I get 0 after the attributerounder. I even tried a stringformatter to reduce the amount of decimals but theses objets also turn into 0.

Is there a way to avoid getting so much precision when calculating the lenght ? Or another way to calculate the lenght (I get the same result with an attribute creator and length() ) ?

 

Many thanks in adance!

Ann

Best answer by david_r

7.566995918750763e-10 (scientific notation) is the same as 0.0000000007566995918750763, so unless you round down to more than 9 decimals the result is actually going to be 0.

I'm guessing these are extremely short line segments?

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

3 replies

david_r
Celebrity
  • david_rCelebrity
  • Best Answer
  • October 9, 2020

7.566995918750763e-10 (scientific notation) is the same as 0.0000000007566995918750763, so unless you round down to more than 9 decimals the result is actually going to be 0.

I'm guessing these are extremely short line segments?

ebygomm
Influencer
Forum|alt.badge.img+46
  • ebygommInfluencer
  • Influencer
  • October 9, 2020

What distance units are you working in, 7.566995918750763e-10 is a very small number which is why it rounds to zero

C
  • cadfmegeo
  • Author
  • October 9, 2020

Thank you very much. I understand better what it means, it is logical that it rounds to 0. Indeed these are extremely short lines. Thanks!

Community Stats

32,154
Posts
122,047
Replies
39,812
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.