Question
Dividing Integers, Managing Number Types
How do I specify what type of number I want for arithmetic operations? For example, when use the Attribute Creator and then the Arithmetic Editor to divide two numbers that are integers, the result is a rounded/truncated integer, but I want a number that is 3 decimal places...
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
+22- Safer
Hi,
The ExpressionEvaluator documentation states:
"5 / 4
returns 1, while
5 / 4.0
and
5 / (4 + 0.0)
both return 1.25.
Hope this helps.
+40- Contributor
if you're using any constants in the expresion make them explicitly real numbers (ex. 2.0 instead of 2), if not multiply the entire expression by 1.0.
After your expression evalutaor, use an AttributeRounder with the number of decimal places set to 3
+22- Safer
... plus you can explictly call @double( ) to ensure that whatever argument in there is treated as a double .. . (in the math editor). And be sure, as Itay suggests, to add a .0 to any constants you are working with...
Login to the community
No account yet? Create an account
An FME Account is required to contribute
LoginEnter your E-mail address. We'll send you an e-mail with instructions to reset your password.