In a bind and need help with decimal number and Float 32 object

grasshopper

I am trying to use a small number like 0.001506785 but it seems that the Float 32 is truncating it to 0.0015067.

Is the limit 7 decimal places?

Mike Green

Yes, roughly 7 significant digits. There's only about 23 bits available for the mantissa of the floating point number and that's enough to give you about 7 decimal digits' worth. There's a Wikipedia article on the IEEE floating point format (which this library uses) that describes how numbers are put together.

cessnapilot

Try to enter value as 0.1506785, then divide it by 1000.0 to obtain a closer float approximation of your original "0.0001506785" real number. Foat32 can really provide you 7 significant digits as mike said. Maybe the problem here is with the code that converts float constants in the source into the IEEE format. Significant digits is just not the same as decimal digits. The difference between the two is really the basic idea of the float representation. Divide your number with a million and you get 0.0000000001506785 again with 7 significant digits (but no more).

Drone

Grasshopper,

Ale contributed the double precision routines available in the Math area on the Propeller Wiki:

http://propeller.wikispaces.com/MATH

Regards,

David