floatMaths or floatString limited to 7 digit result?

John_B

Hi All,
I've got some intensive floating maths to do.·The execution time isn't a constraint, so the floatMaths Spin version should be fine.
This is my first dabble·with the floatMath and floatString so I'm just trying a few functions out to see if the numbers are correct.
The routine below only displays a result with 7 digits, with the number of decimal places being determined by the·magnitude of the result. ie.

123.456 x 987.654 = 121931.8
23.456 x 87.654 =··· 2056.013
3.456 x 7.654 =······ 26.45223

········· tempx := fMath.fMul(3.456, 7.654)
········· com1.str(fString.FloatToString(tempx))

Is this correct? Or am I doing something daft? I can't find any documentation that gives an answer (no pun intended).
John

Mike Green

The floating point libraries use what's known as IEEE 754 floating point format (en.wikipedia.org/wiki/IEEE_754-2008) which provides roughly 7 decimal digits of precision (actually 23 bits). There's no discussion in the documentation since IEEE 754 is a widely used standard with lots of documentation already available.

John_B

Thanks Mike, I will have a read of that.
John