FlexProp: 16.16 fixed vs IEEE float

Mickster

dim as single T, A, C

T = 0.001
A = 20000
C = A * T

print C

Compiled with IEEE option, the result is the expected 20.0000
Compiled with the 16.16 option, the result is 20.141

Is this reasonable?

evanh

Yep, because it's stored as binary rather than decimal. In 16.16 format 0.001 = 65.536, or 66 rounded, therefore 20000 * 66 / 65536 = 20.1416015625 ... The "/ 65536" is to convert a 16.16 stored value into a decimal printout.

evanh

IEEE also has similar issues but is more subtle. It's the nature of rounding errors rounding at different ratios for different bases and our desire to see clean round numbers when we know it should be.

Mickster

@evanh said:
IEEE also has similar issues but is more subtle. It's the nature of rounding errors rounding at different ratios for different bases and our desire to see clean round numbers when we know it should be.

I doubt that anyone will notice the 3uS difference

pik33

That's why Atari 8-bit Basic used a 6502 in decimal mode.