Shop OBEX P1 Docs P2 Docs Learn Events
FlexProp: 16.16 fixed vs IEEE float — Parallax Forums

FlexProp: 16.16 fixed vs IEEE float

MicksterMickster Posts: 2,698
in Propeller 2 
dim as single T, A, C

T = 0.001
A = 20000
C = A * T

print C

Compiled with IEEE option, the result is the expected 20.0000
Compiled with the 16.16 option, the result is 20.141

Is this reasonable?

Comments

  • evanhevanh Posts: 16,029

    Yep, because it's stored as binary rather than decimal. In 16.16 format 0.001 = 65.536, or 66 rounded, therefore 20000 * 66 / 65536 = 20.1416015625 ... The "/ 65536" is to convert a 16.16 stored value into a decimal printout.

  • evanhevanh Posts: 16,029
    edited 2021-12-09 10:12

    IEEE also has similar issues but is more subtle. It's the nature of rounding errors rounding at different ratios for different bases and our desire to see clean round numbers when we know it should be.

  • MicksterMickster Posts: 2,698

    @evanh said:
    IEEE also has similar issues but is more subtle. It's the nature of rounding errors rounding at different ratios for different bases and our desire to see clean round numbers when we know it should be.

    :+1:

    I doubt that anyone will notice the 3uS difference :)

  • pik33pik33 Posts: 2,387

    That's why Atari 8-bit Basic used a 6502 in decimal mode.

Sign In or Register to comment.