Shop OBEX P1 Docs P2 Docs Learn Events
about float calculate setprecision problem — Parallax Forums

about float calculate setprecision problem

ggg558877ggg558877 Posts: 40
edited 2013-03-27 08:36 in Propeller 1
Hi everyone
I have some problem about float calculate setprecision
I want to calculate (10*60+12.3456)*100000 the answer must be 61,234,560
but in propeller 
aa :=math.FMul(math.FAdd(math.FMul(10.0,60.0),12.3456),100000.0)
it show 61,235,000
how can i do to make answer is 61,234,560
thanks

Comments

  • kuronekokuroneko Posts: 3,623
    edited 2013-03-26 21:56
    How did you print the value? By using FloatString's FloatToString method I do get 61234560.
  • ggg558877ggg558877 Posts: 40
    edited 2013-03-26 22:07
    sorry i have wrong example
    I want to calculate (100*60+12.3456)*100000 the answer must be 601,234,560 
    aa :=math.FMul(math.FAdd(math.FMul(100.0,60.0),12.3456),100000.0)
    it show 601234600
  • kuronekokuroneko Posts: 3,623
    edited 2013-03-26 22:25
    You could try FloatString v1.1+ and then use FloatToFormat(aa, 9, 0). Note that you'll run into similar problems for bigger numbers (max width is 9). I'm not aware of any more relaxed FP (printing) package(s).
  • ggg558877ggg558877 Posts: 40
    edited 2013-03-26 23:20
    thanks kuroneko
    I got it
  • John AbshierJohn Abshier Posts: 1,116
    edited 2013-03-27 08:36
    You are pushing the limits of precision for a single precision float.

    John Abshier
Sign In or Register to comment.