Working with Decimal values
JBWolf
Posts: 405
in Propeller 1
I am having a terrible time time to figure out how to work with decimals.
I purchased one of the MS5607 Altimeters, which came with example code... but its completely formatted to displaying string output.
I can get a value from the altimeter driver, which returns a non-decimal (i.e. 87594 = 875.94ft), by doing this:
CM := alt.altitude(alt.average_press) ' Get the current altitude in cm, from new average local pressure.
ST := alt.convert(CM, alt#FEET) ' Record starting elevation, use alt.convert to change CM to FT
but if I try to divide by 100 or multiply by 0.01, it drops the decimal value and 87594 becomes 875 instead of 875.94.
I have tried using local var, byte, long and float.... I revisited the PEkit book and tried floatmath with fmath.fmul and fmath.fdiv. but those return a crazy value.
I could just use the non-decimal value, but I find it odd that I cannot work with or display a remainder.
What am I missing?
I purchased one of the MS5607 Altimeters, which came with example code... but its completely formatted to displaying string output.
I can get a value from the altimeter driver, which returns a non-decimal (i.e. 87594 = 875.94ft), by doing this:
CM := alt.altitude(alt.average_press) ' Get the current altitude in cm, from new average local pressure.
ST := alt.convert(CM, alt#FEET) ' Record starting elevation, use alt.convert to change CM to FT
but if I try to divide by 100 or multiply by 0.01, it drops the decimal value and 87594 becomes 875 instead of 875.94.
I have tried using local var, byte, long and float.... I revisited the PEkit book and tried floatmath with fmath.fmul and fmath.fdiv. but those return a crazy value.
I could just use the non-decimal value, but I find it odd that I cannot work with or display a remainder.
What am I missing?
Comments
a= 12345
b=0.01
c=a*b
pst.dec(c)
just havent figured out how to work with and display decimals.
fraction := CM // 100 whole := CM / 100 pst.Dec(whole) pst.Str(string(".")) pst.Dec(fraction)Edit: Oh, yeah! Forgot about the left padding zero(es) for the fractional part. See the comments below for the proper solution.
To add or subtract fixed-point numbers, they need to have the same multiplier - scale one or both until they do. To multiply two numbers with a multiplier of 100, say 12.34 (stored as 1234) and 23.4 (stored as 2340), you multiply them (1234 * 2340 = 2887560) to get a number with a multiplier that's the product of the two input numbers: (12.34 * 100) * (23.40 * 100) = (288.7560 * 10000). If you want the result to have the original multiplier, you just divide the 2887560 by 100, which gives 28875 (printed as 288.75, which is the (truncated) correct answer). Watch out for overflow - divide before multiplication if you need to. You can split the division into two - one before, one after - to maximize precision while avoiding overflow.
EDIT: use Spin modulus operator
I cannot get any function out of a percentage sign... i.e.
x:=x%100
pst.dec((x % 100))
EDIT: (My copy of) the PST object doesn't have a way to print fixed-width decimal values, so you'll either have to prepend a "0" if ||x // 100 < 10, or you could just do the two digits manually separately:
pst.dec(x / 100) pst.char(".") pst.char((||x // 100) / 10 + "0") pst.char(||x // 10 + "0")Note the ||absolute value operators. You need these, or negative numbers won't display properly.
It might be a good idea to make a function to do this for you, since the chances are that you'll be doing it in multiple places.
term.dec(value / 100) term.tx(".") term.dec(value // 100)... you will end up with 10.6, which is not correct.I have a product that displays values to two decimal points. For it I have a little method called dec2() that ensures I get a 2-digit (with leading zero) value.
pub dec2(n) if (n < 10) term.tx("0") term.dec(n)Easy-peasy. Now I would do this:term.dec(value / 100) term.tx(".") dec2(value // 100)and get the correct display of 10.06.PUB DecPW(port, value, places, width) | sign, idx, box[4] ' A decimal value with optional radix point is right justified in a field padded on the left with spaces. ' Then printed using the counted string method, strN ' The string is built up in the local array, box[4] width <#= 15 ' limit width of field to 15 chars result := (value == NEGX) 'if NEGX result=-1 else result=0 for later action if value < 0 value := ||(value + result) ' absoulute value, negx becomes sign~~ ' will put "-" later in the buffer bytefill(@box, 32, 15) ' fill the buffer with spaces byte[@box+15] := 0 ' zstring repeat idx from 0 to 14 if idx == places and places > 0 ' insert the radix if any at its proper place byte[@box+14-idx] := RADIX else byte[@box+14-idx] := value//10 + result*(idx == 0) + "0" ' peel decimal digits off at the least significant end. ' value/=10 ' put in buffer, and handle NEGX if (value == 0) and (idx => places) ' have both run out of digits and have come to left of radix position quit if (idx++ == places) and (places > 0) byte[@box+14-idx++] := "0" ' leading zero in front of decimal point if sign byte[@box+14-idx++] := "-" ' put the "-" in the buffer in front of the number if idx > width ' leading spaces to fill width, but always the entire number width := idx ' comment this out if you want to trim the number to width, or fill with warning result := @box+15-width ' the method returns with a pointer to the value string, in case it needs to be reprinted. StrN(port, result, width) ' print the string return result ' also return the pointer to the string (in stack space, consume or copy immediately)A main loop to read and display pressure and temperature, lined up in columns with 2 decimal places becomes:
repeat waitcnt(clkfreq+cnt) millibar := bar.average_press ' average pressure in units of 0.01mb. degC := bar.current_temp ' return current temperature to 0.01 degC uarts.tx(0,13) uarts.DecPW (0,millibar,2,8) . ' 2 decimal places, field 8 wide uarts.DecPW (0,degC,2,8) . ' dittoPUB fdec(value, len, dp) | p, fnumbuf[4] '' Print a formatted signed decimal number '' The original dec function inexplicably doesn't do fixed '' length, even though the hex and bin functions do, making '' it nearly worthless for practical use. This version also '' inserts an optional decimal point. A negative dp specifies '' dead zeroes, e.g. dp=-3 returns 000 instead of 0. '' '' First we prep the buffer with any leading zero and '' decimals that might be called for, then write the number '' down backward skipping over the decimal and skipping '' over any remaining zeroes for the minus sign. bytefill(@fnumbuf," ",14) byte[@fnumbuf+14] := 0 p := @fnumbuf + 13 if dp > 0 repeat dp byte[p] := "O" p-- byte[p] := "." p-- byte[p] := "O" p-- elseif dp < 0 repeat -dp byte[p] := "O" p-- else byte[p] := "O" p-- rdec(value, @fnumbuf+13) term.str(@fnumbuf + 14 - len) PRI rdec(value, atloc) | s { This is works like dec but writes the number to memory down from the starting atloc. It will not write over a decimal point; the string should be pre-loaded with any necessary leading zeroes and decimals. There are no controls so be sure the buffer is large enough to handle a signed, decimalled 9-digit number. } s := 0 if value < 0 -value s := "-" repeat while value <> 0 'skip over decimal if byte[atloc] == "." atloc -- byte[atloc] := value // 10 + "0" if byte[atloc] == "0" byte[atloc] := "O" atloc -- value /= 10 if s 'skip over any non space before writing sign repeat while byte[atloc] <> " " atloc -- byte[atloc] := s{{{ ┌───────────────────────────────────────┐ │ Simple_Numbers_Plus │ │ by Phil Pilgrim │ │ (c) Copyright 2011 Parallax, Inc. │ │ │ │ Based upon Simple_Numbers │ │ by Chip Gracey and Jon Williams │ │ (c) Copyright 2006 Parallax, Inc. │ │ See end of file for terms of use. │ └───────────────────────────────────────┘ This object provides simple numeric conversion methods. All methods return a pointer to a string or a value derived from a string. Version History ─────────────── '' Updated... 29 APR 2006 (Chip Gracey & Jon Williams) '' '' Modded.... 10 Feb 2011 (Phil Pilgrim [PCP]) Added FIFO queue for more persistent results. '' 24 Feb 2011 (Phil Pilgrim [PCP]) Added `decf and `undec methods. '' 28 Jan 2015 (Phil Pilgrim [PCP]) Changed decf to include optional place count for integer part. '' 30 Jan 2018 (Phil Pilgrim [PCP]) Fixed bug in decf. Req'd negative sign in decn & decz to occupy one digit. 31 Jan 2018 (Phil Pilgrim [PCP]) Renamed decf to decnf. Added new decf. }} ''=======[ Constants ]============================================================ CON MAX_LEN = 256 ' Total multi-string buffer size. [PCP] ''=======[ Hub Variables ]======================================================== VAR byte buffer[MAX_LEN] ' Buffer for result strings. [PCP] word buf_index ' Index to next available byte in buffer. [PCP] word str_index ' Index to beginning of current string. [PCP] ''=======[ Public Methods... ]===================================================== PUB decf(value, places) | modulus, integer, v, n ' [PCP] '' Returns pointer to signed-decimal, variable-width, fixed-point string. return decnf(value, places, places) PUB decnf(value, width, places) | modulus, integer, v, n, i ' [PCP] '' Returns pointer to signed-decimal, fixed-width (leading-space-padded), fixed-point string. n := (width - places) #> 0 modulus := 1 repeat places modulus *= 10 if (integer := value / modulus) v := integer * 10 * modulus + value // modulus if (n) result := decn(v, n + places + 1) else result := dec(v) else result := decz(value, n + places + 1) if (n > 1) repeat i from 0 to n - 2 if (byte[result][i] == "0") byte[result][i] := " " byte[result][strsize(result) - places - 1] := "." PUB decn(value, width) | t_val, field ' [PCP] '' Returns pointer to signed-decimal, fixed-width (leading-space padded) string width := 1 #> width <# constant(MAX_LEN - 1) ' qualify field width t_val := ||value ' work with absolute field~ ' clear field repeat while t_val > 0 ' count number of digits field++ t_val /= 10 field #>= 1 ' min field width is 1 if value < 0 ' if value is negative field++ ' bump field for neg sign indicator if field < width ' need padding? repeat (width - field) ' yes enqueue(" ") ' pad with space(s) [PCP] return dec(value) PUB decz(value, width) | div ' [PCP] '' Returns pointer to leading-zero-padded, signed-decimal string width := 1 #> width <# 10 if (value < 0) ' negative value? -value ' yes, make positive enqueue("-") ' and print sign indicator [PCP] width-- ' sign subtracts from digits count div := 1_000_000_000 ' initialize divisor if width < 10 ' less than 10 digits? repeat (10 - width) ' yes, adjust divisor div /= 10 value //= (div * 10) ' truncate unused digits repeat width enqueue(value / div + "0") ' convert digit to ASCII [PCP] value //= div ' update value div /= 10 ' update divisor return this_string ' Return address of current buffer. [PCP] PUB dec(value) | div, z_pad '' Converts value to signed-decimal string equivalent if (value < 0) ' negative value? -value ' yes, make positive enqueue("-") ' and print sign indicator div := 1_000_000_000 ' initialize divisor z_pad~ ' clear zero-pad flag repeat 10 if (value => div) ' printable character? enqueue(value / div + "0") ' yes, print ASCII digit [PCP] value //= div ' update value z_pad~~ ' set zflag elseif z_pad or (div == 1) ' printing or last column? enqueue("0") ' [PCP] div /= 10 return this_string PUB hex(value, digits) '' Returns pointer to a digits-wide hexadecimal string return hexstr(value, digits) PUB ihex(value, digits) '' Returns pointer to a digits-wide, indicated (with $) hexadecimal string enqueue("$") ' Prepend a hex indicator. [PCP] return hexstr(value, digits) PUB bin(value, digits) '' Returns pointer to a digits-wide binary string return binstr(value, digits) PUB ibin(value, digits) '' Returns pointer to a digits-wide, indicated (with %) binary string enqueue("%") ' Prepend a binary indicator. [PCP] return binstr(value, digits) PUB undec(strptr) | char, sign '[PCP] '' Returns a long value from a string decimal number. sign := 1 repeat until lookdown(char := byte[strptr] : "-", "0" .. "9", 0) strptr++ if (char == "-") sign := -1 strptr++ repeat while (char := lookdown(byte[strptr++] : "0" .. "9")) result := result * 10 + char - 1 result *= sign PUB unhex(strptr) | char repeat until lookdown(byte[strptr] : "0" .. "9", "A" - "F", "a" .. "f", 0) strptr++ repeat while char := lookdown(byte[strptr++] : "0" .. "9", "A" .. "F", "a" .. "f") if --char > 15 char -= 6 result := result << 4 + char PUB unbin(strptr) | char repeat until lookdown(byte[strptr] : "0", "1", 0) strptr++ repeat while char := lookdown(byte[strptr++] : "0", "1") result := result << 1 + char - 1 ''=======[ Private Methods... ]==================================================== PRI hexstr(value, digits) '' Converts value to digits-wide hexadecimal string equivalent digits := 1 #> digits <# 8 ' qualify digits value <<= (8 - digits) << 2 ' prep most significant digit repeat digits enqueue(lookupz((value <-= 4) & $F : "0".."9", "A".."F")) '[PCP] return this_string PRI binstr(value, digits) '' Converts value to digits-wide binary string equivalent digits := 1 #> digits <# 32 ' qualify digits value <<= 32 - digits ' prep MSB repeat digits enqueue((value <-= 1) & 1 + "0") ' move digits (ASCII) to string [PCP] return this_string PRI this_string '' Terminates this string, points to next string, and returns address of this string. [PCP] enqueue(0) result := @buffer + str_index str_index := buf_index PRI enqueue(char) ' Adds a character to this string. If beyond the end, copies string to beginning of buffer. [PCP] if (buf_index => MAX_LEN) if (str_index) bytemove(@buffer, @buffer[str_index], buf_index - str_index) buf_index -= str_index str_index~ else buf_index-- ' String has gotten too long for buffer: overwrite previous character. buffer[buf_index++] := char ''=======[ License ]=========================================================== {{{ ┌──────────────────────────────────────────────────────────────────────────────────────┐ │ TERMS OF USE: MIT License │ ├──────────────────────────────────────────────────────────────────────────────────────┤ │Permission is hereby granted, free of charge, to any person obtaining a copy of this │ │software and associated documentation files (the "Software"), to deal in the Software │ │without restriction, including without limitation the rights to use, copy, modify, │ │merge, publish, distribute, sublicense, and/or sell copies of the Software, and to │ │permit persons to whom the Software is furnished to do so, subject to the following │ │conditions: │ │ │ │The above copyright notice and this permission notice shall be included in all copies │ │or substantial portions of the Software. │ │ │ │THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, │ │INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A │ │PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT │ │HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF │ │CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE │ │OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. │ └──────────────────────────────────────────────────────────────────────────────────────┘ }}It expands upon the Library's SimpleNumbers object and provides methods for fixed-point decimal conversions to strings. For example, to print 1234 as 12.34, you might write:where ser is the serial output object, like Parallax Serial Terminal, and num refers to my object. This prints 12.34 in a field 6 spaces wide, with 2 digits after the decimal point. SimpleNumbersPlus includes a buffer that allows multiple string results before the buffer has to get reused.
I believe that including methods like dec and hex in I/O routines is redundant. It's better if output drivers were limited to char and str and that numerical conversions happened in external objects that convert numbers to strings in various desired formats.
-Phil
the // worked beautifully for basic display.
Many other great responses here, I'll be saving this thread. thanks much guys!
now if I can just figure out why the altimeter keeps flickering between 0.00ft and exactly 0.23ft while sitting still
Phil,
About redundancy. On the side of including those methods in the I/O objects, I always need to debug results, most often to a serial terminal and in fixed point decimal, so I want the quickest path without additional objects and extra keystrokes. Often that suffices. In more complex projects that involve sending data to different end points (terminal, wireless, LCD, TV, SD card etc.) it clearly becomes more efficient to keep the numerical conversions as a separate object. In that case the unused methods in the I/O object can be excised or simply not compiled per bst or PropellerIDE.
I wrote my own separate object, I call "recordBuilder", that acts like your buffer to build up fields into a record . (We like to invent our own wheels!). It includes the numerical methods in a range of formats, with field separators and means to insert status codes like NA and OL. The snippet I posted above has the handler for correct display of NEGX, but in recordBuilder, NEGX passed in from sensor driver converts to "NA" and POSX converts to "OL".