srting/array parsing/processing question

sharpie · 2006-07-24 08:38

Ok..... so now onto my next step and I'm hoping this will be an easy one for someone, but has been·a real pain for me...
I want to accept data from a serial line, pluck out certain parts of it and use those for variables in other subs..· Imagine something like an nmea sentence from a gps..· And I want to identify a certain sentence by the first few characters, then take the rest apart delimited by commas in that case and use altitude for this example in a calculation or something..·
Using $GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47 as an example...· Altitude is listed as the tenth value.. "545.4"
Where:
···· GGA········· Global Positioning System Fix Data
···· 123519······ Fix taken at 12:35:19 UTC
···· 4807.038,N·· Latitude 48 deg 07.038' N
···· 01131.000,E· Longitude 11 deg 31.000' E
···· 1··········· Fix quality: 0 = invalid
······························ 1 = GPS fix (SPS)
······························ 2 = DGPS fix
······························ 3 = PPS fix
······ 4 = Real Time Kinematic
······ 5 = Float RTK
······························ 6 = estimated (dead reckoning) (2.3 feature)
······ 7 = Manual input mode
······ 8 = Simulation mode
···· 08·········· Number of satellites being tracked
···· 0.9········· Horizontal dilution of position
···· 545.4,M····· Altitude, Meters, above mean sea level
···· 46.9,M······ Height of geoid (mean sea level) above WGS84
····················· ellipsoid
···· (empty field) time in seconds since last DGPS update
···· (empty field) DGPS station ID number
···· *47········· the checksum data, always begins with *

I would like to be able to accept the sentence as a whole string using the 'FullDuplexSerial' object, but it looks like I have to take it in byte by byte...· So, if I construct an array of say "SERIN[noparse][[/noparse]65]" and have each character in that array (leaving out the fact that the sentence will probably be different lengths at different times)..· And then I pluck out my "5" and "4" and "5", ".", "4", etc....· I've tried putting that into another array, and numerous failed attempts at consolidating it into a string, number, or whatever..·

Does anyone have a suggestion for making that a whole value I can use?· I've searched as much as my brain can take... The RFID posting was handy, but hasn't helped me solve it yet..
Thanks!

Mike Green · 2006-07-24 14:07

Here's an example for an integer with an optional sign:

PRI getInteger : value | input, negative
  value := 0
  negative := false
  input := ser.rx
  negative := input == "-"
  if input == "+" or input == "-"
    input := ser.rx
  repeat while input => "0" and input =< "9"
    value := value * 10 + input - "0"
    input := ser.rx
  if negative
    value := - value

Basically, this reads the serial line, allows a leading + or -, then a sequence of digits ended by some non-digit character which is ignored (like a comma), and returns the binary value. You can do the equivalent thing for floating point values using FloatMath for the arithmetic.

Mike Green · 2006-07-24 14:46

Here's another example for floating point:

OBJ
  flt : "FloatMath"

PRI getFloat : value | input, negative, decimals
  value := 0
  negative := false
  decimals := 0
  input := ser.rx
  negative := input == "-"
  if input == "+" or input == "-"
    input := ser.rx
  repeat while input => "0" and input =< "0"
    value := value * 10 + input - "0"
    input := ser.rx
  if input == "."
    input := ser.rx
    repeat while input => "0" and input =< "9"
      if decimals < 9
        value := value * 10 + input - "0"
        decimals := decimals + 1
      input := ser.rx
  if negative
    value := - value
  value := flt.FFloat(value)
  if decimals > 0
    decimals := lookup(decimals:1E-1,1E-2,1E-3,1E-4,1E-5,1E-6,1E-7,1E-8,1E-9)
    value := flt.FMul(value,decimals)

This basically accumulates an integer value, but keeps a count of the number of decimal places. It ignores decimal places beyond 9, partly because of simplicity, and partly because, in the usual case, they're not likely to be significant. If this is an issue, the routine can be changed to handle a wider range, but it is limited to 9 significant digits. The floating point format won't keep that many significant digits anyway.

Post Edited (Mike Green) : 7/24/2006 2:51:17 PM GMT

sharpie · 2006-07-24 16:52

Wow. Thanks Mike... I'm going to have to take a minute to digest all of that.

John Abshier · 2006-07-25 02:45

A couple of "got'chs" on this NMEA sentence:· The check sum is a hex number so input - "0" will not always work. The number of decimal points in the location fields in not always 3.· My Garmin Etrex Vista outputs 4 decimal points.·

sharpie · 2006-07-25 03:13

Thanks John.. Although I will probably be looking at nmea sentences later on, I only used it as an example for this question.. The number of decimals doesn't have to be a factor either since I have control of the sentence on the other end as well.. I am way more interested in being able to combine the received bytes so I can use them for values in either calculations or parameters for stuff like servo position... Hopefully what I learn from here can be applied to either.. I am just about to sit down with Mike's example and see if I can get a grasp on that..

sharpie · 2006-07-25 04:50

So, I am still confused...
If I receive a sequence of data into an array...· We'll say "byte input[noparse][[/noparse]64]".. And we'll assume my sentence is the same length all the time, and all the segments are the same number of digits every time....· Using the sentence below, say I want to extract the "545" in bold·from it and use that for a value somewhere else in the program.

$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,[b][i]545.4[/i][/b],M,46.9,M,,*47 
                                              |
                                              |
                                              |
                                          5 is the 47th character, so 4 is the 48th, and 5 is the 49th

So, I know the positions of the characters I want to extract from the array...· Using code similar to the following, each character of the sentence is put into the array of "newArray" and "ndx" being the indexer...

    input := ser.rx
    newArray[noparse][[/noparse]ndx++] := input

·So, then now I know that in "newArray[noparse][[/noparse]47]" I have the number "5", and "newArray[noparse][[/noparse]48]" I have "4", and a "5" went into "newArray[noparse][[/noparse]49]"..· Great, now how can I consolidate these into a whole single number that I can feed maybe as another variable to a parameter for something like "svo.set(0, 4, IWANTMYNUMBERHERE)"?? Assuming that number would be a value usable for that parameter, and since these are examples and I'm not actually trying to dump my altitude from a gps into a servo position...

Mike, I am a little confused about your example and how it would relate to a sentence that doesn't contain 100% digits..· For example, the nmea sentence has "N" for north, and "E" for east... etc.....

Thanks for the help!!!

Mike Green · 2006-07-25 05:24

The routines I posted will also work with characters from an array. Just substitute "newArray[noparse][[/noparse]ndx++]" for "ser.rx" (after you've read in all the characters and reinitialized ndx). My examples use any non-digit character as the delimiter at the end of number (and they throw the character away). If you want to look at the delimiter, just declare a byte VAR like "delimiter" and put a "delimiter := input" before the "if negative" at the end of each routine.

For the non-numeric input stuff ... you can do something like ...

  case ser.rx
    "N": ' whatever you want for "N"
    "S":
    "E":
    "W":
    other: ' some kind of error message for invalid compass direction
  if ser.rx <> ","
    ' some kind of error message for missing comma after direction

sharpie · 2006-07-25 06:38

So, is what you are saying then is to build my array from the serial data..· Then pass that array through your routine...· And that will pump out a reconstituted value?· Something like the following.· (I am sure I made all kinds of mistakes)

    rcv := ser.rx
    rcx[noparse][[/noparse]ndx++] := rcv
    if rcv == "X"       '<-- we'll say that my sentence ends with "X"
      getFloat(rcx)

So I have the "rcx" array containing the whole sentence...· and it sends that through your getFloat and returns my value?· I apologize for being so dense, but I think I am still missing something..· How do I specify which part of the sentence I need?· As I want to pluck out the "5", "4" and "5" from the sentence and get a variable back that equals a numeric 545...·
Previously, I was thinking of something like

    rcv := ser.rx
    rcx[noparse][[/noparse]ndx++] := rcv
    if rcv == "X"
      temp := rcx[noparse][[/noparse]47] * 100
      temp := temp + (rcx[noparse][[/noparse]48] * 10)
      temp := temp + rcx[noparse][[/noparse]49]

or even

temp := rcx[noparse][[/noparse]47] & rcx[noparse][[/noparse]48] & rcx[noparse][[/noparse]49]

I'm a bit new to the prop and I'm still struggling with variables..· and some of the syntax..
·

sharpie · 2006-07-25 07:35

I was just reading page 289 of the manual, and it brushes across what I think part of my problem is...· I'm just shy of being able to apply this to a practical bit of code..·
To simplify my question... If I have an array with a length of say, 4 characters...· And I'd like to pass that array as a value to something like svo.set(0, 4, myVariable)

Or would something like this work?

      temp.byte[noparse][[/noparse]1] := rcx[noparse][[/noparse]8]
      temp.byte[noparse][[/noparse]2] := rcx[noparse][[/noparse]9]
      temp.byte[noparse][[/noparse]3] := rcx[noparse][[/noparse]10]
      temp.byte[noparse][[/noparse]4] := rcx[noparse][[/noparse]11]

Kevin Wood · 2006-07-25 10:23

I don't know too much about Spin, but couldn't you just create an NMEA Sentence class?

I'm thinking of an object similar to a C struct or C++ class. Then you could just parse the string using the first character, commas, and last character as the delimiters.

This should allow you to access your values as MySentence.Altitude, etc.

It just seems clunky to store each individual character as an array element that deconstructs your logical units of information.

sharpie · 2006-07-25 10:28

Kevin, I agree.. It would be great if we could do that.... But if you can, I haven't figured out how yet.. Even if it were mid$ or a split....

M. K. Borri · 2006-07-25 13:35

Careful, GPS NMEA sentences are variable length so you can't assume something will be in this or that position in the array.

I'm having the same problem -- will post results.

Mike Green · 2006-07-25 15:21

sharpie,
I think it will help for you to understand the routines I've posted. Take the simpler one for integer input. Sit down with a piece of lined paper and put each variable name on a line (value, input, negative) and make up a proposed serial input sequence and write those values on a line labelled "ser.rx". Then go through the routine statement by statement and "execute" the SPIN statements yourself, not skipping or assuming anything, changing the values on the paper as you go along. If this seems excessive, I do it myself when I get stuck. Sometimes you make too many assumptions as you read programs and miss something simple.

I think you are getting hung up on this array thing. Most of the time, when you need to process an input stream, you can "do it on the fly" without saving the characters you've read into an array. The most common exception is when the input speed is too high for the program to do anything but save the characters for processing later. The FullDuplexSerial routines have a buffer already and the Propellor is fast enough that you are unlikely to have problems processing such a simple input stream "on the fly".

Phil Pilgrim (PhiPi) · 2006-07-25 20:21

Sharpie,

Here's a subroutine (match_inp) that will help you parse your strings. It's designed to look for incoming strings on the fly. The demo shows how it can be used with the $GPGGA sentence.

[b]CON[/b]

  [b]_clkmode[/b]      = [b]xtal[/b]1 + [b]pll[/b]16x
  [b]_xinfreq[/b]      = 5_000_000
  maxstr        = 256

[b]VAR[/b]

  [b]byte[/b]  return_string[noparse][[/noparse]maxstr + 1]

[b]OBJ[/b]

  sio : "FullDuplexSerial"
  pr :  "tv_text"

[b]PUB[/b] start

  sio.start(31, 30, 0, 4800)
  pr.start(12)
  [b]repeat[/b]
    match_inp([b]string[/b]("$GPGGA"), 1)                      'Look for "$GPGGA".
    match_inp([b]string[/b](","), 9)                           'Look for nine commas.
    pr.[b]str[/b](match_inp([b]string[/b](","), 0))                   'Get everything from there until the next comma, and display it.
    pr.out(13)                                          'Display a carriage return.
     
[b]PUB[/b] match_inp(string_to_match, times) : ret_val | len, rstart, rend

'' Match incoming characters against a string, given by the string address: string_to_match.
'' If times > 0, then match the string that many times, and return the matched string.
'' If times == 0, then match the string once, and return everything up until the matched string.
'' The return value is a pointer into the array return_string. Therefore it's only valid until
'' the next time match_inp is called.
'' The array return_string must be at least as long as string_to_match, if times > 0;
'' it must be at least as long as string_to_match PLUS the length of everything not matched until that point, if times == 0.
'' If these conditions aren't met, or if string_to_match is a null string, match_inp will return false.

  len := [b]strsize[/b](string_to_match)
  [b]if[/b] len == 0 [b]or[/b] len > maxstr
    [b]return[/b] [b]false[/b]
  rstart := rend := 0
  [b]repeat[/b]
    return_string[noparse][[/noparse]rend++] := sio.rx
    return_string[noparse][[/noparse]rend] := 0
    [b]if[/b] rend - rstart > len
      rstart++
    [b]if[/b] rend - rstart == len
      [b]if[/b] [b]strcomp[/b](string_to_match, @return_string[noparse][[/noparse]rstart])
        [b]if[/b] times == 1
          [b]return[/b] @return_string[noparse][[/noparse]rstart]
        [b]elseif[/b] times == 0
          return_string[noparse][[/noparse]rstart] := 0
          [b]return[/b] @return_string
        [b]else[/b]
          times--
          rstart := rend := 0     
    [b]if[/b] rend == maxstr
      [b]if[/b] times [b]and[/b] rstart
        [b]bytemove[/b](@return_string, @return_string[noparse][[/noparse]rstart], rend - rstart + 1)
        rend -= rstart
        rstart := 0
      [b]else[/b]
        [b]return[/b] [b]false[/b]

-Phil

sharpie · 2006-07-25 23:25

OMG.. Thanks Phil, that is EXACTLY what I was looking for!!!!!!!!!! It works GREAT! It works for my application as well as the nmea stuff.. I'm definately going to study up on this some more and get a better understanding.. It's always so helpful when you have a working example to dissect.

Thanks so much to everyone... Mike, thanks for your input as well.. I am sure I will find a use for your suggestions when I get a better grasp on spin.

Paul_H · 2007-04-22 16:53

Phil,

This is an excellent string extraction tool . I have a question about its repeated operation though. When I try to use it more than once for different variable, my values are set to the same value! It's probable my code, but if you have a moment here it is for review. I used your above code in the calls.

PUB start

  sio.start(8, 9, 1, 4800)
  pr.start(12)

  getLat                               ' calls the match_inp routine three separate times and data gathers over 3 seconds.     
  getLon
  getAlt

  ShowData
  
PUB GetLat
  match_inp(string("$GPRMC"), 1)       'Look for NMEA string 
  match_inp(string(","), 3)            'Look for # of commas
  Lat := match_inp(string(","), 0)     'latitude
  pr.out(13)                                   
  pr.str(Lat)                                   ' show it
    
PUB GetLon
  match_inp(string("$GPRMC"), 1)       'Look for NMEA string 
  match_inp(string(","), 5)            'Look for # of commas
  Lon := match_inp(string(","), 0)     'longitude
  pr.out(13)                           
  pr.str(Lon)                                    ' show it 

PUB GetAlt
  match_inp(string("$GPGGA"), 1)       'Look for different NMEA string 
  match_inp(string(","), 9)            'Look for # of commas
  Alt := match_inp(string(","), 0)     'altitude
  pr.out(13)                           
  pr.str(Alt)                                    ' show it
  
PUB ShowData  
  pr.out(13)                           'Display a carriage return.

  pr.str(string("Lat: "))
  pr.str(Lat)
  pr.out(13)                           

  pr.str(string("Lon: "))
  pr.str(Lon)      
  pr.out(13)                           

  pr.str(string("Alt: "))
  pr.str(Alt)      
  pr.out(13)

The screen displays:

0001.4322
00013.1433
104
Lat: 104
Lon: 104
Alt: 104

The top three lines are correct, but the assigned variables inthe next three lines are changed since their first display in the capture routing. And ideas?

Thanks,
Paul

Post Edited (Paul H) : 4/22/2007 4:57:54 PM GMT

Phil Pilgrim (PhiPi) · 2007-04-22 18:43

Paul,

The return value from match_inp is a pointer into the global array return_string. This array gets reused every time match_inp is called, so pointers from previous calls no longer point to valid data. To save your results for later display, you need to copy the pointed-to strings into their own separate arrays.

-Phil

Mike Green · 2007-04-22 18:55

Paul,
Do have a look at the Extended Full Duplex Serial object from the Propeller Object Exchange. It's a wrapper object for the FullDuplexSerial driver that does some of this type of scanning and conversion for you. It may save you some work. This object also uses a private buffer whose address is returned from some of the calls and will be reused with the next call.

Paul_H · 2007-04-22 21:45

Thanks guys,

So I now understand the match_inp routine hands back a pointer (not a value) that I have been copying into my variables. The display shows the last value, because they all point to the same location. It's starting to make sense...

If I use a ByteFill (or ByteMove) to copy the data, doesn't that necessitate that I know the length of the data acquired by the match_inp routine? Is there a more direct method that avoids this, such as:

  temp := match_inp(string(","), 0)     'latitude
  Bytemove(@Lat, temp, strsize(temp))   ' not a working line of course....)

The data is not going to be zero terminated either.

I unfortunately am having difficulties dealing in the "pointer-to-variable" world and am more used to the "variable just is" in higher levels of programming. I am rereading the Extended Full Duplex serial to see if I can repurpose parts, specifically i'm looking at RxStr and RxDec. I guess I need to know the data type I am receiving (which I do) then apply the proper set of steps to assign to a global variable.

Comments?
Paul

Mike Green · 2007-04-22 22:13

In your example, you can simply write "bytemove(@Lat,temp,strsize(temp)+1)" to copy the zero byte and the string data. That makes the statements the equivalent of 'Lat := match_inp(string(","),0)' for a string transfer.

Phil Pilgrim (PhiPi) · 2007-04-23 00:04

Paul,

Another thing you might consider: In your code, latitude and longitude come from two separate NMEA strings. It would be more accurate if both came from the same string, since you might change position between two of them. Once you've obtained the latitude, you can get the longitude just by scanning past two more commas, instead of scanning for the $GPRMC header again.

-Phil

Paul_H · 2007-04-23 00:21

Phil,

I totally agree. Thats actually what I had started doing until I ran into the re-use of the match_inp array and kept getting the same values. I simplified it for my postings as this was the same issue issue either way. Though I hadnt started out with this goal, I thought I might put together a general purpose NMEA reader for the code library as I work this out specifically for my projects. It's funny - the more I understand the process, the less I see the need for a general purpose SPIN code, but then I remember that it still isnt working for me and there must be others I can help out down the road!

Mike
I'll try your ideas tonight - as soon as i understand where the 0 came from... And of course post the results

Thanks,
Paul

Paul_H · 2007-04-23 15:45

Well I'm still stymied :-(

I've changed the code to read


PUB GetGPRMC
  match_inp(string("$GPRMC"), 1)                    'Locate NMEA string name.

  match_inp(string(","), 3)            ' count commas
  temp := match_inp(string(","), 0)    ' get all serial values until next commas (=latitude)
  pr.dec(strsize(@temp)+1)             ' show what was captured 
  bytemove(@Lat,@temp,strsize(temp)+1) ' move to the Lat variable location

  match_inp(string(","), 1)            ' count commas
  temp := match_inp(string(","), 0)    ' get all serial values until next commas (=longitude)
  pr.dec(strsize(@temp)+1)             ' show what was captured 
  bytemove(@Lon,@temp,strsize(temp)+1) ' move to the Lon variable location

Which should bytemove the first string to variable Lat, and the second to variable Lon (I see that the strsize(temp)+1 yields the next memory location past the captured values which is a zero, sh what I copy has a zero termination now).

But! - I am still getting the same values for both of my variables.

Phil Pilgrim (PhiPi) · 2007-04-23 15:51

temp is already an address. You don't need the @ in front of it when you refer to it.

-Phil

Paul_H · 2007-04-23 15:54

Phil,

I remarked that I was trying to use a single data string to capture all variables at once but there is a hitch using match_inp -- I cannot capture consecutive data values with your code because the delimiter (a comma) is used to both locate the data, then terminate its capture.

By process definition, the termination of one datum is the location of the next. So if I try to capture the 3rd and 4th delimited values, I get the 3rd comma, the 3rd data, a terminating comma (the 4th!), then search for the next comma (want the 4th, but it has already past) so I get the next comma seen (the 5th) and we capture the 5th data value.

If I place the bytemoves (once they work) into a match_inp derivitive, it can work. But you code is so much more elegant than mine!
Of course its all moot until i fix my overwrite issue! lol
Paul

Paul_H · 2007-04-23 21:41

Phil and Mike,

I think I get it (or least it's working!)

It appears that I have been copying the data to the new memory locations, but was just not displaying it properly! I needed to add the @ Symbol Address to my output lines to see the actual data located at these addresses.

PUB start

  sio.start(8, 9, 1, 4800)
  pr.start(12)

  match_inp(string("$GPRMC"), 1)       ' Locate NMEA string name.

  match_inp(string(","), 3)            ' count commas
  tmp := match_inp(string(","), 0)     ' get all serial values until next commas (=latitude)
  bytemove(@Lat,tmp,strsize(tmp)+1)    ' move to the Lat variable location
 

  match_inp(string(","), 1)            ' count commas
  tmp := match_inp(string(","), 0)     ' get all serial values until next commas (=latitude)
  bytemove(@Lon,tmp,strsize(tmp)+1)    ' move to the Lat variable location

  pr.str(string("Lat: "))
    pr.str(@Lat)
  pr.out(13)                           
  pr.str(string("Lon: "))
    pr.str(@Lon)

Thank you both!

Paul

mre_rob · 2007-05-12 17:51

I need help bad... I am trying to get to the same goal as Paul... but trying to think in bytes is killing me.... Is there an easy way to accept the entire sentence, then split that up into separate strings?

It appears as tough this match_inp routine is keeping pointers between calls. Would it not be easier to come up with a routine that take in a delimiter to know when the sentence rx is complete, then parse it into it separate parts? I am way too new to Spin, but have tons of C++ and C# experience. All I need is some "pointers"...no pun intended.

Every time I try to get this right something else get whacky... right now the FixTime never shows up....

<code>
match_inp(string("$GPRMC"), 1)

match_inp(string(","), 1)

vga.str(string("Data: "))

tmp:=match_inp(string(","), 0)
bytemove(@fixTime,tmp,strsize(tmp)+1)
vga.str(string("FixTime: "))
vga.str(@fixTime)
vga.out(13)

tmp:=match_inp(string(","), 0)
bytemove(@isvalid,tmp,strsize(tmp)+1)
vga.str(string("Fix: "))
vga.str(@isvalid)
vga.out(13)

tmp:=match_inp(string(","), 0)
bytemove(@lat,tmp,strsize(tmp)+1)
vga.str(string("Lat: "))
vga.str(@lat)
vga.out(13)

tmp:=match_inp(string(","), 0)
bytemove(@ns,tmp,strsize(tmp)+1)
vga.str(string("N/S: "))
vga.str(@ns)
vga.out(13)

tmp:=match_inp(string(","), 0)
bytemove(@lon,tmp,strsize(tmp)+1)
vga.str(string("Lon: "))
vga.str(@lon)
vga.out(13)

tmp:=match_inp(string(","), 0)
bytemove(@ew,tmp,strsize(tmp)+1)
vga.str(string("E/W: "))
vga.str(@ew)
vga.out(13)
</code>