How to? Read an SD card a line at a time and parsing the data

SteveWoodrough

I’m in over my head again. Here is what I have done so far. I‘ve done a few exercises writing to an SD card, and reading it all back to the PST. Here is what I would like to do:

For my Magellan Bot, I want to prepare a list of way points on my lap top and write them to an SD card, in a simple txt file format. The format would be something simple like:

Waypoint, Lat, Lon
1, 123456, 123456
2, 234567, 765432

Simple, comma delimited data.

What I am having trouble with is how to read back the data from the SD a line at a time, and parse the data. I’ve looked at GPS_IO as parsing example but I’m not sure exactly how that works and how it “stiches” together a bunch of characters into a string. Once I have a string, I know how to convert that to a number, yada yada. Currently I am doing my waypoints as DAT fields in RAM but I would like to migrate to an SD card.

Any examples you all can share on this are appreciated.

Regards,
Steve

https://www.youtube.com/watch?v=M8u0VsNM0II


https://www.youtube.com/watch?v=-SFVx0i-tGE

SRLM

Here are the steps I take:
1: write a function to read a character at a time from the SD card, and store it into buffer, until a terminating character is found (such as '\r' or '\n').
2: write a function to parse a buffer, and split it on the first occurrence of a specified "split character (such as ',').
--- I'd make this function put the first half of the string in one buffer, and the second half in a second buffer.
3: Now, write the glue logic:
--- a) use function 1 to read in a line from the SD card
--- b) use function 2 to split the line for the first time. One buffer has the waypoint number, the second buffer has the remainder (lat, long, comma, and possibly '\r' and '\n').
--- c) use function 2 again to split the lat and long.
--- d) use a string to decimal function to convert all your character representations of numbers, to numbers.

Mike G

I do something similar. I'll tokenize a buffer and save pointers to the tokens in an array.

This is a somewhat simplified outline

• Write a block of bytes to a buffer
• Loop
• Read a byte and replace delimiter values with a zero terminator.
• Keep reading until an ASCII character is found then add that memory address to pointer array
• Repeat until the end of the buffer is reached

When the parsing is over you'll have an indexed table of values. I use this concept to parse HTTP headers.

MagIO2

Reading character by character and storing in a buffer is inefficient. You have a lot of function calls (getc or however it is called) which costs runtime.
Reading the stuff sector-wise is better, as it is only one call to get 512 bytes at once. Then you simply loop over the content to find parameters and lineend.

You could have a look at the ConfigReader in the object exchange:
http://obex.parallax.com/objects/598/

It only needs some little changes to make it read a file like you have it:
1. The ConfigReader currently only needs a space as separator (it was originally designed to read command lines from keyboard), so if you insist on your comma + space separation, you have to modify that a little bit. As Mike also suggests, each space is replaced by $00 which makes the part in front of the space a valid string which can be used with any function that works with strings.
2. The ConfigReader treats the first string found special in a way that it generates a hash value out of it. This is also due to the nature of being developed as a command line reader. It's easier/faster and needs less RAM for a command-table if you compare hashes instead of doing a strin-compare. As your first part is just a waypoint-number you could keep it as it is - or replace the hash-function in a way that the first parameter is also parsed like all following parameters.

The rest is magic. It simply parses through the file, line by line and in case a line is done you'll find all the content in a parameter array.
Parameters starting with % are parsed like a binary, parameters starting with $ are parsed like a hex-number, parameters starting with a number are parsed decimal, parameters starting with anything else are treated as strings and you'll find it's string address in the array.

If the buffer has been parsed and the line is not finished yet, the next bunch of data is loaded from SD card.

SteveWoodrough

Thank You. Let me take a look at the suggestions and let you know what develops.
Regards,
Steve

nomad

hi,
here is a little programm (PropC3)
it works:
-1) gps-datas as vars
-2) mount sdcard
-3) look for file "gps1.txt"
-4) if file on the sd-card delete it
-5) open a new file "gps1.txt
-6) write gps-datas into file as:
$nnn.nn:nnn.nn:....
-6) close file
-7 open it for read
-8) read the string to pst.terminal
-9) close file
-10) unmount sdcard

attachment: c3-quadV2-SD-02.spin
regards
nomad

SteveWoodrough

Thank you for your responses. This is elementary programming for most of you, so please excuse my ignorance.

Let me see if I have one concept down correctly. I asked how strings are "stiched" together. As I played with Jon McPhalen's jm_sd_text_read.spin it occured to me that a byte aligned array forms a string merly through the act of calling the array root. I've probably butchered that, here is an example.

VAR BYTE Newbie[10] is an array of 10 bytes (0-9)

assuming that I bytefill Newbie to put zeros in the array
and if then I make:
Newbie[0] = "S"
Newbie[1] = "t"
Newbie[2] = "e"
Newbie[3] = "v"
Newbie[4] = "e"

then to get the "stiched" together string I need to only refer to the variable Newbie.

Am I getting it or still missing something?
Thanks
Steve

JonnyMac

You need to end the array with a 0; then you can use a string function like this.

term.str(@newbie)

String functions need the address of (@) a z-string (zero-terminated string).

StefanL38

Hi Steve,

in the attachmant there is a OpenOffice-sheet (like an Excel-sheet) explaining how strings work
and with some basic democode how to parse a string for a certain character.
I made this maybe two years ago for another propeller-user

best regards
Stefan

softcon

Thank you Jon. You just answered a question I was struggling with in a program I've been working for my wife. Now I know why it wasn't working. I was trying to pass a string to a function, and it wasn't working. Because it wasn't 0 terminated. I think I knew this needed to be done, But being new to propellers like Steve, I didn't realize this was my problem. Now I can go fix my problem code.
That's why I like reading these forums, I learn all kinds of things, and get answers to questions I didn't even know I needed to ask.

SteveWoodrough

OK, I think I get it. Below is a modification to Jon's code. I replace comma's with number zero, and that creates automatic break points for strings, those strings can then be converted into numbers. I can then index the numbers into Longitude and Latitude waypoints as LONGS. Comments for lines I added have ***** in front. The key is knowing where in the buffer array the desired data string begins.

Thanks everyone for your help .

obj

  term   : "fullduplexserial"
  sdcard : "fsrw" 
  num    : "numbers"

var

  byte  sbuf[BUF_SIZE]                                          ' string buffer
  long  waypoint [10],Latitude[10], Longitude[10]                             '****an array of longs to store the lat an long numbers

dat

testfile                byte    "p8x32a.txt", 0

teststring              byte    "Jon McPhalen", CR, LF                  ' 1st line in file
                        byte    "Hollywood, CA", CR, LF                 ' 2nd line in file
                        byte    0


pub main | check , index                                   

  term.start(RX1, TX1, %0000, 115_200)
  pause(1)
  
  term.tx(CLS)
  term.str(string("SD Card Access Test", CR, "-- press key to start", CR, CR))
  term.rxflush
  check := term.rx  

  check := \sdcard.mount_explicit(SD_DO, SD_CLK, SD_DI, SD_CS)
  if (check == 0)
    term.str(string("-- sd card mounted", CR))
    
  else
    term.str(string("-- sd card failed to mount", CR))
    repeat
      waitcnt(0)                                                ' stop here   
    


  check := \sdcard.popen(string("data.txt"), "r")

  if (check == 0)

     term.str(string("-- test file opened", CR))   
     repeat index from 0 to 9
      
      bytefill(@sbuf, 0, BUF_SIZE)                                ' clear buffer
      readln(@sbuf, 80)                                           ' read line from file
      
       term.str(@sbuf[0])           '****start at sbuf[0] since there are zero's buried in the
                                    'string it will stop when it hits a zero
       
       term.str(@sbuf[5])            '****same as above but starting a bit further down the string

                      
       Longitude[index]:= num.FromStr(@sbuf[0],Num#DEC5)   '***Make numbers from the strings
       Latitude[index] := num.FromStr(@sbuf[5], Num#DEC6)
        term.tx(13)                                                'CR
        
         

  else
    term.str(string("-- failed to open test file", CR))
    repeat
      waitcnt(0)                                                ' stop here    


  term.str(string("-- waypoints", CR))  

  repeat index from 0 to 9             '****Display the numerical latitudes and longitudes.  
    term.str(string("Longitude- "))
    term.dec(Longitude[index])
    term.str(string("  Latitude- "))
    term.dec(Latitude[index])
    term.tx(13)


  
  term.str(string("      end of test strings: "))       
  term.str(string(CR, CR, "Done.", CR))   
  \sdcard.unmount

  repeat
    waitcnt(0)                                                  ' stop here    


pub readln(pntr, n) | c, len

'' reads line of text from open file
'' -- terminated by CR or EOF
'' -- pntr is address of string buffer
'' -- n is maximum number of characters to read from line

  len  := 0                                                     ' index into string

  repeat n
    c := \sdcard.pgetc                                          ' get a character
    if (c < 0)                                                  ' end of file
      if (len == 0)                                             ' if nothing
        return -1                                               ' return empty
      else
        quit
    if (c == CR)                                                ' if CR we're done with line
      quit
    else
      if (c <> LF)                                              ' if not a line feed
         if c ==","                                            '***look for comma
          c:= 0                                                '***convert comma to the number zero 
       byte[pntr++] := c                                       ' else move c to buffer
        len++                                                   ' update character count
    

  byte[pntr] := 0                                               ' terminate end of line
  
  return len

  
pub pause(ms) | t

'' Delay program ms milliseconds

  t := cnt - 1088                                               ' sync with system counter
  repeat (ms #> 0)                                              ' delay must be > 0
    waitcnt(t += MS_001)

JonnyMac

As you gain more experience you'll put my original code back the way it was. Why? Well, you've turned a generic line reading method into a parser which should in fact be a separate method. When use configuration files I read the line, then scan through them (with a parser method) looking for delimiters. Knowing where they are and what the fragments between should be, the necessary conversions can be dealt with.

SteveWoodrough

Following Jon’s cue, below is the parsing method. The parse method is called after Jon’s line reader method, readln(@sbuf, 80) , has read in all 80 characters. Parse(@sbuf,80) works just fine. A few observations: the Numbers object seems to essentially self parse strings. Assuming that the numbers object is configured for standard base 10 conversion, once the pointer reaches a comma the Numbers object processes everything to that point as a single number.

Anyway, I’m a bit more dangerous thanks to everyone’s input. Hope these notes will help the next person along.

Regards,
Steve

Pub EOS(pntr,n)    |   index                        'replaces commas with zero's to aid in breaking up strings

          repeat index from 0 to n
           if byte[@sbuf][index] == ","
              byte[@sbuf][index]:= 0
 

MagIO2

Buggy code ....

char := sbuf[index]

should be replaced with

char := byte[pntr][index]

otherwise pntr is of no use at all. (similar replacement needed for sbuf[index] := 0 )

char itself is a waste of stack-space because you could come along without.
And finally I'd not call this a parser, as it does not parse anything, it's only replacing comma with stringend.

StefanL38

Jon's right. Every method should do just ONE thing. Test this method and if it works in every thinkable situation.
write next method. This way you will avoid a lot of bugs. This seems to be slower in progress but in the end it is much faster
because you don't waste any time on bug-searching across many lines of code.

Now replacing the comma with a zero terminates the string. Every string-out-put method stops if it cames across a zero.
If you want to access the characters behind the zero you have to go on parsing it and copying to some other string-variable.

best regards
Stefan

SteveWoodrough

Tightened up method above based on feedback.