Reading CSV files from an SD card
Sariel
Posts: 182
Ok.. I have looked over the Gadget Gangster site on this very issue, but thier explanation does not really help me try and do exactly what I would like. What I am looking for is a method that will take a series of 128 binary longs that are stored in a .CSV file and dump it into a variable array (called "L[0]" to L[127]") for use elsewhere in my program. I am getting a lot of garbled data whenever I try this:
Here is the CSV file I am working with. Please note that I would rather not change the formatting if I did not absolutely have to, since it is generated by another Excel file
master.zip
This is just the start of something bigger. Eventually, I would like to write the same formatted CSV files to the SD card, allowing the user to type the file name on a keyboard. But for now, I just want to concentrate on getting the data off the card, before I try to write it back.
Can someone show me the error of my ways, or at least point me in the right direction?
CON
_clkmode = xtal1 + pll16x
_xinfreq = 5_000_000
DO = 12 'Set these pins to match your SD connections.
CLK = 13
DI = 14
CS = 15
OBJ
sdfat : "fsrw"
debug : "FullDuplexSerialPlus.spin" ' PST Driver
VAR
Long DATA
Long l[128]
PUB Main| mount, r, counter
Debug.start(31, 30, 0, 57600) ' Start Debugger
waitcnt(clkfreq*2 + cnt) 'Increase the 4 if more time is required.
mount := \sdfat.mount_explicit(DO, CLK, DI, CS)
if mount < 0
debug.str( string( 13, "Failed to mount", 13 ) )
abort
debug.str(string(13,"SD was card found and mounted fine.",13,13))
sdfat.popen(string("master.csv"),"r")
counter:=0 'Setup a counter starting at zero.
repeat 'Start a repeat loop.
r := sdfat.pgetc 'Get a character from master.csv
if r == 44 'If character is a comma
l[counter] := @data
if r > 44 'If the character "greater than 44"
data[counter]:=data[counter]+r ' send character to data string.
counter++ ' increment the counter.
counter++ ' and again since there is always a CR
repeat counter
debug.bin(l[counter], 32)
debug.str(string("Closing file and unmounting SD card.",13))
sdfat.pclose
'Unmount the card and end program.
sdfat.unmount
Here is the CSV file I am working with. Please note that I would rather not change the formatting if I did not absolutely have to, since it is generated by another Excel file
master.zip
This is just the start of something bigger. Eventually, I would like to write the same formatted CSV files to the SD card, allowing the user to type the file name on a keyboard. But for now, I just want to concentrate on getting the data off the card, before I try to write it back.
Can someone show me the error of my ways, or at least point me in the right direction?
zip
272B
Comments
1) When your program sees a comma, you do "l[counter] := @data" assuming that the character string in data will be converted to a binary number. What you get in the array is the address of data.
2) When your program sees a non-comma character, you do "data[counter] := data[counter]+r". I don't know what you're trying to do, but it doesn't do anything useful.
I suggest you look at the BS2_Functions object from the Object Exchange and the SERIN_DEC method particularly. This uses a call to SERIN_CHAR to get a character and you could replace those calls with sdfat.pgetc. SERIN_DEC shows you how to read in a string of digit characters to a buffer and convert the digits into a numeric value. It uses a CR as the delimiter, but you could change that to a comma and CR.
I've found Excell doesn't like to play nice with csv files.
I use the Propeller as a data logger almost everyday. I save my data using comas as delimiters.
When I had the Prop save the file as a csv file, Excel wouldn't open it correctly.
I needed to save it as a txt file and use the "Import" feature in Excel. (I know of at least two other forum member who had the same problem.)
I believe Excel uses and expects tabs as delimiters. IIRC tab is ASCII 9 (I'm not sure).
There might be a way of having Excel use comas instead of tabs in the csv files.
I personally just use comas as field separators. Each line should end with a carriage return (ASCII 13).
I have the Prop save these files as txt files on a SD card and then either import them into Excel or "open with" OpenOffice.
I generally use OpenOffice more than Excel now.
Here's a section of code:
Sd.writeCharacters(@rf_rxbuf + 1) Sd.writeCharacter(44) Sd.writeCharacters(@timeArray + 5) Sd.writeCharacter(13)When I import the data, the above code writes, in Excel, I get two columns.
(The first column) rf_rxbuf holds data received from a wireless module. I use "+ 1" to have the Prop skip the first character of the string when it writes to the SD card.
timeArray holds a time-stamp (again I want to skip the first 5 characters)(this data makes up the second Excel column).
"Sd.writeCharacter(44)" inserts a coma (creates a new Excel column).
"Sd.writeCharacter(13)" inserts a carriage return (creates a new Excel row).
The method names are to Kye's "FATEngine.spin". I believe the method names have been changed in his new SD drivers.
I know you're asking about reading from an SD card but you said writing to the SD card was also one of your goals.
Duane
pub readln(pntr) | c, len '' reads line of text from open file '' -- terminated by CR or EOF '' -- removes linefeeds if present len := 0 ' index into string repeat c := \sd.pgetc ' get a character if (c < 0) ' end of file if (len == 0) ' if nothing return -1 ' return empty else quit if (c == CR) ' if CR we're done with line quit else if (c <> LF) ' if not a line feed byte[pntr++] := c ' else move c to buffer len++ ' update character count byte[pntr] := 0 ' terminate end of line return lenConverting your buffer back to a number should be easy; you can do it with this:
pub bin2dec(pntr, len) | bval, c '' Converts binary string at pntr to decimal value '' -- removes non-binary characters at head of string '' -- allows underscore character in binary string bval := 0 repeat ' remove non-binary header c := byte[pntr] case c "0".."1": quit other: pntr++ repeat while (len > 0) ' convert string c := byte[pntr] case c "0".."1": bval := (bval << 1) | (c - "0") pntr++ len-- "_": pntr++ other: quit return bvalI use the bin2dec method in the AP-16+ to convert a set of flag bits from the SD card; for example:
SFX=0000_0000_1111_0011
... to a number I can use the code. I like having the underscore character in my numbers to separate groups, hence the method skips over them.
BAH! good catch! I'm still trying to get used to this concept of accesing Data via address. By the number of questions everyone else has responded to regarding that, no suprise that I still make an error in that once in a while.
Honestly, that is a bit of code that I forgot to comment out from the Gadget Gangster example
Thanks for the suggestion. I will read through it and tear it apart this weekend before I have to come back into work on monday.
Duane--
Honestly, It is ok that excel cannot read the files back. In the end of this project, I want to give the user 2 ways of generating files for reading by the prop.
1.) Input taken from a formatted Excel file that saves to a .csv
2.) Propeller-generated via program that dumps 128 longs.
For a clearer picture of what I am doing, here is my Excel file. What's going on is that user enters in the wiring on the "Pinout" Sheet, then goes to the "Cable Data" Sheet, and hits the "Make CSV" button. That will maake the current sheet dump to a CSV. User then copies it to the SD card, and my program will read that information, and use it for the test.
I will look into everything you guys said. THank you for your replies.
NEW Cable Data Master(MKII).xls
I've not had these problems. Can you attach one of the comma-separated value files that
you say Excel can't load? If you would, I'll probably be able to tell you what's wrong with it.
-tom
I don't have Excel on this computer so I can't test to be sure a file wont open or not.
I just know I've made the suggestion to "Import" a txt file rather than opening a cvs file in Excel to several people on the forum they agreed it solved their difficulty.
It's not longer an issue for me since I use OpenOffice most of the time.
Duane
OBC
When I read "data.csv"(from GadgetGangster example), I get 2 values to print out (both 0's). Thinking there may be some extra header garbage that microsoft puts into the CSV file that is not visible in notepad, I re-created the master.csv as first a normal every-day txt file, and and kept one copy as a txt, and another renamed as a csv (hopefully eliminating the header Smile), it still won't read it. Either one. (yes, I am changing the string in line 42 to reflect the file I want read). Here is the code that I am trying to use now, and the associated CSV and TXT file I am trying to get to work.
data_master.zip
CON _clkmode = xtal1 + pll16x _xinfreq = 5_000_000 DO = 12 CLK = 13 DI = 14 CS = 15 LF = 10 CR =13 OBJ SDFat : "fsrwFemto" Debug : "FullDuplexSerialPlus.spin" VAR Long l[128] long ioControl[2] long counter long c long len long bval long pntr PUB Main | mount Debug.start(31, 30, 0, 57600) ' Start Debugger waitcnt(clkfreq*2 + cnt) 'Increase the 2 if more time is required. debug.str(string(16)) sdfat.start(@ioControl) ' Start SPI driver mount := \sdfat.mount(DO, CLK, DI, CS) if mount < 0 Debug.str( string( cr, "Failed to mount", cr ) ) abort debug.str(string("SD card was found and mounted fine.",cr,cr)) result := SDFAT.popen(string("master.csv"),"r") if result < 0 Debug.str(string("Failed to load file.", cr)) abort counter := 0 repeat 128 '' reads line of text from open file '' -- terminated by CR or EOF '' -- removes linefeeds if present len := 0 ' index into string repeat c := \sdfat.pgetc ' get a character if (c < 0) ' end of file if (len == 0) ' if nothing return -1 ' return empty else quit if (c == CR) ' if CR we're done with line quit else if (c <> LF) ' if not a line feed byte[pntr++] := c ' else move c to buffer len++ ' update character count byte[pntr] := 0 ' terminate end of line '' Converts binary string at pntr to decimal value '' -- removes non-binary characters at head of string '' -- allows underscore character in binary string bval := 0 repeat ' remove non-binary header c := byte[pntr] case c "0".."1": quit other: pntr++ repeat while (len > 0) ' convert string c := byte[pntr] case c "0".."1": bval := (bval << 1) | (c - "0") pntr++ len-- "_": pntr++ other: quit l[counter] := bval debug.dec(l[counter]) debug.str(string(CR)) counter++ 'Unmount the card and end program. SDFAT.unmount debug.str(string("Closing file and unmounting SD card.",CR)) SDFAT.pcloseI know it is probably just something simple that I am overlooking, but can someone please take a glance for me?
And, again, I just wanted to say that once the file leaves excel, it will never, ever need to go back into it, or any other spreadsheet program. Excel is just being used as a dirty way to make these variable sets, without having techs setting these up ever be forced to mess with, or even see the actual source code used. I guess you can say that I am using excel as a config file generator.
Awesome. thank you very much, and it works like a charm. I already have it integrated into my base program, and it is humming along nicely. Now, on to adding the OS block I already got going so that one can specify the config file to load up.This project is very quickly becoming a beast for a beginner like myself.
I am integrating this in with my auto tester, and a modified version of PropDOS. Any good ideas on how I can make a command in PropDOS that instead of always loading "master.cvs", it loads in whatever comes next after someone types in "TEST"? to make the tester do exactly what it is now, one would type in "test master". I have poured over PropDOS a few times, and I see how you can do it if the option is known like with |more, or /w, but what if the option can and will be different every time?
The command-line parsing code in PropDOS suffers from the fact that I was a beginner in spin when I wrote it. It works, but it's painful.
I did some work similar to this recently, (in the last month). I've got a lot going on this week with expo coming Saturday, but I'll see if I can round up some better code for that job.
Edit:
One trick that might help you is to use the STRINGS object. Once you have the file name parsed from the command line, the following will help you combine the untyped file extension with it.
OBJ str : "STRINGS" sdfat : "fsrw" PUB Example(commandline) extension:=string(".cvs") filename:=str.Combine (commandline,extension) sdfat.popen(filename,"r")OBC
Get outta my head! I was just thinking about that, and trying to get my idea of a secondary command entry to work. I cut and pasted the meat from your command line interpreter, and was trying to think of a way to do just that to the "cmd" variable.
(I don't want to confess how long it took me to figure out that simple trick.)
OBC
Thanks for the help, Jeff.
I try to transmit a self generated google KML file trough a wifi ESP as serial bridge.
I get the issue with large file the data is not transmited compleet.
Serial terminal pause , and missed some part and do not go til end?
I get the same issue on serial terminal using You code jm_read_text
I only change " check := \sd.popen(string("MLYN0606.txt"), "r") ")
My file attached
I guess it is an buffer issue?
Thank you
PS. I have to change .KML to .TXT to uploade on parallax? but is is a normal google .KML file
Some file formats are not supported as attachments.
Thanks