Datalogger: 4GB (2 2GB SD cards), 24000Hz, 5 pins, 2 COGS
parsko
Posts: 501
Hi all,
I thought I'd toss this out there right now. I finally got it working, after many many revisions. The whole concept may seem silly to those out there, but I have a purpose in the long run.
First: Let me begin by saying that this is the first working version, which will likely under-go many changes in the future, so stay tuned for updates.
Second: The actual logger will use 2 Cogs, one to sort the data, and one for the SD cards. You will need more cogs to gather data (for instance, from an ADC). The COG that is sorting data is my top object, and written in SPIN. This may be why it's only working at 24,000Hz. I am pretty confident that this can be made to be faster. Also, my ultimate goal for this is to have the sorting cog written in assembly, which I have half complete already. I only want to use SPIN for humans (visual outputs and inputs from dudes). Now that I have the SD part working, and a working SPIN version, finishing the assembly routine will go pretty quick (but I'm gonna need to take a few days off to hang with the family, who have suffered).
Third: I know the code and all routines are wicked messy, and will get cleaned up soon.
Forth: The gem to this, if anyone has been reading to find out, is that I have modified Tom Rokiki's sdspiqasm.spin routine to accept two SD cards. This routine has been highly documented with pseudocode (please don't be too ticked at me Tom for revealing your secrets, your routine is genius!!!), and can be used to learn (mostly) how the low level SD routine functions. Also, this is the high speed routine, running a 10,000,000Hz clock.
This routine handles the initialization of BOTH SD cards, and on a fresh startup finishes in about 2/3 of a second. This will likely vary depending on the card. I only use SanDisk cards, as they seem to be the ones who wrote the original standards, and seem to stick to them.
Important note, I have removed the byte write routine, so don't plan on using this for that, ONLY BLOCKS (512 bytes) can be written using this routine. One issue with my routine is that it will occasionally miss a block of data. I've run the routine a dozen or so times, and it comes up about 3% loss, due to the timeout. I have changed Tom's timeout to be 0.005seconds. This is needed because of using two cards on one bus. This is why it loses data. Normally not a big deal if you're datalogging, especially not a big deal if your logging that fast, unless you need the speed and all the data (you data snob!). This may get fixed if I can figure out how, but not on my priority list. You can change the timeout on line 410 of sdspiqasm_Luke... (timecheck2 long 80_000_000/200). This problem, actually, I think is really only related to the read. And, the 3% figure may not be real, it could be less.
(added post haste) This routine does NOT use the filesystem on the SD card. It only uses the card as a MONSTER size memory location. All routines need to be written to output the data to a computer. I haven't gotten this far. And did it this way cause I'm not sure a computer can open a single 2GB file on an SD card.(post haste addition complete)
Fifth: This, as is, may not be too useful to most. If you want to use it, you will likely need to add a cog of your own code to gather data from something, like an ADC. You will need this to be written to line 171 (luke1.byte i+(block0*4) := luke1.byte[noparse][[/noparse]1 <-forum doesn't like this format!) of the top object. I'm currently only logging one thing, and I think the speed will go way up when I start using many variables. I will obviously post an update when I get to higher speed routines. This will run through one loop, loggin onto 32 blocks of both cards, and read out and onto a TV screen. See attached picture for my test output. Line 1 (or two if you live in the US) shows the cogs that the TV and ADC are in, and a "0" next to the SD indidcates a successfull startup. BUF is unused so far. Line two is the startup time in clocks (on left), with the total loop time, in msec, of the test loop of 64 blocks (on right). Then you get the first byte of each of the 64 blocks, followed by PhiPi's cogcounter, and an ongoing output from the ADC's in the lower right (4 numbers).
Why did I do this?
To log data for my brothers racecar. Oh, you meant, why two SD cards? Mostly for volume of data. Also, for redundancy, since every now and them you miss a block of data. I thought I'd be safe and use two SD cards, in case one decided to get pissed and not work, there's a backup.
Did I mention this works on 5 pins?
At 24,000Hz, you can log 49 hours of data. Is that enough? Nah. Imagine what can be done with 10 or 15 pins??? That's 12GB of data. Want to watch a movie???
Anyway, I think Paul had mentioned earlier this week about someone doing exactly this (writing a routine to use 2 SD cards). I was almost done, so I kept quiet. Now here it is, in all it's glory, my 1000 quid (anyone catch that reference and I'll send you one of my 2GB cards!).
Any thoughts???
-Luke
(edit) - You will need to have two SD cards on the same bus 3 line SPI bus, with 2 CS lines. This is an important point I didn't mention. My sdspiqasm_Luke routine has a start routine where you can customize this. I DO NOT use the same pin configuration as Tom's SD card routine, so BEWARE.
Post Edited (parsko) : 6/26/2008 3:10:03 AM GMT
I thought I'd toss this out there right now. I finally got it working, after many many revisions. The whole concept may seem silly to those out there, but I have a purpose in the long run.
First: Let me begin by saying that this is the first working version, which will likely under-go many changes in the future, so stay tuned for updates.
Second: The actual logger will use 2 Cogs, one to sort the data, and one for the SD cards. You will need more cogs to gather data (for instance, from an ADC). The COG that is sorting data is my top object, and written in SPIN. This may be why it's only working at 24,000Hz. I am pretty confident that this can be made to be faster. Also, my ultimate goal for this is to have the sorting cog written in assembly, which I have half complete already. I only want to use SPIN for humans (visual outputs and inputs from dudes). Now that I have the SD part working, and a working SPIN version, finishing the assembly routine will go pretty quick (but I'm gonna need to take a few days off to hang with the family, who have suffered).
Third: I know the code and all routines are wicked messy, and will get cleaned up soon.
Forth: The gem to this, if anyone has been reading to find out, is that I have modified Tom Rokiki's sdspiqasm.spin routine to accept two SD cards. This routine has been highly documented with pseudocode (please don't be too ticked at me Tom for revealing your secrets, your routine is genius!!!), and can be used to learn (mostly) how the low level SD routine functions. Also, this is the high speed routine, running a 10,000,000Hz clock.
This routine handles the initialization of BOTH SD cards, and on a fresh startup finishes in about 2/3 of a second. This will likely vary depending on the card. I only use SanDisk cards, as they seem to be the ones who wrote the original standards, and seem to stick to them.
Important note, I have removed the byte write routine, so don't plan on using this for that, ONLY BLOCKS (512 bytes) can be written using this routine. One issue with my routine is that it will occasionally miss a block of data. I've run the routine a dozen or so times, and it comes up about 3% loss, due to the timeout. I have changed Tom's timeout to be 0.005seconds. This is needed because of using two cards on one bus. This is why it loses data. Normally not a big deal if you're datalogging, especially not a big deal if your logging that fast, unless you need the speed and all the data (you data snob!). This may get fixed if I can figure out how, but not on my priority list. You can change the timeout on line 410 of sdspiqasm_Luke... (timecheck2 long 80_000_000/200). This problem, actually, I think is really only related to the read. And, the 3% figure may not be real, it could be less.
(added post haste) This routine does NOT use the filesystem on the SD card. It only uses the card as a MONSTER size memory location. All routines need to be written to output the data to a computer. I haven't gotten this far. And did it this way cause I'm not sure a computer can open a single 2GB file on an SD card.(post haste addition complete)
Fifth: This, as is, may not be too useful to most. If you want to use it, you will likely need to add a cog of your own code to gather data from something, like an ADC. You will need this to be written to line 171 (luke1.byte i+(block0*4) := luke1.byte[noparse][[/noparse]1 <-forum doesn't like this format!) of the top object. I'm currently only logging one thing, and I think the speed will go way up when I start using many variables. I will obviously post an update when I get to higher speed routines. This will run through one loop, loggin onto 32 blocks of both cards, and read out and onto a TV screen. See attached picture for my test output. Line 1 (or two if you live in the US) shows the cogs that the TV and ADC are in, and a "0" next to the SD indidcates a successfull startup. BUF is unused so far. Line two is the startup time in clocks (on left), with the total loop time, in msec, of the test loop of 64 blocks (on right). Then you get the first byte of each of the 64 blocks, followed by PhiPi's cogcounter, and an ongoing output from the ADC's in the lower right (4 numbers).
Why did I do this?
To log data for my brothers racecar. Oh, you meant, why two SD cards? Mostly for volume of data. Also, for redundancy, since every now and them you miss a block of data. I thought I'd be safe and use two SD cards, in case one decided to get pissed and not work, there's a backup.
Did I mention this works on 5 pins?
At 24,000Hz, you can log 49 hours of data. Is that enough? Nah. Imagine what can be done with 10 or 15 pins??? That's 12GB of data. Want to watch a movie???
Anyway, I think Paul had mentioned earlier this week about someone doing exactly this (writing a routine to use 2 SD cards). I was almost done, so I kept quiet. Now here it is, in all it's glory, my 1000 quid (anyone catch that reference and I'll send you one of my 2GB cards!).
Any thoughts???
-Luke
(edit) - You will need to have two SD cards on the same bus 3 line SPI bus, with 2 CS lines. This is an important point I didn't mention. My sdspiqasm_Luke routine has a start routine where you can customize this. I DO NOT use the same pin configuration as Tom's SD card routine, so BEWARE.
Post Edited (parsko) : 6/26/2008 3:10:03 AM GMT
Comments
Parsko,
I thought the limit of storage was only 32MB even if you have a larger card?
I'm afraid you're talking apples while I'm talking oranges. Though, I agree that your code can run faster, it uses two more cogs, and, from what I gather, only logs ~1900 bytes of data. Is this true?
My concept will allow one to use 6 other cogs for other stuff, while having a datalogger which will be customizable to suit the needs of the programmer. In my assembly routine I pass a logging frequency, with the number of bytes wanted to log. So, anyone will be able to start the cog and run it at, say, 100Hz, 10 data points, for a 1000Hz sampling data rate. One of my requirements (for myself) was to fit this into two cogs, minimum pins, fast speed. I will need the other 6 cogs for other stuff. I'll eventually, way down the road, be logging around 20 channels of data from the car, and I'll need the room on my prop to allow this.
After having slept on what I proposed, I thought that my data rate of 25kHz can pretty easily go up the the write limit of the SD cards, which should be well over 100kHz. My code only uses 1 data point, logging as fast as my SPIN routine will go, which turns out to be 25kHz. When I move it all into assembly, and combine multiple data points, (aka 2,3,4,etc... ADC's), it will be able to log much faster.
I still have to figure out how to get the data out easily, as reading 4GB into a serial port at 115k rate will take a bit. But, for most people, this won't be an issue. In my case, I'll download the data from the car about once every two weeks (the frequency I am able to dedicate my time at). I'm guessing that I'll have much less than 1 GB of data. But, Excel only allows 2^16 (65536) rows of data, by 256 columns, which totals out to be 16MB of data. Obviously, this will be an issue. I need to find a (free) program that can record much more data than that, or figure out a way to break it up into smaller, more easily consumed, files.
The reason I didn't go with the file system is that it has limitations to the number of files one can have per card. This, along with using two cards (and needing two files open!), would be too much to handle. Read speed is NOT nearly as critical as the write speed, IMHO, since I can sit there consuming beers while data is downloading.
My goal was to provide this routine to the masses at a write speed of around 100kHz. That's 100 bytes of data per every millisecond. I don't know of many things that need more than this at the hobby level. I'm thinking a good fit for this will be the rocket guys. They could make a little module that would fit into many different rockets, and record the whole days worth of launches, on all rockets, wicked fast. Just move the module from rocket to rocket. Or, another thought, is a dude like Tracy Allen could set one of these up to record about 10 years of weather data, in the most remote of locations, and not worry about running out of room. Or, write a movie to the card (or multiple buses of cards) and watch movies.
I'm mostly curious if anyone out there will actually build a setup to try this out. I'd love to know it works for someone else, too. I'll try to get pictures of my frankenstein SD boards (one uses paperclips, while the other uses an actual SD socket), it's pretty ugly, but works.
-Luke
For crunching data nothing beats Matlab, unfortunately it's far from free. Good news though, someone's making an open source clone, Octave. Haven't tried it yet, but if the help files are even half as good as the original it's well worth trying out.
115kbaud does sound a bit slow for up to 4GB of data. Might look into sweet talking Hanno to let you borrow the serial code he's using for Viewport. He runs 2-3Mbaud over a prop plug.
Marty
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Lunch cures all problems! have you had lunch?