Datalogging at higher speeds

-GRIMM-

I am working on a project where i need to log speeds at fast rates (30 readings per second +) for 50 minutes. (about 220,000 readings)

The values for time and a sensor need to be recorded, so I need a device that can handle 60+ writes per second.

Currently with a parallax USB datalogger we are getting about 20-30 writes per second.

Are there any other sensors out there that can meet the speed requirements?

I read about Ramtron F-Ram but i can not find a source with a pre-mounted circuit board.

Ideally this would be the best bet, but my Surface mount soldering skills arent too great.

Any help is appreciated.

Thanks

-Steven

Mike Green

You can easily get rates like those you've implied by using an SD card. Look at Kye's SD driver in the Object Exchange. He has a thread with some actual data rates posted by a variety of people with different SD cards. Both Parallax and SparkFun have microSD card boards and SparkFun has an SD card board as well.

You can also attach a single Winbond flash memory chip (8-pin ... comes in PDIP too) to 4 I/O pins and use the I/O driver in the Object Exchange. These come in capacities up to 2MBytes. I think a microSD card would be better if you have the physical room.

turbosupra

Are you able to have a pc around when doing this? Hanno's Ezlog can write to an excel spreadsheet for reasonably easy data managment

http://hannoware.com/ezlog/

lonesock

I second the uSD card. If you don't need directory support, and need faster speed than Kye's drivers, check into FSRW (version 2.6, also in the obex).

Jonathan

RS_Jim

Grimm,
Since you already have the hardware, why not accumulate 30 readings in a 32 byte,word or long circular buffer and write to the datalogger 1 per second?
RS_Jim

-GRIMM-

The circuit is a standalone project for the 50 minutes it is recording, after which we will remove the data from it and transfer it to a computer.

I think we will order the micro-SD card adapter and experiment.

RS_Jim, I am not too familiar with the propeller chip yet, how would I go about doing that?

The values being recorded come from a gyroscope (values from 0-1100 roughly) and clock time, cnt.

Thanks

-Steven

Mike Green

What RS_Jim suggests has nothing to do with the Propeller specifically. It's a general purpose programming technique. You have an array of some length and you stuff a value into it (one long, two longs, whatever) at the index specified by some variable like X. You then increment X by the number of items (long, word, byte, etc.) used for an entry. You wraparound the value of X to the size of the array. If the array is 10 elements long, you'd increment X like "X := (X + 1) // size" where size is 10. This creates a circular buffer for the values. To read the values out of the buffer, you'd have another index, say it's Y. You'd treat Y the same way and increment it with wraparound the same way. You'd also initialize both X and Y to the same starting value (like zero). If X == Y, then the buffer is empty. If it's not empty, you can remove a value from it and increment Y. If (X+1)//size == Y, then the buffer is full (think about it) and you have a problem. If it's not full, you can put a new value into it and increment X.

The portion of the program that fills the buffer can run in one cog while the portion of the program that empties the buffer can run in another cog. Look at FullDuplexSerial for an example of this technique with one side written in Spin and the other side in assembly.

RS_Jim

Thanks for that explanation Mike, I hadn't been able to get back to a computer to reply to his question about circular buffers. Full duplex serial is a great example of spin/pasm access to a circular buffer. Grimm I had additional thoughts based on the additional info supplied in your last post. Your gyro info is one word in length and time data for one second is only 30. I would use a circular buffer of longs, writing the gyro data into a long and just adding %1_0000_0000_0000_0000 to the long each tick. Then you just do wrlong data,buffpntrtail, increment the buffpntr tail, test for rollover and write it back to the hub as a long. After 30 ticks clear the long and start over. Update the hrs,mins,secs once per sec when you pick up the longs of the buffer and transfer it to storage. Each second would represent 30 longs + an hrs,mins,secs time stamp. Let the reading software parse the longs for the hrs,mins,secs,ticks info. I hope this all makes sense. I am going to use a similar technique to pass info from a cog that is reading an adc at the rate of once per millisecond, and I want to be sure that the receiving cog gets all of the readings in correct sequence.
RS_Jim

-GRIMM-

I understand the method in theory. In SPIN, however, I would not know where to start!

Perry

-GRIMM- wrote: »

I understand the method in theory. In SPIN, however, I would not know where to start!

My stupid video capture uses double buffering to record 30 frames per second of video/audio to an SD card http://forums.parallax.com/showthread.php?98516-stupid-video-capture&highlight=stupid+video

You might get some ideas from that.

Perry

-GRIMM-

I downloaded Kye's FAT32 driver for SD cards.

Im playing with it now trying to get it to the point where I can wire it up and test it when the sd card adapter comes in (Wednesday)

-GRIMM-

What i get from the driver produced by Kye is; to write I use:

fat.writeLong(value)

So if Im not mistaken, it should be as simple as taking my current USB datalogger program, and replace:

logger.writeToFile(@buffer2,strsize(@buffer2),0)

with

fat.writeLong(@buffer2)

Sound correct?

Perry

-GRIMM- wrote: »

What i get from the driver produced by Kye is; to write I use:

fat.writeLong(value)

So if Im not mistaken, it should be as simple as taking my current USB datalogger program, and replace:

logger.writeToFile(@buffer2,strsize(@buffer2),0)

with

fat.writeLong(@buffer2)

Sound correct?

No!
You need to initialize theSD device
open a file for write
then you can write to the file
when finished writing
you need to close the file
and probobably then close the device

You should really look at how my Stupid Video works
even Kye admits that fsrw is faster
also writing short blocks is slower than writing long blocks (less over head)
with "double" buffering you can receive new data while the last block is being written by the SD i/o card.

-GRIMM-

Initialize with

programMount(stringPointer)
programMakeDirectory(stringPointer)
programMakeFile(stringPointer)

Loop with data

fat.openFile(fat.newFile(string("testfile")), "W")
fat.writeData(@sectorBuffer, 512)
fat.closeFile

(possibly not opening/closing file every time)

%%%%%%%%%

Finish with

fat.unmountPartition


Forgot there was a little more to it than that.

I didnt realize the attachment in your video thread, I'm reading through your codes right now to try and understand it.

Thanks

-Steven

Perry

OK now you have it.

actually I think 30 readings per second is quite slow. My only concern was that the writing might run into problems because the timing of the write to the SD card would be variable and make your data not be written at exactly equal time periods.

-GRIMM-

dt isnt as crucial as frequency.

I just got some of the parts, I will begin experimenting and let you know how it goes. First obstacle is learning how to use a 10-bit adc instead of 8-bit.

Thanks

-Steven

-GRIMM-

Im hoping to save from creating a new thread to tackle this problem. Im using an ADC driver I found on this forum to read the value to save to the uSD card. I have a few questions on this.

Im sorry I dont recall where I got this from, but the files are: ADC_INPUT_DRIVER.spin and ADC_INPUT_example.spin

I took this code and modified it to work with our 1 channel 10 bit ADC. For some reason it is not behaving as I would expect.

When using the command:

DEBUG.dec(chanmax[1]) - The channel minimum value (of the entier run)
DEBUG.dec(ADC.getval(1)) - I get 0
DEBUG.dec(Chanval[1]) - I get the channel maximum (from the entire run)
DEBUG.dec(chanmin[1]) - I get 2

I can solve this by using 'ADC.resetmaxminall' after either Chanval or chanmax to get the current value, but I would think there is a command available to not have to reset every loop.

Also, The value being output is 479, which coordinates to 1.479 (measured) volts. Is there a way to have it output all 4 digits (1479)?

Thanks!

I will begin experimenting with the uSD card soon

-GRIMM-

So far i cant get the uSD to work either. I have double checked the wiring (redid it twice just to make sure)

I am using Kye's '#SD3.01FATDemo' to get things working. I changed the pins at the beginning of the program to validate everything. I did set this:

_SD_WP = -1 ' -1 ifnot installed.

because I'm not sure what it is, and I dont see a pin with this label.

When I connect with PST i type in help and get the screen, but when I try anything card related (mount, format, etc) It just gives me: Disk IO Error

Not sure what steps to take from here, any hints?

In case it may be a concern, my pins are:

_SD_DO = 21
_SD_CLK = 20
_SD_DI = 19
_SD_CS = 18
_SD_CD = 16

3.3V, GND verified.

Thanks

-Steven

Duane Degn

_SD_WP is write protect. uSD card don't have a write protect tab so you would want to use "-1".

I have set _SD_CD to -1 too otherwise Kye's driver wont work for me.

I've also had to reformat many of my SD cards before they would work with Kye's newest driver. "Fast Format" wont do it neither will the format command within Kye's driver.

What ADC chip are you using?

Duane

Perry

Kye's code need a real time clock chip DS1307, do you have one of those? so you need to use 6 pins.
Somewhere I can't find it now he suggested you could comment that out.

  _clockDataPin = 29
  _clockClockPin = 28
  
  _cardDataOutPin = 11
  _cardClockPin = 10
  _cardDataInPin = 9
  _cardChipSelectPin = 8

I don't care about date stamps so I use the "fsrw" code.

I dont' see enough info about your ADC chip to comment on it.

-GRIMM-

Sorry the ADC is an 8 pin MCP3001.

I do not have an external clock. I dont need real time but i do, however, need relative time (time between readings)

I have a 6.25 MHz crystal.

I tried formatting it in all the FAT formats, not quick format. Still isnt working. I will try out fsrw tomorrow.

I was having a hard time following fsrw which is why I didnt use it at first. Trying to read through the 'Smart Video' capture files.

Thanks

-Steven

Duane Degn

Steven,

If you don't have an external clock you'll want to set _clockDataPin and _clockClockPin to -1.

Duane

Perry

You will find http://forums.parallax.com/showthread.php?87289-SPI-SHIFTIN-High-Speed&highlight=MCP3001 will answer your questions about MCP3001.

The Video capture can be used without the video part for just audio recording or data recording.

You may need to change the clock settings if you do not use the standard 80MHz clock

-GRIMM-

Thanks for all your help.

So far i have gotten the MCP3001 working with the program linked by Perry. I'm not sure how often it is reading, but it is only putting values out in bursts to PST. Also, it is only showing the '479' from 1.479 volts instead of 1479.

also, I have the uSD card working with Kye's driver and demo, but not standalone.

Im trying to get them working together, but have a few issues. Im trying two combinations. (MCP3001 code with fsrw and the same ADC code i had with Kye's driver.

This is the first one I tried:

'after setting all the pins

  fat.FATEngineStart(_SD_DO, _SD_CLK, _SD_DI, _SD_CS, -1, -1, -1, -1, -1)                                                                           
  ADC.start_pointed(Vo_p, Vn_p, Vclk_p, Vcs_p, 1, 1, 10, 1, @chanstate, @chanval, @chanmax, @chanmin)
  
  fat.newDirectory(string("Steriotort"))
  fat.changeDirectory(string("Steriotort"))
  fat.newFile(string("DataFile.txt"))
  fat.openFile(string("DataFile.txt"), string("W"))
  fat.writeString(string("Data"))
  fat.closeFile
  fat.openFile(string("DataFile.txt"),string("A"))

repeat 1000
        dec(Chanval[1],@buffer2)
        fat.writeLong(@buffer2)
        ADC.resetmaxminall
    
  fat.closeFile  
  fat.unmountPartition
  fat.unmountPartition

This is where I made it to using fsrw.

fat.FATEngineStart(_SD_DO, _SD_CLK, _SD_DI, _SD_CS, -1, -1, -1, -1, -1)                                                                          
ADC.start_pointed(Vo_p, Vn_p, Vclk_p, Vcs_p, 1, 1, 10, 1, @chanstate, @chanval, @chanmax, @chanmin)
  
fat.mount_explicit(_SD_DO, _SD_CLK, _SD_DI, _SD_CS)
sd.popen(string("Data.txt"),"w")

repeat 1000
       dec(Chanval[1],@buffer2)
       sd.pwrite(@buffer2,12)
       ADC.resetmaxminall 

sd.pflush  
sd.pclose
sd.unmount  

I am basically following the "Record Video" section from 'Amazing Video' and the fatengine demo from Kye to try and follow the way they create/write files.

Neither are creating the file though. Does anyone see any large errors with either of these methods?

Ill keep trying it out.

Thanks

-Steven

Perry

Try something like this

'Start of ADC process
'SDA    = Data Pin
'SCL    = Clock Pin
'CS     = Chip select
'Input  = State of the pins
'Bitcnt = Variable used to count the input bits (in this case 10)
'Mask   = used to filter the bits into the output variable (adcvalue)
'ADCVal = the value to write to main memory
''========================================================================================
'' 1= output
'' 0= input
''======================================================================================
{{   }}
con


  _clkmode = xtal1 + pll16x
  _xinfreq = 5_000_000

  SD_DO         = 0     ' mount_explicit(DO, CLK, DI, CS)
  SD_CLK        = 1
  SD_DI         = 2
  SD_CS         = 3

obj
  text  : "tv_text"
  sd    : "fsrw"

var
  long ADCval
  byte output_str[32]

PUB Start | writ,v

  text.start(12)
  waitcnt (80_000_000 + cnt)
  cognew (@shiftin, @ADCval)
  sd.mount_explicit(SD_DO, SD_CLK, SD_DI, SD_CS)
  sd.popen(string("DataFile.txt"),"w")
  repeat v from 0 to 99
    writ := dec_to_string(v)
    text.str(@output_str)

    write_to_SD(@output_str,writ)
    write_to_SD(string(","),1)
    text.out(",")

    writ := dec_to_string(ADCval)
    text.str(@output_str)
    text.out(13)
    write_to_SD(@output_str,writ)
    write_to_SD(string(13),1)

    sd.pflush

  sd.pflush
  sd.pclose
  sd.unmount
  text.str(string("all finished "))

PUB write_to_SD(some_string,len) | di

    di:=sd.pwrite(some_string,len)
    if di<0
      text.str(string("SD write error "))
'      quit
    sd.pflush

PUB dec_to_string(value) | i,c

'' Print a decimal number
  c := 0
  if value < 0
    -value
    output_str[c++] := "-"

  i := 1_000_000_000

  repeat 10
    if value => i
      output_str[c++] := value / i + "0"
      value //= i
      result~~
    elseif result or i == 1
      output_str[c++] := "0"
    i /= 10
    output_str[c] := 0

  return c


DAT
shiftin
                org 0
''Set the direction and state of the I/O pins and non-recurring events
''                        
                or              dira,   OS            ''Set OS to output
                or              dira,   SCL           ''Set SCL to output
                or              dira,   CS            ''Set CS  to output
                xor             outa,   SCL           ''Set SCL pin high (1)
                xor             outa,   CS            ''Set CS  pin high (1)
                                                      ''
                mov             wcnt,   #0            ''Zero wcnt
                add             wcnt,   cnt           ''Add system counter to wcnt
                add             wcnt,   #13           ''Add 13 clock delay to wcnt
                waitcnt         wcnt,   Tsucs         ''Pause COG TSucs clocks

                'rdlong          ADCtemp,PAR
                
''==========================================================================================
''Start of conversion
:loop1                                                ''Loop1  Main loop
                xor             outa,   OS            '' Toggle OS pin
                xor             outa,   CS            '' Toggle CS  pin low (10)
                xor             outa,   SCL           '' Toggle SCL pin low (10), 0th falling edge

                mov             ADCtemp,#0
                                                      '' 
                mov             wcnt,   #0            '' Zero wcnt
                add             wcnt,   cnt           '' Add system counter to wcnt
                add             wcnt,   #13           '' Add 13 clock delay to wcnt
                waitcnt         wcnt,   Tsucs         '' Pause COG until wcnt is reached by cnt
                                                      ''
                xor             outa,   SCL           '' Toggle SCL pin high (01)
                                                      ''   1st clock, 1st rising edge, start sample period
                waitcnt         wcnt,   Thi           '' Pause COG Thi clocks
                                                      ''
                xor             outa,   SCL           '' Toggle SCL pin low  (10), 1st falling edge
                waitcnt         wcnt,   Tlo           '' Pause COG Tlo clocks
                                                      ''
                xor             outa,   SCL           '' Toggle SCL pin high (01), 2nd clock, 2nd rising edge
                waitcnt         wcnt,   Thi           '' Pause COG Thi clocks
                                                      ''   
                xor             outa,   SCL           '' Toggle SCL pin low  (10), 2nd falling edge
                waitcnt         wcnt,   Tlo           '' Pause COG Tlo clocks
                                                      ''
                xor             outa,   SCL           '' Toggle SCL pin high (01), 3rd clock, 3rd rising edge
                waitcnt         wcnt,   Thi           '' Pause COG Thi clocks
                                                      ''
                xor             outa,   SCL           '' Toggle SCL pin low  (10), 3rd falling edge
                waitcnt         wcnt,   Tlo           '' Pause COG Tlo clocks
                                                      ''
                xor             outa,   SCL           '' Toggle SCL pin high (01), 4th clock, 4th rising edge
                                                      ''  
                mov             Masktmp,#0            '' Zero Masktmp
                xor             Masktmp,Mask          '' Set initial mask = .. 0010_0000_0000 
                mov             bitcnt, #10           '' Set bitcnt = 10 (10bit ADC)
 
                                                      ''
:loop2                                                '' Loop2  This is the start of the reading of the bits
                waitcnt         wcnt,   Thi           ''  Pause COG wcnt clocks

                mov             input,  #0            ''  Zero input  
                or              input,  INA           ''  Place the state of all the pins into input
                and             input,  SDA           ''  Place the state of SDA into input (if state=1 AND SDA=1  1)
                test            SDA,    input   WZ    ''  If input AND SDA = 0  WZ=1 else = 1  WZ=0
        if_nz   or              ADCtemp,Masktmp       ''  If Z flag is false (0), then "or" ADCtemp with Masktmp
                shr             Masktmp,#1            ''  Shift mask right 1 bit

                xor             outa,   SCL           ''  Toggle SCL pin low (10), nth clock, nth falling edge
                waitcnt         wcnt,   Tlo           ''  Pause COG wcnt clocks

                xor             outa,   SCL           ''  Toggle SCL pin high (01)                                

                djnz            bitcnt, #:loop2       ''  Decrement count by 1, jump to #loop
                
                wrlong          ADCtemp,PAR           ''  Write ADCtemp to ADCval in main memory

                xor             outa,   CS            ''  Toggle CS pin high  (01)
                waitcnt         wcnt,   Tcsh
        jmp #:loop1                                   ''Jump to Loop1, repeat ADC endlessly                  
                
SDA     long  |<0               ''SDA = 0000_0000_0000_0000_0000_0000_0000_0001
SCL     long  |<1               ''SCL = 0000_0000_0000_0000_0000_0000_0000_0010 
CS      long  |<2               ''CS  = 0000_0000_0000_0000_0000_0000_0000_0100
OS      long  |<3
Mask    long  |<9               ''Mask= 0000_0000_0000_0000_0000_0010_0000_0000 = bitcount being measured - 1 (10bitADC-1=9)
Tsucs   long  30                ''Tsucs = 100ns min = 8  clocks @80Mhz
Thi     long  40                ''Thi   = 160ns min = 13 clocks @80Mhz
Tlo     long  40                ''Tlo   = 160ns min = 13 clocks @80Mhz
Tcsh    long  50               ''Tsch  = 350ns min = 28 clocks @80Mhz
Masktmp       res 1             ''Masktmp = volatile value for Mask
Bitcnt        res 1             ''Bitcnt  = volatile value for a bit counter
wcnt          res 1             ''wcnt    = waitcnt variable 
input         res 1             ''input   = variable to store I/O pin states 
ADCtemp       res 1             ''ADCtemp = variable to store converted ADC value, gets written to Main Memory

-GRIMM-

That helps alot, thanks.

I've been playing with that code for a few days. I understand it is an adaptation onto the MCP3001 file. That file itself works (it outputs in bursts, is that how it measures as well?)

The file you posted works (modified it to write cnt along with the values). When i run it as it all the values are 1007, and when I change the pins to line up with the gyro, it writes cnt correectly, but then random numbers all over the board for the adc value. They vary completely, from a single digit 0, two digits, three digits, four digits.

Im not quite sure why?

Thanks

-Steven

Kye

Yeah, sorry about my demo. It's more like a feature test program of everything. Try this out, http://forums.parallax.com/attachment.php?attachmentid=79484&d=1300756068.

It's a bit more commented.

Thanks,

-GRIMM-

Examples:

Code:

PUB Start | writ,v

  text.start(12)
  waitcnt (80_000_000 + cnt)
  cognew (@shiftin, @ADCval)
  sd.mount_explicit(SD_DO, SD_CLK, SD_DI, SD_CS)
  sd.popen(string("DataFile.txt"),"w")

  repeat 1000
  
    writ := dec_to_string(cnt)
    text.str(@output_str)
    write_to_SD(@output_str,writ)

    write_to_SD(string(","),1)
    
    writ := dec_to_string(ADCval)
    text.str(@output_str)
    write_to_SD(@output_str,writ)

    write_to_SD(string(13,10),2) 
    sd.pflush

  sd.pflush
  sd.pclose
  sd.unmount
  text.str(string("all finished "))

Saved Results (Excerpt):

-1306321753,3
-1303027433,0
-1299322665,81
-1296054649,0
-1292374857,0
-1289120377,0
-1285447481,748
-1281800521,0
-1278340889,419
-1274847081,118
-1271392345,612
-1267873849,620
-1264399625,484
-1260889305,0
-1257419881,0
-1253893465,0
-1250438297,0
-1246961113,80


Code 2:

I added the ADC input program "ADC : "ADC_INPUT_DRIVER"" found on here to attempt to write it.

Verified it works as a standalone, but when used to write to the SD the results were:

-1329216274,999
-1325896850,996
-1322119666,998
-1318815090,1007
-1315067682,1007
-1311794642,1001
-1308081666,1007
-1304184754,1000
-1300900018,996
-1297360370,1001
-1293856066,1007


(Test output is 487)

Ill keep trying.

Thanks for the help

Perry

This problem intrigues me as an example of real data logging with an SD.

but there are several things that seem to be overlooked in questions of timing

writing small strings to the SD card is very inefficient, you need to write blocks at the size of the underlying hardware/software

binary to decimal conversion , it would be faster to dump the raw bytes to the SD. what tools are you going to use for data analysis?

the propeller clocks rolls over in a few seconds , using the "cnt" is not very informative , you probably need an external reference clock

SD cards have variable write times depending on when they need to erase a new block, size of SD card is a great factor in this. Also there are so called high speed products. I am using a generic 1Gb micro SD in an adapter.

I have made another program to address some of these problems.

I don't have an MCP3001 so I am using sigma/delta to do the a/d conversion, my experience with these has been very good, you should expect that the conversion will provide values at their rated conversion time and only need to read them when you want the most recent value.

I running the new version for full 60 minutes, there are few pauses so far....

OK all done, but the buzzing has made me tired will review the results to-morrow.

RS_Jim

Grimm,
Jonnymac has posted a program to obex called Etimer that will give you 1 millisecond resolution. You could tie that together with your data from your adc and write the data to the card once per second using a circular buffer to store the data in hub ram until you write it to the card.
RS_Jim

-GRIMM-

Thanks for the replies.

Perry, thanks. I will run that program when I get back to my computer.

RS_Jim. I will use that program to record time instead of cnt. I did not know one was available!