Shop OBEX P1 Docs P2 Docs Learn Events
buffer 10 seconds of data until trigger then record to SD? — Parallax Forums

buffer 10 seconds of data until trigger then record to SD?

Dwayne DibbleyDwayne Dibbley Posts: 63
edited 2011-08-10 10:02 in Propeller 1
Is it possible for the propeller to have a 10 sec buffer of data, that when a trigger is tripped this 10 sec + live data is recorded to a sd card?

idea is for a gps data logger, that records your track data only when moving over a certain speed. then when you stop the log is closed

eg

gps incoming data ,add this live data to a 10 sec buffer purging old data out when new is stored then when speed > 30 mph record the 10sec buffer + all live data to a file then when speed drops to 0 for x secs close log file?

Thanks
«1

Comments

  • Mike GMike G Posts: 2,702
    edited 2011-08-08 06:05
    Sure, the Prop has 32k of RAM minus your program. How much buffer memory do you need?
  • Dwayne DibbleyDwayne Dibbley Posts: 63
    edited 2011-08-08 06:54
    here's a sample with 10 secs of data txt file comes out at 15kb not sure if this would represent the same size in memory?

    LOG04.txt

    Thanks
  • Mike GMike G Posts: 2,702
    edited 2011-08-08 07:05
    It should work as long as your program is 32-15 = 17kb or less. You could also rethink the 10 sec buffer.
  • mindrobotsmindrobots Posts: 6,506
    edited 2011-08-08 07:23
    Since you are going less than 30mph before your SD logging starts, your 10 second buffer could at most contain about 440feet worth of data. Depending on what type of accuracy you need, you can certainly play with your 10 second buffer. I'm not sure how many samples are in your 15K sample but by playing with teh sample rate and your needed accuracy, you may be able to have a lot more data in buffer before your start logging to the SD.
  • PublisonPublison Posts: 12,366
    edited 2011-08-08 07:42
    Take a look at this Data Logger:

    http://forums.parallax.com/showthread.php?104049

    I built this on a Proto Board with all Parallax parts.

    Since most GPS's will output once a second, you have plenty of time to write info to an SD card once a second. No need to buffer. I have used this program to log 300+ miles at a time.

    You just have to do a little logic to extract the speed out of the $GPRMC sentence to start the logging. The program puts the header and footer for Google Earth when it opens and closes the SD file.
  • Dwayne DibbleyDwayne Dibbley Posts: 63
    edited 2011-08-08 10:25
    mindrobots wrote: »
    Since you are going less than 30mph before your SD logging starts, your 10 second buffer could at most contain about 440feet worth of data. Depending on what type of accuracy you need, you can certainly play with your 10 second buffer. I'm not sure how many samples are in your 15K sample but by playing with teh sample rate and your needed accuracy, you may be able to have a lot more data in buffer before your start logging to the SD.

    i need at least 10hz output rate, so i can overlay my videos with the data like my example here:
    this data was recorded on the sparkfun logomatic ( as all i had at the time :) )
  • pgbpsupgbpsu Posts: 460
    edited 2011-08-08 11:15
    Hi Dwayne-

    The version of FSRW (V2.6 the SD card object) I checked uses just over 4Kbytes of memory. 32k - 1k (for FSRW) - 15k (buffer) you'll need to fit you stuff into remaining ~16Kbytes. Depending on what else you need to do that should be doable.
  • Dwayne DibbleyDwayne Dibbley Posts: 63
    edited 2011-08-08 13:31
    I dont think i will have enough room in the room as i will need some float math / gps driver / lcd driver to fit in :( can you add external memory and can you access it fast enough?

    Thanks
  • Mike GMike G Posts: 2,702
    edited 2011-08-08 13:49
    @Dwayne Dibbley, you don't need a 15k buffer. You simply need to work out a process to serialize data to RAM then to the SD card. One processes buffers data while another reads the next byte.
  • Dwayne DibbleyDwayne Dibbley Posts: 63
    edited 2011-08-09 02:54
    So would an array be the best solution for this like:

    index := 1
    repeat
    add array(index) "text(index)" to array ' at index 1
    index := index + 1
    if index := 4 then index := 1

    only guesses the code to try an explain

    then when trigger is triggered :) dump the array from the current index + 1 to current index to sd card?

    or maybe another spin way ?

    Thanks
  • pgbpsupgbpsu Posts: 460
    edited 2011-08-09 08:36
    Hi Dwayne-

    Without knowing what else you needed to fit, I thought you'd have enough space. What about cutting down the NMEA messages. Do you really use the checksum? Couldn't you use the position, speed, etc as numbers rather than strings? Do you need the time twice? I think you could probably get all the information you need from the messages into half the current size. That sounds much simpler than adding additional memory.

    What speed is your GPS serial port? Writing to the SD card will almost certainly be faster, however you have the overhead of opening and closing files and there is some jitter in the time it takes to dump a buffer to SD card.

    I would probably start by setting up as large an array as possible. Cog A fills the buffer. Cog B dumps the buffer. When cog A has filled at least 1/2 the buffer it alerts cog B. It continues filling the second half of the array. Cog B dumps the 1st half, then waits for the second half to fill. You continue ping-ponging back and forth. If you want to squeeze a bit more performance from this setup you could use a circular buffer. It's a bit more complicated but give you more time to write to disk.

    Where did the 10 second buffer requirement come from? I suspect two 2-3 second buffers would be more than enough to keep up with things.

    Or you could go Mike G's route and skip the buffer.
    pb
  • pgbpsupgbpsu Posts: 460
    edited 2011-08-09 08:39
    I see now where the 10 seconds come from. You want the 10 seconds of data just prior to hitting 30mph. Is that correct?
  • Dwayne DibbleyDwayne Dibbley Posts: 63
    edited 2011-08-09 10:04
    pgbpsu wrote: »
    I see now where the 10 seconds come from. You want the 10 seconds of data just prior to hitting 30mph. Is that correct?

    Yes that's right so I get all the data from a standing start automagicly recorded, instead of recording all the time and starting stopping manually. so i would need some sort of circular logging for that 10 sec buffer to loop round.

    And yes I could simplify the NMEA strings but the current video overlay software imports NMEA strings, and I would need to additionally ( later ) create an import template for the new format.

    Thanks
  • Mike GMike G Posts: 2,702
    edited 2011-08-09 10:15
    Dwayne Dibbley, Take a look at FullDuplexSerial for buffer ideas.
  • pgbpsupgbpsu Posts: 460
    edited 2011-08-09 10:32
    Dwayne-

    I understand the problem better now. One could cut down the size of the NMEA strings before storing and then "grow" them back to just before writing them, although that's starting to sound cumbersome. Redoing my math from earlier. FSRW needs about 4Kbytes of memory. FloatMath and FloatString need an additional 1.6Kbytes. A 10 second circular buffer would need (80chars*2messages*10hz*10seconds) = 16Kbytes. 32Kbytes-16Kbytes-4Kbytes-1.6Kbytes= 10.4Kbytes.

    How large is your LCD driver? And what else do you use the gps for? Do you make use of it in the program, or do you just need to grab GPS serial data and spit it to the sd card? If the latter you don't really need a GPS driver. One of the serial port drivers would do it.

    I still believe there's a way to do this, if I understand all this program has to do, without resorting to external memory.

    p
  • Duane DegnDuane Degn Posts: 10,588
    edited 2011-08-09 11:24
    Dwayne, I agree with Mike G. Take a look at the way the circular buffer is done in FullDuplexSerial.

    I think it was Phil who told someone to "Just learn to do it the right way" or something like that when it came to a circular buffer. I had been avoiding the subject myself but I took Phil's advice (though not directed to me) and learned to use a circular buffer. This is really the only "right" alternative for your need.

    I think it was Phil who suggested the Wikipedia article on circular buffers. I learned a lot from it.

    I also agree with pb. You should be able to reduce the data needed during those 10 minutes by changing the way the data is stored.

    If you decide to use external memory, you'll want a kind that doesn't "wear out." SRAM is a good choice.

    Here's my attempt at a DIY SRAM module.
  • Dwayne DibbleyDwayne Dibbley Posts: 63
    edited 2011-08-09 11:36
    pgbpsu wrote: »
    Dwayne-

    I understand the problem better now. One could cut down the size of the NMEA strings before storing and then "grow" them back to just before writing them, although that's starting to sound cumbersome. Redoing my math from earlier. FSRW needs about 4Kbytes of memory. FloatMath and FloatString need an additional 1.6Kbytes. A 10 second circular buffer would need (80chars*2messages*10hz*10seconds) = 16Kbytes. 32Kbytes-16Kbytes-4Kbytes-1.6Kbytes= 10.4Kbytes.

    How large is your LCD driver? And what else do you use the gps for? Do you make use of it in the program, or do you just need to grab GPS serial data and spit it to the sd card? If the latter you don't really need a GPS driver. One of the serial port drivers would do it.

    I still believe there's a way to do this, if I understand all this program has to do, without resorting to external memory.

    p

    not got as far as the lcd driver yet :). gps will also be used for a trip meter ( currently just using math with the speed in knots from the gps ) this will hopefully displayed on the lcd when i get that far. apart from that in the future i hope to add accelerometers to the log.

    im currently breadboarding with a quickstart board is that suppose to have a 64k eeprom?

    Thanks again
  • Duane DegnDuane Degn Posts: 10,588
    edited 2011-08-09 11:57
    Dwayne, (man, it hard to type it that way)

    Most of the QuickStart boards have 64K EEPROMs. The first batch made have 32K EEPROMs.

    Here's a program you can use to test your board.

    If it's 64K every other LED will flash on and off every four seconds.

    If it's 32K all the LEDs will flash on and off every two seconds.

    This is not the purpose of this program, but it should to the trick for you.

    Duane (Now that flows from the fingers.)

    Edit: I was correct about the number of LEDs but not the druation of the flash.
  • pgbpsupgbpsu Posts: 460
    edited 2011-08-09 12:07
    Dwayne-

    I don't think the EEPROM space will be much use. The read/write nature of this will kill it off too quickly in my opinion.

    I would actually suggest using the quickstart board to wire up the SD card and GPS. Use the 4-serial ports in one object to talk to the GPS and send debug messages to the terminal/PC. Get that code working and see if you can log data the way you want to. When that works you can go back and wire up the LCD.

    Have you picked an LCD? If you haven't I'd suggest a unit that accepts serial commands. You can then use the 4-serial-ports in one object, to read the GPS and write to the LCD. Doing this should still get you under the code limit.

    Duane is right that learning to do a circular buffer correctly is worth the effort. If you have one cog feeding it and one cog emptying it you need to be careful that you don't overwrite data that haven't been dumped yet, handle buffer overrun, etc. It can be done. I believe the FSRW object writes 512 bytes at a time to the SD card so you may want to use a circular buffer and anytime it contains more than 512bytes you write that chunk. Most of the time your buffer will be essentially empty. At start-up when you are trying to collect the 10 seconds of stationary data and when the SD card is opening and close files, you'll get a small amount of filling, but generally the SD card is more than fast enough to dump serial port data.

    In fact, I would probably go with a 12 second circular buffer. You'll need a little bit of time (storage) to handle incoming data while you are writing the 10 seconds of stationary data. Once you start writing, you'll quickly drain the buffer. But that's alright.
  • Duane DegnDuane Degn Posts: 10,588
    edited 2011-08-09 12:23
    Dwayne,

    Since pb mentioned the four port serial object, I thought I'd mention my "improved" (in my opinion) versions.

    I doubled the RX buffer from 64 bytes to 128 bytes without increasing the program size more than a few longs. I just rearranged the "circular buffers" so they reused space in the cog image. It was my attempt to understand Tim Moore's driver when I finally learned to use a circular buffer.

    The four port driver (Tim deserves the credit for it) has really help me in a lot of my projects. You might want to look through the thread with my versions. Besides the 128 byte RX buffer version there is a 512 byte RX buffer version.

    Duane
  • Dwayne DibbleyDwayne Dibbley Posts: 63
    edited 2011-08-09 12:39
    Duane Degn wrote: »
    Dwayne, (man, it hard to type it that way)

    Most of the QuickStart boards have 64K EEPROMs. The first batch made have 32K EEPROMs.

    Here's a program you can use to test your board.

    If it's 64K every other LED will flash on and off every four seconds.

    If it's 32K all the LEDs will flash on and off every two seconds.

    This is not the purpose of this program, but it should to the trick for you.

    Duane (Now that flows from the fingers.)

    Edit: I was correct about the number of LEDs but not the druation of the flash.

    64K quickstart is what i have :) but as pgbpsu says eeprom would not be the best place from the buffer.

    of to learn some more on the circular buffer

    Thanks
  • Dwayne DibbleyDwayne Dibbley Posts: 63
    edited 2011-08-10 03:50
    does this look like that same idea regarding circular buffering? http://forums.parallax.com/showthread.php?96863-Circular-buffer-for-flight-recording
  • Duane DegnDuane Degn Posts: 10,588
    edited 2011-08-10 05:19
    Dwayne,

    Yes, that thread does discuss a circular buffer.

    I think it would be good to figure out if you'll need some external RAM. External RAM isn't so hard (especially since I know how to do it now). I think one 23K256 SRAM would be plenty of memory (32KB).

    The driver I wrote can read and write to the external chip faster than 100KB/s. There are faster drivers than the one I wrote (I wrote mine as a learning exercise). It looks like you need to write at about 1.5KB/s so the SRAM would more than meet you needs.

    The SRAM I used cost $1.14 each (in quantity of 25).

    15KB is lot of hub RAM to give up for a buffer.

    If you think you can get by with hub RAM, that would probably be an easier solution. But hey, wouldn't it be cool to have the option of using external memory if you needed it?

    Duane
  • Mike GMike G Posts: 2,702
    edited 2011-08-10 06:42
    Why not start the SD card write on a trigger like a change in acceleration. It seems kinda silly to wire up RAM or EEPROM when you can just write directly to the SD card. Is the SD card too slow?
  • Duane DegnDuane Degn Posts: 10,588
    edited 2011-08-10 06:52
    He want the 10 seconds of data before the trigger also.
  • Dwayne DibbleyDwayne Dibbley Posts: 63
    edited 2011-08-10 07:23
    Yes DD is correct i still need the 10 sec data as well as the live stream once triggered. eg 10 sec recording loop then when triggered (eg 30mph) start log file on sd card dump the 10sec loop ( in the correct order oldest to newest ) and add all the live data until speed drops below x mph for x seconds the close file.

    still haven't a clue how a circular buffer works yet

    DD
  • Mike GMike G Posts: 2,702
    edited 2011-08-10 08:14
    You want 10 seconds of data at zero acceleration? So the device is at a constant velocity and not standing still? Well I still don;t see why you don't write directly to the SD card. I must be missing something.
  • Duane DegnDuane Degn Posts: 10,588
    edited 2011-08-10 08:48
    DD,

    With a circular buffer you have variable "head" keeping track of where new data should be written. You have a second variable "tail" keeping track of where the data should be removed.

    Once the head reaches the last position in the buffer it jumps to the first position (0).

    In FullDuplexSerial, if the head catches the tail the method waits for room in the buffer before proceeding. In your case you'll want the tail to move to make room for the head.

    If you use external memory and need about 15,000KB of data in the buffer, you might want to use 16,384 byte buffer. 16,384 is two to the power of 14. There are tricks you can do with power of 2 numbers that make them useful for buffers.
    head &= $3FFF
    

    if easier to write than
    if head > 16383
        head := 0
    

    BTW 16,383 = $3FFF.

    I like to use the hex notation when using the bitwise AND "&" (I can go from hex to binary a lot easier than decimal to binary in my brain).

    DD
  • Duane DegnDuane Degn Posts: 10,588
    edited 2011-08-10 08:52
    Mike G, the trigger isn't a change in acceleration. It's a speed and DD wants 10 seconds of information from before the trigger.
  • Dwayne DibbleyDwayne Dibbley Posts: 63
    edited 2011-08-10 10:02
    Mike G wrote: »
    You want 10 seconds of data at zero acceleration? So the device is at a constant velocity and not standing still? Well I still don;t see why you don't write directly to the SD card. I must be missing something.

    speed data looks like this for example ( each digit is 1 sec and using a 3 sec buffer 123 )
    000 000 000 000 000 0112233344455566768987654321000 000 000 000 000
    
    123 123 123 123 123 123                             123 123 123 123
    -----buffering----- --record all this to sd card--- ---buffering---  
    
    

    so the above show the 3 sec buffer looping 6 times till speed =1 then record to uSD the 6th buffer (011) + all the live data and then stop logging / close file when speed trigger = 0 for 3 secs then back to buffering again

    Does that make things clearer

    DD
Sign In or Register to comment.