Datalogger with memory stick

Mike87

Hi,

I'm trying to do a simple datalogger for an accelerometer and I want to log X, Y and Z data on my usb key at 20 Hz. I'm using the Memory Stick Datalogger v1.1 object by Paul via uart at 9600 bps. I'm able to write my data to my usb key, but only at lower speed (4 hz). As you can see below, my code is really simple. Will it be faster if I used the SPI protocol instead of uart? And what about an SD card? Could it be faster then a memory stick? Thanks!

  Time := cnt
  Duration := clkfreq/1000 * 50

  repeat
    USB.WriteLine(NUMBER.ToStr(~XYZData,%1010))
    waitcnt(Time += Duration)

Mike87

Nobody ever tried to write at 20 samples per second with a memory stick?

SRLM

Hmmm. I thought I replied to this thread. I guess not. Anyway, in my ghost reply I suggested that you take a look at a datalogger object that I posted in the OBEX. It samples as fast as possible, and it appears (through my testing) to be limited by the flash drive, so it should be theoretically the fastest possible that you can log something to a memory stick. Anyway, it's called Autodatalogger and all the documentation is bundled with it.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Powered by enthusiasm

Mike87

Thanks, I'll look at it. Is it possible to limit the samples per second or it's only as fast as possible?

SRLM

It's as fast as possible, and automatic. It records the clock so you can determine later what frequency it was recorded at, and perhaps cut out the data that you don't want.

Also, a quick glance at your code: my first guess is that your delay is too small, since 9600 bps is fairly slow you're probably running over the 50ms delay while you're still recording your numbers.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Powered by enthusiasm

Mike87

I've tried your autodatalogger and it works pretty well. I'm able to get 65 samples per second. I was wondering if the autodatalogger is "thread safe" (or cog safe). I mean, if I change the variable, and at the exact same time, the datalogger is reading the variable, will it produce an unpredictable result? I'm asking this question because during my data acquisition (2 minutes of logging) from an accelerometer, all my data were between 0 and 1023 (10 bits) except a couple (3 or 4). The wrong data were over 1023, which is impossible to coming out from my accelerometer. Thanks.

SRLM

It should be, since each cog is given in turn exclusive access to the hub, and the program reads the entire value at once. Can you post your entire code up, and I'll take a look at it? From your top file, just select File->Archive->Project and it will make a zipped up folder of all the files. Also, if you could post your data file that would help me find the problem.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Powered by enthusiasm

Mike87

Here's my code. It's still a work in progress, so I need to do some clean up in my code. I have included my data file (acqui.txt). My accelerometer wasn't moving, so the data should be stable during all acquisition long. There's some values over 1023 (line 491, 974) and sometime the value is dropping (line 789). I'll do some tests on my side to be sure that it's not the accelerometer who's giving my wrong values.

rokicki

BTW, yes, an SD card would be a *lot* faster (but if you don't need that much speed it's not a big deal).

Mike87

By "a lot faster", you mean over 100 writes per second?

rokicki

By "a lot faster" I mean up to 2MB/sec. One hundred writes per second is childplay.

You *do* need to buffer samples since the SD card can take some time to do its stuff (but the same is true for the flash datalogger).
So your acquisition cog needs to be decoupled from the cog that is writing the data to the SD card, with buffering between the two.

See fsrw2.6 from the obex.

rokicki

Not to mislead anyone, though, this is a better estimate for data logging to SD using fsrw. The burst throughput is way high enough; the main
problem is occasionally the flash card needs some time to shuffle things around. By spec, this should never be more than 500ms, but let's
call it a second (since we actually have metadata to write too). So your buffering should be sufficient for one second of data. If you are
willing to use 16K of the prop's memory for that buffering, that means 16K *bytes* per second. So if your samples have a 4-byte clock
and 4 bytes of data, that's 2K samples/sec (in binary).

If you are willing to tolerate rare (but occasional) dropouts in the data, then your rate can be substantially increased, since we can indeed
write to the card pretty close to 2MB/sec.

If you need a higher sampling rate and cannot tolerate dropouts, the use of external buffering memory is recommended; there are a number
of such products for the propeller. To use this, your data acquisition code will store the samples in the external RAM, and the writing COG
will read them from that external RAM and store them on the SD card. Even SPI memory will work just fine; you don't need to get crazy
bandwidth. Your sample rate will then be proportional to the size of your external memory (until you start hitting the limits of SD card write
speed).

Using external memory can also lower the average power consumption of the datalogger, since the SD card itself *can* take a fair amount
of power when writing. By bursting the writes, the SD card will spend more time in a very low power state.

SRLM

I believe that I've found your problem. Your ADC function accesses the same memory location multiple times, and uses it to hold intermediary values. The fix is to use a temporary variable to hold the calculations in, then in one shot store it into the memory location (XYZData[noparse][[/noparse] i ]). As a practical note, you may want to store all the data in the array at once, at the end of the loop so that the readings are taken as closely together as possible.

PRI StartRead10BitData
  repeat
    SPI.write(SPI#MCTL, (%0110 << 4)|SPI#G_MODE)                 'Initialize the Mode Control register
    DataIn_High      := SPI.read(SPI#XOUTH)                       'repeat for X-axis
    DataIn_Low       := SPI.read(SPI#XOUTL)
    XYZData[noparse][[/noparse]0] := DataIn
    XYZData[noparse][[/noparse]0] := ~XYZData[noparse][[/noparse]0] + 511                               'Convert from -511 to 511 data to 0 to 1023 (Always positive)
    DataIn_High      := SPI.read(SPI#YOUTH)                       'repeat for Y-axis
    DataIn_Low       := SPI.read(SPI#YOUTL)
    XYZData := DataIn
    XYZData := ~XYZData + 511                               'Convert from -511 to 511 data to 0 to 1023 (Always positive)
    DataIn_High      := SPI.read(SPI#ZOUTH)                       'and Z-axis
    DataIn_Low       := SPI.read(SPI#ZOUTL)
    XYZData := DataIn
    XYZData := ~XYZData + 511                               'Convert from -511 to 511 data to 0 to 1023 (Always positive)
    waitcnt(clkfreq/100+cnt)

@rokicki

Thanks for the information. I'm considering making a similar object for an SD card, and you've just saved me a bunch of time for when I get big gaps in the data. Interesting to note the much higher sampling rate. I went with the datalogger since I prefer flash drives, but an SD card may be another good solution.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Powered by enthusiasm

rokicki

The SD card *should* lower your total parts count, physical size, and overall power consumption while increasing
(possibly significantly) the supported data rate. Since fsrw26 supports FAT32, you can use virtually any size SD
card. Even adding an external SPI memory chip for buffering, I think the SD card solution is a clear win.

Mike87

@SRLM

You were right. I now put my data on a temp variable, and copy it in one shot and no more weird data. Thanks a lot!

And one more thing, when I use the "stop" function of your autodatalogger, should I be able to restart by only calling the "start" function? Right now, the first acquisition is working well, but when I recall the "USB.start" function, I got stuck at that command.

SRLM

Hup. I didn't really put much effort into the stop function, simply because my use for the object was to record sensor information in a robot for post crash analysis.

Anyway, as it stands when the Stop function is called the cog is stopped, but the file on the datalogger is left open and more importantly the baud is left high. The hardware fix would be to just add a transistor to the power line of the datalogger (or reset line, if it has one) and toggle that whenever you want to restart the logging. Software would be better, of course. If there is a demand for it (you), I can do some work on it this weekend and make a nicer fix for that. I've started on the fix, but there it may take me awhile without motivation.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Powered by enthusiasm