Fast Datalogging: USB, SD, other?

FlyingFishFinger

Hi-
We'd like to design a CAN datalogger for our solar car. Since the maximum data rate on CAN is 1MBit / second, we want to design for this.
Since race days can literally last all day (10-12 hrs) without a long enough break for downloading data, we need to record the full day. Assuming we use an 8GB media, that gives us about 17 hours, which is nice. With this in mind, should we use a USB or SD card solution? Seems to me SD interfaces can't handle SDHC very well yet, but USB is a little more expensive...

Rafael

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
UC Berkeley '12 EECS
CalSol: Berkeley Solar Car team
www.calsol.berkeley.edu

Mike Green

It depends on the specific interface you plan to use, the specific SD card or USB memory stick, etc. You can get widely differing throughput rates depending on the manufacturer and model of memory device you plan to use let alone the interface to the memory device. Both SD cards and USB memory sticks have variable throughput since there are internal buffers and writing to flash memories can take several milliseconds.

Can the data be compressed? What's the worst case data rate (not the raw limit for CAN)?

FlyingFishFinger

Assuming we get our system working properly, we'll be at the limit. Some of our off-the-shelf devices use 1MB/sec as default rate. We're probably going to use Arduinos, but there'll be at least one Propeller (driving Ray's screen [noparse]:)[/noparse] [noparse]:)[/noparse] ) that we might use for the job as well. I was looking at Parallax's USB Data logger, for one.
Compression, no. We want the raw CAN ID's and messages (ID's may be mapped to device names, but that's about it) in plain text files.

Rafael

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
UC Berkeley '12 EECS
CalSol: Berkeley Solar Car team
www.calsol.berkeley.edu

mctrivia

you can compress without lossing data.

there are many ways to compress:

*If you are likely to get lots of sequential values then setting a symbol as repeat followed by a number of times will give good compression
*If you have a set number of symbols and some are used more often then others the hufman compression is a good way to go.
*Use numbers to refer to names
...

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
24 bit LCD Breakout Board now in. $21.99 has backlight driver and touch sensitive decoder.

mikestefoy

the CAN bit rate of 1Mbit/second is NOT the data transfer rate.

it is only possible to send 8 bytes in a single packet period !!


to send more than 8 bytes, it is necessary to break the data into 8 byte packets, then re-assemble at the other end, which of course takes time.


so with the 11 bit identifier frame, you are sending 50 bits of "wrapping" with every 64 bits (8x8) bits of data.

using a 29bit identifier frame makes that worse of course.

thus, in an ideal world, the maximum data transfer rate would be 500Kbits/second.

I would probably divide that by say 4 ( a complete guess) giving 125Kbits of data per second.

that would give 15KBYTES/second.


below is quite a good thread on the subject

http://www.electronics-related.com/usenet/design/show/20604-1.php

I am probably being conservative, but illustrating the main point that 1Mb/s CAN will only give tens of kilobytes per second.

Mike

mikestefoy

found this which you may find useful

http://instruct1.cit.cornell.edu/Courses/ee476/FinalProjects/s2008/dhp22_spg32/dhp22_spg32/index.html


Mike

FlyingFishFinger

Cool!
We don't know for sure what and how many messages we'll be sending yet, as that's a software implementation at least 6 months away. We'd like to build the hardware such that it's adaptable to whatever throughput we end up using. I'll consult with our electrical lead, but the logger project you mentioned should be very helpful[noparse]:)[/noparse]
Thanks!

Rafael

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
UC Berkeley '12 EECS
CalSol: Berkeley Solar Car team
www.calsol.berkeley.edu
KJ6AWU