Advice needed for logging data onto a USB thumbdrive

ElectricAye

Hello all,

I'm interested in·logging data onto a USB thumbdrive (or something similar).· On the Object Exchange and in my search through the forum I've seen a couple of things that look like possibilities.· But I thought I'd toss up the question to the forum to hear people discuss the pros and cons of each or enlighten us all on anything new.·

For my application, on each measurement, I'm hoping to store the date, time down to·5 second accuracy, and roughly 15 variables (about·5 temperature readings and·10 counter channels, each counter variable needing to hold a value·up to·about 1 million events).· Data will be written about every 5 to 10 seconds and will run for maybe 3 to 6 months.·· That works out to about ·3.1 million data records, I think.· I'm not sure how many gigabytes of storage that would require.·

Advice, comments or suggestions would be greatly appreciated.

Mark

Mike Green

Do your homework! Figure out how many characters it would take for the data as if you were printing it or displaying it on a screen. That's probably the worst case for the amount of space for each measurement. If it's 3-4 million data records and each record is maybe 256 characters, that's about 1GB. A GB is a very big file. I suggest you plan on separate files for each day, grouped into subdirectories for each month. If you include the date as part of the file name, then you don't need to store that in the file itself. That can reduce the size of the files a little.

Depending on the details of the actual data being recorded, you may do better if you record it as binary rather than character strings. You'll have to work it out both ways and see also the issues around processing it. It's often easier to create the data in a form that can be input directly to a spreadsheet for example.

ElectricAye

Thanks, Mike.· Your suggestion for writing data into daily files is something I hadn't thought of.· But my main reason for posting this on the forum was to find out more about experiences people might have had with the various objects available for thumbnail dataloggers.· I posted my particular needs to provide an order of magnitude to what I have in mind.

thanks,
Mark

SRLM

I've found the Datalogger that Parallax sells to be very difficult to use (see my "Datalogger Distress" under the BS section: no answers!). Easily half of my programming time is spent debuging that one piece of hardware/software. I find invaluable for communicating data to a desktop computer, but I hate it. (Strong word, but the truth). However, once I manage to get it working it seems to work all the time, and I'm sure it would work in your situation. You can get thumb drives up to about 10GB nowdays, and you could probably hook up an external hard drive to the datalogger if you have a grid connected power source.

You could use external EEPROM, but those are very limited in size. However, they use less mA than the datalogger and are smaller in size, if that makes a difference. I think you best route is the datalogger and an 100GB+ external hard drive. That way you don't have to worry about the size of the files or how long you forget between data download sessions. Just be prepared for a struggle programming the datalogger.

ElectricAye

SRLM,

For me, electrical power is not a problem. But do tell: can off-the-shelf (I'm talking Walmart here) external hard drives simply plug into the Parallax Datalogger and give me 100Gbytes to do as I please?

thanks for sharing,
Mark

rokicki

Any reason not to just hook four wires up to a SD socket and use the SD write routines
in the object repository? 2GB of data in a standard FAT16 format should be all you
need, and you can swap out the cards whenever you want to look at the data.
At the rate you're collecting the data, bandwidth is clearly not an issue. And this
just might be the overall cheapest and simplest route.

ElectricAye

rokicki,
Hey, I'm open to all suggestions. Is this SD socket do-hickey something you've worked with yourself? Who sells it and where can I learn more?

a thousand thanks,
Mark

Mike Green

This is literally an SD card socket. SparkFun has what they call a breakout board (www.sparkfun.com/commerce/product_info.php?products_id=204) which I've used because it allows you to bring the connections to a 0.1" header. Both Rokicki's FAT file system in the Propeller Object Exchange and my own FemtoBasic which uses a modification of his FAT file system include instructions on wiring. Neither of these routines currently support subdirectories, so all the files need to be in the root directory and I think there's a limit of 64 files in that root directory.

rokicki

I'm pretty sure the limit is 512 files more or less (actually a tad less, but not *too* much less). And I am pretty sure it can be increased at format time,
but the Windows format program may not do it for you, and the resulting disk may not be compatible with all devices.

SRLM

If you go the route of the SD card, be sure to get the biggest size card you can. Although, I think the idea of using a single file for each day is a good one. That way, if something bad were to happen while writting data to the file, you would only lose a day, not three months. Anyway, an external hard drive should be plug and play. As far as my computer is concerned, a 160GB desktop plugged in USB hard drive is the same as a 512 mb stick. However, I haven't tested this with the datalogger (160GB on a robot?!). As soon as I get my code working, I'll plug it in and give it a whirl. I expect that it will work the same. If you need a quick answer, drop the parallax tech support and ask them. They're really good at helping out.

ElectricAye

I just skimmed the article about SD cards on wikipedia. Apparently there are various larger size cards (greater than 2Gigabytes) ) on the market that, though physically compatible with older systems, are not compatible with the older software that drives them. FAT vs. FAT32? This seems to be a source of consumer confusion. Anyone know what the Propeller SD card slot product is limited to in terms of storage? I saw a FAT16 on the Object Exchange but I don't know enough about files, etc. to judge what is the capacity of this. I probably don't need anything beyond 2 Gigabytes, but... call me greedy.

Mark

Mike Green

FAT16 is normally limited to 2GB. Sectors are 512 bytes and 2^31 sectors comes out to 2GB. I believe Rokicki's drivers can handle 2^32 sectors (or 4GB), but there's no guarantee that other devices can read sectors beyond 2GB.

rokicki

I'd recommend staying below 2GB if possible. I've never tested 4GB (and 4GB ordinary SD is out-of-spec anyway). The routines
will *not* support SDHD, and it is unlikely they will anytime soon (unless someone can pry out an SDHD spec so I can find out
how to drive those things). 2GB cards are pretty cheap right now, cheaper than the socket, probably.

BradC

rokicki said...
I'm pretty sure the limit is 512 files more or less (actually a tad less, but not *too* much less). And I am pretty sure it can be increased at format time,
but the Windows format program may not do it for you, and the resulting disk may not be compatible with all devices.

By default it is 512 "entries", be they files or directories.. When you go to VFAT this becomes somewhat distorted by LFN entries.. but none the less, it's 512.
FAT-16 with 8.3 filenames should allow you the full 512..

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Pull my finger!

rokicki

Well, pretty close to the full 512, anyway; at least one entry is typically used for the volume name, and then you have the default . and .. entries
that are created by most tools. But like I said, the format does permit this to be extended, and the fsrw code *should* support such extensions.

If anyone really needs more than about 500 files, let me know and I'll test a larger root directory. There's not a whole lot of downside; the
space requirements of a larger root directory on an SD card is pretty minimal. Of course, file search takes time linear in the number of files
in the directory (at least for fsrw) so too many files and you'll slow down file opens.

There is a maximum limit of about 32K files due to the FAT itself (each file having at least one sector); with some effort this could be raised to
nearly 64K by formatting a 2GB device with 32K sectors.

ElectricAye

Okay, it looks like rokicki's system is the way to go. Thanks to everyone for helping me out.

Mark

lonesock

I've been looking into extending rokicki's routines to handle SDHC, and it should be fairly easy to do. Here are my sources:

www.sdcard.org/about/memory_card/pls/
www.compuphase.com/mbr_fat.htm
elm-chan.org/docs/mmc/mmc_e.html

Basically it seems you just have to extend the SD initialization a bit. If the card _is_ SDHC then you pass in a block address instead of a byte address, where the block-size is fixed to 512.

The part where I'm running into trouble is Windows will not let me put multiple partitions on a removable drive. So, even if I made the modifications to read SDHC cards, I still have a 2GB limitation because of the FAT16 limits. I could easily use Linux to do the partitioning, then upgrade the SDSPI* drivers to mount multiple partitions, but I'm not sure any of my end users would want to duplicate the effort, and I'm not sure Windows will read the 2nd partition even if it exists (on removable media).

(Moving to FAT32 would be nice, but I want to avoid all licensing issues, not to mention we already have very nice FAT16 routines working perfectly )

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
lonesock
Piranha are people too.

rokicki

Well, that's nice, actually. We can use that to get 4GB SDHC cards working, I believe, and indeed test that signed arithmetic isn't hurting us
anywhere.

Are there licensing issues for making an SPI SDHC host device? (My understanding is that using the actual SD protocol requires licensing, but
using the SPI protocol to talk to the SD cards does not. This in spite of the fact that the two "protocols" have more in common than not. Have
they continued this "exemption" for SDHC, or are *all* SDHC host devices required to pay licensing fees? Any sort of official or semi-official
word on this would be nice.)