Advice needed for logging data onto a USB thumbdrive
Hello all,
I'm interested in·logging data onto a USB thumbdrive (or something similar).· On the Object Exchange and in my search through the forum I've seen a couple of things that look like possibilities.· But I thought I'd toss up the question to the forum to hear people discuss the pros and cons of each or enlighten us all on anything new.·
For my application, on each measurement, I'm hoping to store the date, time down to·5 second accuracy, and roughly 15 variables (about·5 temperature readings and·10 counter channels, each counter variable needing to hold a value·up to·about 1 million events).· Data will be written about every 5 to 10 seconds and will run for maybe 3 to 6 months.·· That works out to about ·3.1 million data records, I think.· I'm not sure how many gigabytes of storage that would require.·
Advice, comments or suggestions would be greatly appreciated.
Mark
I'm interested in·logging data onto a USB thumbdrive (or something similar).· On the Object Exchange and in my search through the forum I've seen a couple of things that look like possibilities.· But I thought I'd toss up the question to the forum to hear people discuss the pros and cons of each or enlighten us all on anything new.·
For my application, on each measurement, I'm hoping to store the date, time down to·5 second accuracy, and roughly 15 variables (about·5 temperature readings and·10 counter channels, each counter variable needing to hold a value·up to·about 1 million events).· Data will be written about every 5 to 10 seconds and will run for maybe 3 to 6 months.·· That works out to about ·3.1 million data records, I think.· I'm not sure how many gigabytes of storage that would require.·
Advice, comments or suggestions would be greatly appreciated.
Mark
Comments
Depending on the details of the actual data being recorded, you may do better if you record it as binary rather than character strings. You'll have to work it out both ways and see also the issues around processing it. It's often easier to create the data in a form that can be input directly to a spreadsheet for example.
thanks,
Mark
You could use external EEPROM, but those are very limited in size. However, they use less mA than the datalogger and are smaller in size, if that makes a difference. I think you best route is the datalogger and an 100GB+ external hard drive. That way you don't have to worry about the size of the files or how long you forget between data download sessions. Just be prepared for a struggle programming the datalogger.
For me, electrical power is not a problem. But do tell: can off-the-shelf (I'm talking Walmart here) external hard drives simply plug into the Parallax Datalogger and give me 100Gbytes to do as I please?
thanks for sharing,
Mark
in the object repository? 2GB of data in a standard FAT16 format should be all you
need, and you can swap out the cards whenever you want to look at the data.
At the rate you're collecting the data, bandwidth is clearly not an issue. And this
just might be the overall cheapest and simplest route.
Hey, I'm open to all suggestions. Is this SD socket do-hickey something you've worked with yourself? Who sells it and where can I learn more?
a thousand thanks,
Mark
but the Windows format program may not do it for you, and the resulting disk may not be compatible with all devices.
Mark
will *not* support SDHD, and it is unlikely they will anytime soon (unless someone can pry out an SDHD spec so I can find out
how to drive those things). 2GB cards are pretty cheap right now, cheaper than the socket, probably.
By default it is 512 "entries", be they files or directories.. When you go to VFAT this becomes somewhat distorted by LFN entries.. but none the less, it's 512.
FAT-16 with 8.3 filenames should allow you the full 512..
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Pull my finger!
that are created by most tools. But like I said, the format does permit this to be extended, and the fsrw code *should* support such extensions.
If anyone really needs more than about 500 files, let me know and I'll test a larger root directory. There's not a whole lot of downside; the
space requirements of a larger root directory on an SD card is pretty minimal. Of course, file search takes time linear in the number of files
in the directory (at least for fsrw) so too many files and you'll slow down file opens.
There is a maximum limit of about 32K files due to the FAT itself (each file having at least one sector); with some effort this could be raised to
nearly 64K by formatting a 2GB device with 32K sectors.
Mark
www.sdcard.org/about/memory_card/pls/
www.compuphase.com/mbr_fat.htm
elm-chan.org/docs/mmc/mmc_e.html
Basically it seems you just have to extend the SD initialization a bit. If the card _is_ SDHC then you pass in a block address instead of a byte address, where the block-size is fixed to 512.
The part where I'm running into trouble is Windows will not let me put multiple partitions on a removable drive. So, even if I made the modifications to read SDHC cards, I still have a 2GB limitation because of the FAT16 limits. I could easily use Linux to do the partitioning, then upgrade the SDSPI* drivers to mount multiple partitions, but I'm not sure any of my end users would want to duplicate the effort, and I'm not sure Windows will read the 2nd partition even if it exists (on removable media).
(Moving to FAT32 would be nice, but I want to avoid all licensing issues, not to mention we already have very nice FAT16 routines working perfectly
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
lonesock
Piranha are people too.
anywhere.
Are there licensing issues for making an SPI SDHC host device? (My understanding is that using the actual SD protocol requires licensing, but
using the SPI protocol to talk to the SD cards does not. This in spite of the fact that the two "protocols" have more in common than not. Have
they continued this "exemption" for SDHC, or are *all* SDHC host devices required to pay licensing fees? Any sort of official or semi-official
word on this would be nice.)