Copying all files from SD to USB

TCP71

Hello,
I have an SD card and a USB datalogger attached to a propeller. I store backup files on the SD card (small files, each less than 400bytes) that are identified by their date and time. For example, I would have the following files inside the root directory of the SD:

06241108.401
06241108.403
06241108.405

They would be files written on june 24th, 2011 at 8:40:XX am. These files could span many days and times.

I would like to copy every file in the root directory of the SD to a usb stick in the datalogger.

Is there an efficient way to copy all the data from the SD card directly to the USB stick without identifying each file name, opening the file, copying the data to memory, closing the file on the SD, creating the file on the stick, opening the file on the stick, dumping the data from memory to the stick, closing the file on the stick. Seems horribly slow.

I am using DataloggerSPI and SD3.01_FATEngine to access the card/stick. I am not a programmer, so the more basic the explanation the better. Thank you for any and all ideas. This forum has been a wealth of knowledge in getting my project working.

Mike Green

You'd have to duplicate much of the code in the SD card driver involved with directory lookup, opening a file, and reading a file. Then you'd have to essentially build a USB memory stick image in the Prop's memory and write it out using low level "write any sector" routines in the USB datalogger. If you were an experienced programmer, this might be a reasonable project although I'd probably recommend using a different file format so you'd essentially save your information in something like a tab-delimited or comma-delimited value format (CSV) compatible with Excel including the date and time. It would be much faster to copy since you'd only have one file to copy and could easily be loaded into Excel or some database programs on a PC for later analysis.

TCP71

Ok, so I need to find each file, open and read it into a "super" file (CSV or such), then dump it in one swoop onto the stick. A couple of questions about that:

1: I don't know what exactly the filenames are ahead of time. Should I search for every possible permutation of numbers that make up the 8:3 filenames and whenever one is recognized, pull the data out of it, or is there some way to scroll through each file sequentially on the SD card and move onto the next once I've got the data without pulling data twice? If I had no concerns about the time it took to transfer all the data to the stick would the find/open/copy/close on SD, then create/open/write/close on stick for each file be a poor method?

2: My SD card is 2Gig. Each file is less than 1k in size, but takes up 32K on the disk. Not a problem as the storage of 62500 files are possible with each at 32k. If i were to have 62500 files and each one is 400bytes, there is the theoretical possibility of having 25Meg of actual file data...too big for storage in memory. I would have to create multiple file of a smaller size, I suppose.

3: Is there any way to look at the data on the card without dealing with a file structure? Just transfering all bytes within the used section of the SD card directly to the stick without change or actually looking at the data?

Here's the data within one file (named 06241116.340):

5059.9536,N,11359.7107,W,+01027.2,M,1,03,03.1,000.16,139.8
Hit=06242011,163400.0677965
Hit=06242011,163401.4775487
Hit=06242011,163402.9999477
Acquisition Complete.

Any thoughts appreciated.

Ariba

Why you don't create only one file and append the new data instead of creating everytime a new file. Then you need to copy only this one file.
You need some indentifier in the big file to sparate the single data portions, or perhaps you can use just a fixed datablock size.

Andy

Mike G

1: I don't know what exactly the filenames are ahead of time. Should I search for every possible permutation of numbers that make up the 8:3 filenames and whenever one is recognized, pull the data out of it, or is there some way to scroll through each file sequentially on the SD card and move onto the next once I've got the data without pulling data twice? If I had no concerns about the time it took to transfer all the data to the stick would the find/open/copy/close on SD, then create/open/write/close on stick for each file be a poor method?

Yes, use the list commands in the FAT 3.0 driver.
http://forums.parallax.com/showthread.php?131598-New-AppNote-Posted-FAT16-FAT32-Full-File-System-Driver

2: My SD card is 2Gig. Each file is less than 1k in size, but takes up 32K on the disk. Not a problem as the storage of 62500 files are possible with each at 32k. If i were to have 62500 files and each one is 400bytes, there is the theoretical possibility of having 25Meg of actual file data...too big for storage in memory. I would have to create multiple file of a smaller size, I suppose.

Set aside some RAM, say a 2K buffer. Get total file size: Read 2k from the source -> 2K Buffer -> Append 2k to the destination file -> Decrement total bytes to read by 2k -> repeat

3: Is there any way to look at the data on the card without dealing with a file structure? Just transfering all bytes within the used section of the SD card directly to the stick without change or actually looking at the data?

Sure, similar to the above process.

TCP71

Actually, a lot to think about. Maybe the creation of a single file on the SD card as data comes in, in a CSV format would be easiest. Then a simple transfer of that file to the stick. Maybe listing all the files, reading their names and data and then appending to a large file on the stick. Maybe writing each individual file. Thanks for all the suggestions. I must contemplate.

Cluso99

Not sure if this helps or not. You could remove the SD card (or microSD) and plug it into a USB card reader and use the pc to do the copying.

Tubular

Also DOS (FAT16) used to have a maximum number of files that could be in the root directory.

I'm not sure whether thats still an issue or not, just putting it out there in case you see issues