Problem writing more than 500 files to my SD card - What am I doing wrong?

ElectricAye

Hi all,
I've been running a test to see how fast my SD card can open a new file, write a few bytes, and close up that file and move on to a new file. Everything seems to work great, except the card won't accept more than about 500 files. It writes up to about 498 and then stops. It's a Lexar 2GB regular SD card and I am using it right out of the box. I checked it with WindowsXP and saw that it was formatted for FAT16, so I thought I'd be able to write thousands of files at 32Kb, not 500 files at 32Kb. When I look at the card on my iMac, it says the card is formatted in FAT16 and has used about 17Mbytes, but still has 1.87Gbytes available. What am I doing wrong? Was I supposed to re-format it some special way?

Oh woe is me....

Sapieha

Hi ElectricAye.

You are not doing wrong nothing.

FAT 16 ROT map have only 512 places in its directory.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nothing is impossible, there are only different degrees of difficulty.

Sapieha

Mike Green

I think the root directory, as normally formatted, will only hold about 512 files (with a few set aside for special purposes). Rokicki posted a forum message a few weeks ago about this issue. I've started working on an extended FAT file system object that will handle subdirectories, random file positioning, and multiple open files, but it's kind of stalled out for now and may take a couple of months to get to where it's usable.

Sapieha

Hi Mike Green

It was very fine news.
Yes FAT16 have that limitation. It is old thinking that MS have (Big enowgh).

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nothing is impossible, there are only different degrees of difficulty.

Sapieha

ElectricAye

Thanks everyone, especially for answering my question so insanely quickly! wow!
I'm depressed that I'm limited to only 512-ish files, but I'll recover soon enough.


And Mike, your way of showing me how to increment file names works wonders. After I get it polished up, I'll try to post it for others to use.

Sapieha

Hi ElectricAye.

One sugestion.
If You not needs old files edit You spin. to test if no place and if not delate one oldest.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nothing is impossible, there are only different degrees of difficulty.

Sapieha

rokicki

What kind of machine are you using to format your SD card? Depending on the platform you probably can
format the card to support many thousands of files. Also, how many files would you like?

Oldbitcollector (Jeff)

@Mike:

Does your code support successful reading/writing from a folder yet?

This would allow him to get past the 512ish file limit on the root directory of his SD.

OBC

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
New to the Propeller?

Getting started with a Propeller Protoboard?
Check out: Introduction to the Proboard & Propeller Cookbook 1.4
Updates to the Cookbook are now posted to: Propeller.warrantyvoid.us
Got an SD card connected? - PropDOS

Post Edited (Oldbitcollector) : 9/8/2008 6:48:33 PM GMT

Oldbitcollector (Jeff)

@ElectricAye:

Is is possible for you to "Append" using the "a" command to an existing file so that you can write more to each file?

I'm not sure how you are splitting your data, but if you had to split by hour/day perhaps just adding a
comma delimited field that indicated a time stamp or date stamp would allow you to store more data in
a single file and still easily locate what you are looking for.

OBC

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
New to the Propeller?

Getting started with a Propeller Protoboard?
Check out: Introduction to the Proboard & Propeller Cookbook 1.4
Updates to the Cookbook are now posted to: Propeller.warrantyvoid.us
Got an SD card connected? - PropDOS

ElectricAye

To rokicki: It's not that I need lots of files. Instead, I was dreaming of actually getting 1.8 Gbytes onto my so-called "2Gbyte" SD card. When I saw that each file was in a block of 32Kbytes, I figured I would be able to create thousands of files (to add up to 1.8 GBytes). I had not understood the problem of FAT16 not supporting subdirectories. I had heard you guys talk about it, but it didn't register in my brain. To answer your other question: I used the SD card right out of the package. With both Windows XP on an HP and with Mac OS 10 on my iMac I checked that the card already had FAT16 formatting, shrugged, and moved on. I did not alter its formatting in any way that I'm aware of.

To OBC: my SD card came formatted so that each file I create using the SD object (from the Obex) makes the file 32Kbytes whether it holds 1 byte of data or 31Kbytes of data. I format the data as comma separated variables, and I give the files the extension ".csv" . Part of every data record is a time and date stamp. Each file has a name that looks something like AAB00076.csv AAZ00034.csv etc. Once I someday read all of the data into an Excel spreadsheet, it will be sorted by time and date.

Mind you, not getting 1.8 Gbytes out of my SD card is not a panic situation; it was more like a disappointment. It just means I have to be "Johnny on the Spot" whenever it's time to swap out a card. I'm just thrilled beyond belief that, with the help of everyone here, especially rokicki and Mike "I will teach you how to fish so you can eat forever" Green, I was able to get this contraption up and running in less than a hundred years. <He falls to his knees and weeps with joy....>

thanks,
Mark

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
It might be the Information Age but the Eon of Ignorance has yet to end.

rokicki

With the existing code you can easily put 1.8GB into a *single file* if you want. There's no limit on the file size but the
capacity of the card (and the 2GB partition limit for a FAT16 partition). A file can use as many blocks as it wants.

A reasonable split might be to put 8MB in each of 256 files. That won't stress anything too much.

Note that I do recommend the use of 64K blocks, not 32K blocks. This does increase internal fragmentation a tad,
but if you are limited to ~500 files, that won't be much of an issue (wasting at most 64K*512 or about 32MB of a
2GB card is less than 2%.)

Congratulations on getting things to work!

ElectricAye

rokicki said...

A reasonable split might be to put 8MB in each of 256 files. That won't stress anything too much.

Rokicki, let me make sure I'm hearing you correctly: are you saying that I could actually format the SD card in some way and/or change the SD object code to get the full 1.8Gbytes out of a single card??? If so, that would be awesome! But do tell: where do I find details on how to do this? Do I reformat the card? Change the Obex object code? You have me gasping on the edge of my chair, here.

Mike Green

You don't have to format the card differently or change the SD card object at all.· What you have to do is to make the files larger by writing more data to each file.· The SD card routines can add onto the end of an existing file using "a" (append) mode instead of "r" (read) or "w" (write).· The first time you use a file name, you have to use "w" mode.· Every time after that you have to use "a" mode.· The new data gets written at the end of the existing data.· When the file is read on the PC, it just looks like one long file.· There's no delimiter between sections that were written separately (unless you put it there yourself).

ElectricAye

Thanks, Mike, that's great news! But I have another question that might seem kinda dumb: in order to fill a file beyond its (default?) size of 32Kbytes, MUST I go into "append" mode after creating the file? or can I simply create it and from the get-go fill it beyond 32Kbytes using the "write" mode?

thanks so much,
Mark

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
It might be the Information Age but the Eon of Ignorance has yet to end.

Mike Green

You can keep writing until you run out of SD card space. Once you use up the cluster (32K or 64K typicall), the SD card routines allocate another cluster from the available space and links them together using the FAT (part of the directory space).

ElectricAye

Mike, that's great! So, insofar as using an SD card for storing lots of data, a 2Gbyte card REALLY can store 1.8 Gbytes. I was blind but now I see. For some reason I had thought that the cluster was the limit to the size of the file. But the mere name, File Allocation Table, now makes sense to me. Once again, Mike, you have saved the day! Not only have you saved the day, but if I can really get 1.8 Gbytes on an SD card, you've save me a lot of gas money, too. I won't have to babysit my instruents so much. Mike, your planetary carbon footprint might have just run into the negative numbers!


you're not just a genius, you're a saint!




Mark

Paul Baker

Mark, just to clarify one thing, FAT16 does support subdirectories, however rockicki's fsrw FAT16 driver does not.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Paul Baker
Propeller Applications Engineer

Parallax, Inc.

Mike Green

ElectricAye,
The main reason for closing the file periodically and reopening it in append mode is to make sure the various tables in the SD card directory and FAT are updated so, if the power goes off or some unforseen bug occurs and the program dies or goes crazy, you have a chance to recover the data that's been written so far. Rokicki's routines do not do much buffering so you wouldn't lose much, maybe the last cluster of data or the last 512 byte block. If you divide your 2GB SD card into 500 files, that's roughly 4MB per file. Set an upper limit of 5MB per file and keep track of how much you've written per file. When it exceeds 5MB after a complete line of data, close the file and open a new one with a name with the next higher number ("N000"). You could even use the date and hour of the first data item written for the file name like 08091018.csv for 2008-Sep-10 at 18:00.

rokicki

No need to close and reopen. Call pflush(); it guarantees to update the directory entry (with the full size)
and the last data block (with the most recent data). After a pflush(), power can go out and you won't
lose anything at all.

Of course, you don't want to pflush() after every byte, but there's no real harm (except slowness) to calling
pflush after every data chunk. And pflush() is definitely faster than close/reopen for append.

ElectricAye

To Mike, rokicki, and Paul,

thanks, you guys, your inputs have helped clear my mind on a lot of things.

all the best,
Mark