Problem writing more than 500 files to my SD card - What am I doing wrong?
Hi all,
I've been running a test to see how fast my SD card can open a new file, write a few bytes, and close up that file and move on to a new file. Everything seems to work great, except the card won't accept more than about 500 files. It writes up to about 498 and then stops. It's a Lexar 2GB regular SD card and I am using it right out of the box. I checked it with WindowsXP and saw that it was formatted for FAT16, so I thought I'd be able to write thousands of files at 32Kb, not 500 files at 32Kb. When I look at the card on my iMac, it says the card is formatted in FAT16 and has used about 17Mbytes, but still has 1.87Gbytes available. What am I doing wrong? Was I supposed to re-format it some special way?
Oh woe is me....
I've been running a test to see how fast my SD card can open a new file, write a few bytes, and close up that file and move on to a new file. Everything seems to work great, except the card won't accept more than about 500 files. It writes up to about 498 and then stops. It's a Lexar 2GB regular SD card and I am using it right out of the box. I checked it with WindowsXP and saw that it was formatted for FAT16, so I thought I'd be able to write thousands of files at 32Kb, not 500 files at 32Kb. When I look at the card on my iMac, it says the card is formatted in FAT16 and has used about 17Mbytes, but still has 1.87Gbytes available. What am I doing wrong? Was I supposed to re-format it some special way?
Oh woe is me....
![cry.gif](http://forums.parallax.com/images/smilies/cry.gif)
Comments
You are not doing wrong nothing.
FAT 16 ROT map have only 512 places in its directory.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nothing is impossible, there are only different degrees of difficulty.
Sapieha
It was very fine news.
Yes FAT16 have that limitation. It is old thinking that MS have (Big enowgh).
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nothing is impossible, there are only different degrees of difficulty.
Sapieha
I'm depressed that I'm limited to only 512-ish files, but I'll recover soon enough.
And Mike, your way of showing me how to increment file names works wonders. After I get it polished up, I'll try to post it for others to use.
One sugestion.
If You not needs old files edit You spin. to test if no place and if not delate one oldest.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nothing is impossible, there are only different degrees of difficulty.
Sapieha
format the card to support many thousands of files. Also, how many files would you like?
Does your code support successful reading/writing from a folder yet?
This would allow him to get past the 512ish file limit on the root directory of his SD.
OBC
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
New to the Propeller?
Getting started with a Propeller Protoboard?
Check out: Introduction to the Proboard & Propeller Cookbook 1.4
Updates to the Cookbook are now posted to: Propeller.warrantyvoid.us
Got an SD card connected? - PropDOS
Post Edited (Oldbitcollector) : 9/8/2008 6:48:33 PM GMT
Is is possible for you to "Append" using the "a" command to an existing file so that you can write more to each file?
I'm not sure how you are splitting your data, but if you had to split by hour/day perhaps just adding a
comma delimited field that indicated a time stamp or date stamp would allow you to store more data in
a single file and still easily locate what you are looking for.
OBC
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
New to the Propeller?
Getting started with a Propeller Protoboard?
Check out: Introduction to the Proboard & Propeller Cookbook 1.4
Updates to the Cookbook are now posted to: Propeller.warrantyvoid.us
Got an SD card connected? - PropDOS
To OBC: my SD card came formatted so that each file I create using the SD object (from the Obex) makes the file 32Kbytes whether it holds 1 byte of data or 31Kbytes of data. I format the data as comma separated variables, and I give the files the extension ".csv" . Part of every data record is a time and date stamp. Each file has a name that looks something like AAB00076.csv AAZ00034.csv etc. Once I someday read all of the data into an Excel spreadsheet, it will be sorted by time and date.
Mind you, not getting 1.8 Gbytes out of my SD card is not a panic situation; it was more like a disappointment. It just means I have to be "Johnny on the Spot" whenever it's time to swap out a card. I'm just thrilled beyond belief that, with the help of everyone here, especially rokicki and Mike "I will teach you how to fish so you can eat forever" Green, I was able to get this contraption up and running in less than a hundred years. <He falls to his knees and weeps with joy....>
thanks,
Mark
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
It might be the Information Age but the Eon of Ignorance has yet to end.
capacity of the card (and the 2GB partition limit for a FAT16 partition). A file can use as many blocks as it wants.
A reasonable split might be to put 8MB in each of 256 files. That won't stress anything too much.
Note that I do recommend the use of 64K blocks, not 32K blocks. This does increase internal fragmentation a tad,
but if you are limited to ~500 files, that won't be much of an issue (wasting at most 64K*512 or about 32MB of a
2GB card is less than 2%.)
Congratulations on getting things to work!
Rokicki, let me make sure I'm hearing you correctly: are you saying that I could actually format the SD card in some way and/or change the SD object code to get the full 1.8Gbytes out of a single card??? If so, that would be awesome! But do tell: where do I find details on how to do this? Do I reformat the card? Change the Obex object code? You have me gasping on the edge of my chair, here.
thanks so much,
Mark
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
It might be the Information Age but the Eon of Ignorance has yet to end.
you're not just a genius, you're a saint!
Mark
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Paul Baker
Propeller Applications Engineer
Parallax, Inc.
The main reason for closing the file periodically and reopening it in append mode is to make sure the various tables in the SD card directory and FAT are updated so, if the power goes off or some unforseen bug occurs and the program dies or goes crazy, you have a chance to recover the data that's been written so far. Rokicki's routines do not do much buffering so you wouldn't lose much, maybe the last cluster of data or the last 512 byte block. If you divide your 2GB SD card into 500 files, that's roughly 4MB per file. Set an upper limit of 5MB per file and keep track of how much you've written per file. When it exceeds 5MB after a complete line of data, close the file and open a new one with a name with the next higher number ("N000"). You could even use the date and hour of the first data item written for the file name like 08091018.csv for 2008-Sep-10 at 18:00.
and the last data block (with the most recent data). After a pflush(), power can go out and you won't
lose anything at all.
Of course, you don't want to pflush() after every byte, but there's no real harm (except slowness) to calling
pflush after every data chunk. And pflush() is definitely faster than close/reopen for append.
thanks, you guys, your inputs have helped clear my mind on a lot of things.
all the best,
Mark