SD Card Hiccups every ~64 KB...

dwelve

Greetings,

I've been using the SD Card routines by Tomas Rokicki (http://obex.parallax.com/objects/92/, v1.6) on a 1 GB Sandisk and 2GB Kingston microSD card.

I log 172 bytes of data at 50 Hz, the first 32 bytes from pputc and the rest of the bytes from one pwrite.

On the 2 GB card (with a 32 KB cluster size), I seem to be having problems roughly every 366 cycles, or 366 * 172 = 62952 bytes (or maybe actually 367*172 = 63124, depending on how I count things...)

On the 1 GB card (with a 16 KB cluster size), I seem to have problems roughly every ~180 (top of my head for this number), or around ~32 KB, along with other hiccups that are intermittent.

At least on the 2 GB card, I consistently see perfect timing of my data at 50 Hz, that is the 20 ms in between datasets... until I get to around ~64 KB of data logged, then I lose 160 ms of data.

I think I have verified the rest of my code, and that the SD card is the culprit. My hunch is that I lose time due to the SD Card object finding space in the FAT16 file system on the SD card, though with my limited knowledge of FAT16 file systems I can't say this for sure. Formatting the SD card beforehand does not seem to alleviate the problem.

I tried to pre-allocate file space by writing a bunch of zeroes (3 MB worth) to a file, closing it, and then re-opening it for writing, but this does not seem to help. (Perhaps I am not correctly pre-allocating space).

Is my hunch of the problem correct in that the SD Card object takes time to locate new space on the SD Card? Or is there something more sinister going on...

I'd rather not post code due to the confidential nature of my work, but I'm pretty sure I can describe the problem enough if I have not made myself clear.

... as an aside, I wonder if getting a "faster" microSD card will help ...

(FYI, I launch the SD Card write routine its own COG, but I do initialize it in COG 0. I sync my writes to another COG which handles data from a serial device running at 50 Hz).

Thanks in advance to all.
dwelve

Mike Green

You really can't count on the timing of SD cards. There's a processor in the card that takes care of "wear levelling". Every so often (that varies from manufacturer to manufacturer and card model to card model) the processor has to move the data from one area of the card to another and erase the old areas. This is completely independent from the FAT16 routines and is strictly internal to the card. The FAT16 routines do allocate space from time to time and have to search the directory and add or update directory entries to do this, but this allocation is strictly on a cluster basis. Once the space is allocated, it doesn't have to be reallocated. The FAT16 routines do have to look up the next cluster's directory entry, but the boundary is a cluster, not some odd number of bytes.

You could try a different manufacturer's SD card, but there's no way of predicting what the card will do. The internal buffer sizes tend to vary a lot and you have no idea of how they're used and you don't know the algorithms used.

Bill Henning

If you have a spare cog, and enough data to hold 400ms worth of data, maintain two 200ms buffers, and have the other cog write one buffer while you fill the other... double buffering like that would allow for almost 200ms "hickup"

dwelve said...
Greetings,

I've been using the SD Card routines by Tomas Rokicki (http://obex.parallax.com/objects/92/, v1.6) on a 1 GB Sandisk and 2GB Kingston microSD card.

I log 172 bytes of data at 50 Hz, the first 32 bytes from pputc and the rest of the bytes from one pwrite.

On the 2 GB card (with a 32 KB cluster size), I seem to be having problems roughly every 366 cycles, or 366 * 172 = 62952 bytes (or maybe actually 367*172 = 63124, depending on how I count things...)

On the 1 GB card (with a 16 KB cluster size), I seem to have problems roughly every ~180 (top of my head for this number), or around ~32 KB, along with other hiccups that are intermittent.

At least on the 2 GB card, I consistently see perfect timing of my data at 50 Hz, that is the 20 ms in between datasets... until I get to around ~64 KB of data logged, then I lose 160 ms of data.

I think I have verified the rest of my code, and that the SD card is the culprit. My hunch is that I lose time due to the SD Card object finding space in the FAT16 file system on the SD card, though with my limited knowledge of FAT16 file systems I can't say this for sure. Formatting the SD card beforehand does not seem to alleviate the problem.

I tried to pre-allocate file space by writing a bunch of zeroes (3 MB worth) to a file, closing it, and then re-opening it for writing, but this does not seem to help. (Perhaps I am not correctly pre-allocating space).

Is my hunch of the problem correct in that the SD Card object takes time to locate new space on the SD Card? Or is there something more sinister going on...

I'd rather not post code due to the confidential nature of my work, but I'm pretty sure I can describe the problem enough if I have not made myself clear.

... as an aside, I wonder if getting a "faster" microSD card will help ...

(FYI, I launch the SD Card write routine its own COG, but I do initialize it in COG 0. I sync my writes to another COG which handles data from a serial device running at 50 Hz).

Thanks in advance to all.
dwelve

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.mikronauts.com - a new blog about microcontrollers

localroger

What Bill said. Another way you could do it would be to have one cog log data to a circular hub RAM buffer large enough to absorb the hiccups, and a different cog taking data out of the buffer and writing to the SD. Flash cards of all types are famous for locking for a bit, particularly when you write as writes are slow. You have to plan for that when working with flash storage.

dwelve

Thanks, Mike, Bill, and localroger. I bought a PNY 2 GB MicroSD card and a 2 GB Sandisk MicroSD Card and the 2 GB Sandisk seemed to perform the best, with only a three "hiccups" in 10 minutes of testing (I was out in the field so didn't get a chance to write new code, but was able to stop at a store and get some SD cards!). I was told by someone else that I had to find an SD card with the right (or big enough/any) "cache"... I don't really want to go hunting for "the right" MicroSD Card, and I didn't get a chance to try out a full-sized SD Card yet (which may or may not offer better performance), so I'll probably just end up coding in a large buffer (I got plenty of space left...!).

rokicki

What everyone said is pretty much true. The card is probably copying and/or erasing a new block. I too have
observed this even when only writing block by block (with no file system). Buffering, if possible, is the correct
solution (although buffering when you have only 32K RAM total can be a challenge).

One person who worked on this problem had to actually resort to an external RAM solution for the buffering.

StefanL38

Hello
I don't have special knowledge about SD-cards

Is this a special problem of Micro-SD-cards?

If not how the hell does a SD-Card Videocamera store away a 1280x720 pixel-mpeg-video without hickups ?

best regards

Stefan

MagIO2

Maybe it's a bug in the SD driver? Or the driver itself is OK, but you don't check for error-states it returns?

If the SD driver sends data at a time when the SD card says that it's busy, we might loose the data.

I'm sure if we follow the rules of the protocol we won't loose anything. The difference to the video-camera might be in the buffer-size. It has a very big internal buffer and can handle the delays in write-process.

rokicki

There's a couple of things the video camera does that we do not.

1. It probably uses SD mode, rather than SPI mode, which is much faster.
(But it is not clear we can use that mode without paying someone money.)

2. It probably pre-erases blocks and/or uses other command hints that tell
the card to prepare for more data. (We *could* do this too, if anyone
wanted to spend the time to figure it out.)

3. It probably uses a pretty big buffer.

-tom

Fred Hawkins

How much data is lost before the SD card returns to normal? It seems to me that if you are missing just one or two sets of data, and that data is more or less like the data that precedes and follows it, you could extrapolate a middle replacement value. Or simpler, drop that instance(s) as missing data. Then there may even be possible to handle this dropping or extrapolation when you read and evaluate the data. It may be simpler to do that than to rewrite the SD card handling routines to cope with caching.

dwelve

Thanks for thoughts and suggestions, all.

I ended up putting a pretty large buffer in, (960 longs, in fact.. 20 buffers long to cover 400 milliseconds! ... for this particular application, I have the space).

Everything looked nice with a 15-set buffer on the 2 GB Sandisk microSD until 14.75 minutes hit and I lost over 300 ms worth of data (that's why I have such a large buffer now)... kind of seems a waste to have a large buffer that is there only to account for all the unspec'd [noparse][[/noparse]micro]SD cards... at least if I could dig up some spec's so I don't have to go hunting for the right buffer size...

... also, there's got to be a way to test real-time data-logging systems faster than "real-time"... (I guess the only thing holding me back is accounting for the spec's of the [noparse][[/noparse]micro]SD cards... oh, and time!).

@MagIO2,
I don't think there's a bug in the SD Card object... otherwise I think rokicki would have fixed it already...!
(Oh, there is a mishap in the constant date that rokicki put in, where the date he sets is actually January 7st, 2007 at 4 AM and not April 7th as stated in the code, but no biggie--I'm going to get a real-time clock in anyways!).

@Fred,
Extrapolating the missing data is something my customer/end-user would have to decide... I'd rather not have to explain to them why they would even need to extrapolate the missing data! It probably wouldn't be that hard, but given the non-linearity of the system, well, I think I've said enough >_>


Anyways, I've just now completed three tests of over 17 minutes each with the three microSD cards I have with the 400 ms buffer and all is well (except as stated below).

Concerns:
Right now I don't have pull-up resistors on on my SD card lines at all... could this be the reason why the Prop chip seems to "lock-up" when I take out and re-insert the microSD card? The Prop doesn't go into reset, but I lose the ability to communicate with the Prop via my Bluetooth module... on second though, maybe my Bluetooth module is being reset, and thus I need to re-establish my link to it from my PC...(nope, just checked that and that doesn't seem to be the case)! I'm guessing it might have to do with the contact leads crossing the Power and Ground pins on the microSD card socket... will pull-up resistors even help, then... my power would still get shorted even with the pull-up resistors...!. Guess I'll have to document it as such that the device needs to be turned off when inserting the SD card... I seem to have read someone else that it's not good for the SD card to be put in as such when power is on--does anyone know of this and if there is a protection circuitry I could use to protect both the card and the Prop chip and power circuitry?

I had a second concern but I forgot it since it's taking me over an hour to write this post (with all the waiting in between sd card testing, etc.).


Ahh, yes, my second concern is device specific to one of my serial devices... at varying times I have noticed that, under certain conditions, of which I presume are either code-related or SD card specific, the milliseconds timer of my serial device jitters from 20 ms to either 19 or 21 ms! The data is sent back with a checksum so I know my data is sound, but it's just really odd... my hunch right now is that my power supply circuitry is the culprit, with voltage ripple affecting the clocking of the device... my memory recalls of advice from a professor, "Add a capacitor!", as a solution (a solution to all problems, in fact...).

Ahh, I see now that I don't have a capacitor on my 12 V unregulated input line (though I do have caps on my 5V and 3.3V regulators).

Mike Green

SD cards are designed for "hot" insertion. The ground contact is longer than the others and connects first when inserted and last when removed. The pullups are used so the card socket will be in a defined state with no card in place. That ("hot insertion") should not be the cause of your problems.

dwelve

Ahh, silly me I did not re-mount the card in case the user had unplugged it.

Although I do have my suspicions that the shoddy Transcend microSD socket with uber-wiggle-room can cause problems because I've noticed in some instances that "hot insertion" does cause either a reset, or worse a non-reset short of some kind where my Bluetooth will then spit out garbage...

Time to fix this silly bug...

dwelve

Hmm, why does it give the error "Write failure on COM3".

Oh, nevermind... I must be running out of space even though it tells me I have space free in my Object info page... I seem to recall reading this elsewhere.

dwelve

Ahh, yes, now the only problem left is periodic auto-saving of data, in case the user doesn't send a stop/save command or the power shuts off or the user inadvertently pops out the card while power is on (though I'll have to put put in some routines in so the the Prop will know that the SD card isn't there anymore)... well, I guess I did "solve" the problem by having it save either every second or ten seconds or what-have-you, but now I have the clock "jitter" problem on my serial device, sending me data that is either 19 or 21 ms apart instead of 20 ms (like it should...)... the checksum checks out so the data is all correct. (It probably won't matter that much to my customer, they'll probably just treat it as data that is 20 ms apart.... the weird thing is that as time goes by during data logging the "jitter" occurs less frequently.)

Has anyone else experienced this issue before? I'm sure the unregulated/filtered power I send into the serial device is regulated down to some operating voltage, but the device is encased and shrouded in secrecy so I can't really tell if has bypass/decoupling capacitors... although I should probably throw in some caps on my board now (my co-worker put together much of the board with my evolving schematics and I don't think we have a cap in some places we should...)... but throwing off the serial devices clock by 1 ms! That kind of surprises me...

rokicki

I'd say, just give it a 16K buffer or something. In general, delays of up to 500ms or something
are not shocking, but anything over a second is. If you cannot accomodate that level of delay,
then perhaps writing to SD cards is not for you (or perhaps you can experiment with the routines
and improve them using some of the more advanced commands).

-tom

rokicki

Some notes from a random troll of the web; don't know if there's
anything there you haven't seen before.

http://www.ultimaserial.com/sandisk.html

dwelve

Ahh, thanks for the tip, rokicki. And thanks to for the great SD Card object! It's working like a charm right now. I'll have have to remember in future data logging to SD card applications to account for these delays with a large enough buffer... possibly using external ram if needed!

I've been avoiding PASM for the last month or so because I'm on a schedule to get "things done" and SPIN is so simple to use (though I did have my 'gotcha' moments with byte/long alignment and other things concerning SPIN)... maybe afterwards I can devote some real time to PASM so I can write/improve upon some low-level driver objects for the Prop chip (which won't be work-related so I can actually post and talk about it too!).

---
dwelve

KPR

I have a simple question.. the 2 gig is the limit for fat 16 correct, has anyone done any work on spin code for a larger sd card.. I have kids and portable video game unit.. there isn't a 2 gig card in the house.. they all start at 4 gigs..

Has anyone doen any spin work on code for a larger card (4/8/16 gigs)?

pgbpsu

@dwelve-

My experience matches rokicki's comments (i.e. he knows what he's talking about)- delays of as much as 300ms aren't uncommon but can be overcome by buffering. I've got two different data logging firmwares writing to SD card. One collects 200,000bytes/second which is not always too fast for the SD card, but I need that speed for 30 seconds. The jitter and occasional long return time for the SD card forced us to go to a large external buffer. I also have a similar system that writes 20,000 bytes/second to the SD card. To get that working, I had to set up a 2 buffers. Each holding 1660 longs. I write to one buffer while dumping the other and then flip-flop. At my sample rate those buffers are just over 300msec. For this application I occasionally get missing samples, but I know where they are and how many I missed so I can deal with it. If you can't afford to have any dropped samples you'll have to buffer. Depending on how large your program is you might be able to do all this in the prop. If not I can give you some suggestions for external buffers. But doing it in the prop is MUCH simpler if you have the space. BTW, all my writes are block writes. I've not used the pputc routines. I think you're best bet is to load up a buffer with your data, then dump it using pwrite.

@KPR
I've been using rokicki's low-level routines with modifications by lonesock on cards larger than 2G. There is no filesystem so the disk isn't easily readable by a DOS machine. I use the dd command on a Mac to copy data block by block from the card to the computer, then process it there. Check out this link for more information on doing this:
http://forums.parallax.com/showthread.php?p=746666

pgb

KPR

>> @KPR
>> I've been using rokicki's low-level routines with modifications by lonesock on cards larger than 2G. There is no filesystem so the disk isn't easily readable by a DOS machine. I use the dd command on >> a Mac to copy data block by block from the card to the computer, then process it there. Check out this link for more information on doing this:
>> forums.parallax.com/forums/default.aspx?f=25&m=289153
>>
>> pgb

Thanks a million..