Out-of-order SD card access
mirror
Posts: 322
Well it's not the first time that what I suspected was a hardware bug was in actual fact a software bug (limitation), but I thought it worth sharing as so many others are using SD cards.
If you write sectors in a sequential order, then you'll have no problems, but if you do what I did, then BEWARE.
Basically what I have is an application thats logging data to an SD card. I have regular periodic Header blocks.·If there's more that 512 bytes (1 sector) of data in a period·then it flows on into the Extra blocks. Once I've written all the Extra blocks I·update the Header block with the total number of blocks (Header + number of Extras)·and write it to the·SD card. So, the SD card memory ends up looking something like this: (with order of writing to SD·below)
The reason for doing this is that I can have from 0 to 11 blocks of extra information, and I wanted a way of being able to quickly traverse through the SD card memory reading just the Headers.
The problem: When the out-of-order header blocks are written, they are taking in the order of 30 times longer to write than when the SD card sectors are·written in-order. This was unexpected, so took quite a while to find as I thought I had a hardware problem.
Now that I know about it the workaround is easy enough to write, but it came as quite a surprise. BTW I've·completely changed how the data is logged as I don't have 6k of free RAM to buffer the packet before saving it to the SD card.
If you write sectors in a sequential order, then you'll have no problems, but if you do what I did, then BEWARE.
Basically what I have is an application thats logging data to an SD card. I have regular periodic Header blocks.·If there's more that 512 bytes (1 sector) of data in a period·then it flows on into the Extra blocks. Once I've written all the Extra blocks I·update the Header block with the total number of blocks (Header + number of Extras)·and write it to the·SD card. So, the SD card memory ends up looking something like this: (with order of writing to SD·below)
H H H E E H H H H E H E H H 1 2 5 3 4 6 7 8 10 9 12 11 13 14
The reason for doing this is that I can have from 0 to 11 blocks of extra information, and I wanted a way of being able to quickly traverse through the SD card memory reading just the Headers.
The problem: When the out-of-order header blocks are written, they are taking in the order of 30 times longer to write than when the SD card sectors are·written in-order. This was unexpected, so took quite a while to find as I thought I had a hardware problem.
Now that I know about it the workaround is easy enough to write, but it came as quite a surprise. BTW I've·completely changed how the data is logged as I don't have 6k of free RAM to buffer the packet before saving it to the SD card.
Comments
to not flush too frequently (causes several out of order block writes) and to use as big clusters as possible (32K).
It also depends on the card; what SD card are you using? Different cards may have significantly different characteristics.
It would be worthwhile to duplicate your example in a small test program that measures how long each block write takes, and I can
run that on my collection of cards and see what happens. If we can model the behavior somehow that would be useful for a lot of
people.
There's no question though, even when writing sequentially (on some cards), that writing a block can take a *long* time every
once in a while, even though most block writes go through rather quickly. This has a lot to do with the structure of the flash
memory and how entire pages are erased, and then rewritten on demand, and blocks are shuffled around from one place on
the card to another (wear leveling).
I'm using the low level SD card driver that you wrote, but not the FSRW stuff. I don't need any FAT compatability, so most of the 1GByte card is being used as a gigantic circular buffer. As a result of this, the sectors are normally written (almost)sequentially and wear leveling should not be a major factor.
I have made a minor change to the SD driver code that you wrote.
PUB WriteBlockStart(n, b) {{ Together with WriteBlockWait this is a split version of WriteBlock. Makes sure the previous write has completed before starting this one. }} WaitForIdle param := b blockno := n command := "W" PUB WaitForIdle {{ Together with WriteBlockStart this is a split version of WriteBlock. }} repeat while command if param abort paramThe if command WaitForIdle thing has been included in all the other·functions as well. This allows the other process to continue·while the SD driver goes about it's business of filling up the card. Mind you, this modification doesn't make much difference to the original test code that you produced, but I've got a bit more processing that is able to go on in the background.
I decided not to in order to keep things as simple as possible, but it's a nice
change and one that should be available to others.
One key thing there of course is that the buffer must not be modified after
WriteBlockStart() is called until WaitForIdle() is called. You would of course
do the right thing here, but other people would get bit and wonder why the
routines don't work.