heater said...
jcwren: "...placing that newly written sector in the least recently used block on the SD card.."
There is the part I do not understand. Once you have written to a block, via a FAT or whatever file system, how can the SD card ever know that you no longer need that block? Once a block is written to it is "in use" and remains "in use" forever. The SD card can not know that you have deleted that file and no longer need it's blocks.
Like hard disks, flash disks have a little pool of spares. Additionally, they have really clever firmware that says "this dude has written this sector 6,000 times. Let's swap it with this other sector that has only been written once".
It's not a matter of knowing what is and isn't in use. To the card, it's all in use. It's a matter of knowing what can be swapped where and what is really dead. Clever firmware.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Life may be "too short", but it's the longest thing we ever do.
It does this by reusing a mapping table and a LRU (Least Recently Used) list. These actually may be integrated together, I can think of several ways to do this off the top of my head.
Let's say you have 4096 sectors, plus one for wear leveling, plus a couple extras for bad block mapping and such. The mapping table is empty, and LRU table contains a list of all the sectors available, and this list is in linear order from 0 to 4096 (for a virgin flash).
You write logical sector 0. The controller goes to the front of the LRU table and gets and removes the first entry, which happens to be physical sector 0 because the flash array has never been used. Your data gets written to physical sector 0, the mapping table gets updated to reflect that a request to logical sector 0 gets mapped to physical sector 0. Physical sector 0 is now added to the end of the LRU list. This means that this sector won't be reused for another 4096 requests, because it's at the end of the table.
Now write logical sector 1. The entire process repeats, only physical sector 1 is now at the end of the LRU list, and physical sector 0 is at the next to last in the list.
Now logical sector 0 is written again. So the remapping table now has logical sector 0 mapped to physical sector 2. Physical sector 2 is now at the end of LRU list.
You write lots of sectors, and sooner or later, you write 4096 sectors. Now the magic occurs. Let's say you're writing logical sector 0 again. Last time we wrote it, it was pointing to physical sector 2, which has a use count of 1. So the next entry in the LRU list is read and removed, which is indicating physical sector 4096 (the 4097th sector) is the next to be used. Everything happens as usual, but now the next sector in the LRU list is physical sector 0.
So one more time we'll write logical sector 0. We go to the LRU list and there's physical sector 0 again. So this time our data gets written to physical sector 0, who now will have a use count of 2 when we're done.
And for all 4097 writes we've done, each sector has only been used once, except for physical sector 0 that's been used twice.
The LRU list can be maintained as part of the extra 16 bytes per flash sector where the ECC is stored, along with the use count. There'd also be bits to indicate if a block has gone bad, and it's been moved to the bad block list.
Using this scheme (or a similar one), you never have to know what sort of file system is mapped on top of it. It's purely based on usage count, and rotating sectors through for even wearing.
If you're using one byte of one sector for your entire data storage (no file system, nothing else), and each sector has a life time of 1000 writes, you have sectors-on-disk * 1000 number of writes available. Every time you write the sector, you're changing which physical one gets written to.
In certain types of flash you can rewrite any 1 bit to a 0 without erasing the block it's in. This can be taken advantage of to add information without erasing. And in fact, with EEPROM parts (like the 24Cxxx series), it's a common technique to append data. If you're using ECC in the flash, this isn't possible, since changing any data in the data area (the 512 bytes) will result in an ECC change. However, if you're only changing the 16 byte area in the 528 byte page, you can sometimes use this to your advantage. It's usually more desirable to clear bits (when possible) than set them in flash memory (so it's better to clear an in-use flag than to set one).
heater said...
jcwren: "...placing that newly written sector in the least recently used block on the SD card.."
There is the part I do not understand. Once you have written to a block, via a FAT or whatever file system, how can the SD card ever know that you no longer need that block? Once a block is written to it is "in use" and remains "in use" forever. The SD card can not know that you have deleted that file and no longer need it's blocks.
What I don't get is your other statement "Let's swap it with this other sector that has only been written once"."
Well, if I have written to it once that probably means I want the data I wrote there. So please God the SD card leveling should never recycle it.
So the question still stands: How can the card know what blocks I don't want any more ?
If it does not know that simple fact in cannot use them for "recycling".
Now, there is a new feature in (or coming to) Linux whereby it tells SD cards which blocks it no longer cares about. Can't for the life of me remember what it is called now, so I can not refer to it. The whole idea being to make discarded blocks available again for swapping in the levelling process.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
For me, the past is not over yet.
They're all fibbing so that people won't worry about it all. Like you say the chances of taking that many pics? The camera would die first.
VERY LIKELY!
You can't tell the quality by what you pay anymore.
The only failed flash card I have, which was given to me by someone else because it failed is...
Oh, look at that, a 16MB Sandisk!
Dated 1999, I got it within 2 years of its failure, and with absolute certainty before 2001.
It is still readable, most data is not corrupt, but it rounds its total size to exactly 15MB;
and the others like it do not do that.
Hmmm. I just remembered forgetting to look into what is "SONY Virtual Expander" file on some SONY USB drives.
Perhaps another Rootkit virus? Maybe they are not really 2GB? Fake compression was reported a few years ago,
which overwrites old data when the smaller than marked chip is full. And I do recall only being able to put 1GB
of data on one that was marked 2GB, and about 1.5GB on another.
I have a Gamecube and it is older than my five year old son and the memory card hasn't been replaced.· It is probably seven or eight years that we've had it.
Is it possible that the data logger is exceeding the speed of the card?· Is it possible that the routines aren't up to date or correct with the current standard?· I'm sure memory cards have changed just a little since 1991.
There are some good brands.· Kingston, Adata, Sandisk, and Lexar are brands which I haven't had any problems with.· I have some no name brands that I have used which haven't caused a problem.
I have received free cards with one camera.· They were offers to get you to buy and they may have been defective returns from customers.
One other thing about SD cards.· If you squeeze them they can be damaged according to reports.· So it may not be your memory card but the way they are handled.
I'm sure there is more than one factor.· I have to go but good luck with your work.
Comments
Like hard disks, flash disks have a little pool of spares. Additionally, they have really clever firmware that says "this dude has written this sector 6,000 times. Let's swap it with this other sector that has only been written once".
It's not a matter of knowing what is and isn't in use. To the card, it's all in use. It's a matter of knowing what can be swapped where and what is really dead. Clever firmware.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Life may be "too short", but it's the longest thing we ever do.
Let's say you have 4096 sectors, plus one for wear leveling, plus a couple extras for bad block mapping and such. The mapping table is empty, and LRU table contains a list of all the sectors available, and this list is in linear order from 0 to 4096 (for a virgin flash).
You write logical sector 0. The controller goes to the front of the LRU table and gets and removes the first entry, which happens to be physical sector 0 because the flash array has never been used. Your data gets written to physical sector 0, the mapping table gets updated to reflect that a request to logical sector 0 gets mapped to physical sector 0. Physical sector 0 is now added to the end of the LRU list. This means that this sector won't be reused for another 4096 requests, because it's at the end of the table.
Now write logical sector 1. The entire process repeats, only physical sector 1 is now at the end of the LRU list, and physical sector 0 is at the next to last in the list.
Now logical sector 0 is written again. So the remapping table now has logical sector 0 mapped to physical sector 2. Physical sector 2 is now at the end of LRU list.
You write lots of sectors, and sooner or later, you write 4096 sectors. Now the magic occurs. Let's say you're writing logical sector 0 again. Last time we wrote it, it was pointing to physical sector 2, which has a use count of 1. So the next entry in the LRU list is read and removed, which is indicating physical sector 4096 (the 4097th sector) is the next to be used. Everything happens as usual, but now the next sector in the LRU list is physical sector 0.
So one more time we'll write logical sector 0. We go to the LRU list and there's physical sector 0 again. So this time our data gets written to physical sector 0, who now will have a use count of 2 when we're done.
And for all 4097 writes we've done, each sector has only been used once, except for physical sector 0 that's been used twice.
The LRU list can be maintained as part of the extra 16 bytes per flash sector where the ECC is stored, along with the use count. There'd also be bits to indicate if a block has gone bad, and it's been moved to the bad block list.
Using this scheme (or a similar one), you never have to know what sort of file system is mapped on top of it. It's purely based on usage count, and rotating sectors through for even wearing.
If you're using one byte of one sector for your entire data storage (no file system, nothing else), and each sector has a life time of 1000 writes, you have sectors-on-disk * 1000 number of writes available. Every time you write the sector, you're changing which physical one gets written to.
In certain types of flash you can rewrite any 1 bit to a 0 without erasing the block it's in. This can be taken advantage of to add information without erasing. And in fact, with EEPROM parts (like the 24Cxxx series), it's a common technique to append data. If you're using ECC in the flash, this isn't possible, since changing any data in the data area (the 512 bytes) will result in an ECC change. However, if you're only changing the 16 byte area in the 528 byte page, you can sometimes use this to your advantage. It's usually more desirable to clear bits (when possible) than set them in flash memory (so it's better to clear an in-use flag than to set one).
Post Edited (jcwren) : 1/24/2010 5:05:41 PM GMT
What I don't get is your other statement "Let's swap it with this other sector that has only been written once"."
Well, if I have written to it once that probably means I want the data I wrote there. So please God the SD card leveling should never recycle it.
So the question still stands: How can the card know what blocks I don't want any more ?
If it does not know that simple fact in cannot use them for "recycling".
Now, there is a new feature in (or coming to) Linux whereby it tells SD cards which blocks it no longer cares about. Can't for the life of me remember what it is called now, so I can not refer to it. The whole idea being to make discarded blocks available again for swapping in the levelling process.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
For me, the past is not over yet.
They're all fibbing so that people won't worry about it all. Like you say the chances of taking that many pics? The camera would die first.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Style and grace : Nil point
VERY LIKELY!
You can't tell the quality by what you pay anymore.
The only failed flash card I have, which was given to me by someone else because it failed is...
Oh, look at that, a 16MB Sandisk!
Dated 1999, I got it within 2 years of its failure, and with absolute certainty before 2001.
It is still readable, most data is not corrupt, but it rounds its total size to exactly 15MB;
and the others like it do not do that.
Hmmm. I just remembered forgetting to look into what is "SONY Virtual Expander" file on some SONY USB drives.
Perhaps another Rootkit virus? Maybe they are not really 2GB? Fake compression was reported a few years ago,
which overwrites old data when the smaller than marked chip is full. And I do recall only being able to put 1GB
of data on one that was marked 2GB, and about 1.5GB on another.
Post Edited (VIRAND) : 1/24/2010 11:51:17 PM GMT
Is it possible that the data logger is exceeding the speed of the card?· Is it possible that the routines aren't up to date or correct with the current standard?· I'm sure memory cards have changed just a little since 1991.
There are some good brands.· Kingston, Adata, Sandisk, and Lexar are brands which I haven't had any problems with.· I have some no name brands that I have used which haven't caused a problem.
I have received free cards with one camera.· They were offers to get you to buy and they may have been defective returns from customers.
One other thing about SD cards.· If you squeeze them they can be damaged according to reports.· So it may not be your memory card but the way they are handled.
I'm sure there is more than one factor.· I have to go but good luck with your work.