More help with sd card, fsrwfemto and file systems.

Ravenkallen

Hey, You guys have been so helpful. Alas, i call upon your help once again. I have successfully interfaced the SD card using a demo program called "SD_trainer". Everything has worked out fine, but this demo just covers the basics of SD protocol. The actual object that powers the demo is called "fsrwfemto" and the object that powers that one is called "sdspifemto". I am very new to SD(propeller to·for that matter), And as such i have no clue about some of the stuff in those objects. Like what is the sector size constant in fsrwfemto. Are sectors the number of files you can have on the disk?
Or does that represent how many bytes can fit on a file? Also, i was wondering, How big a given can file be? I have read some places where is says like 8 KB's to 32 KB's. I hope you can make them larger than that. And what the heck are clusters? they keep mentioning them in the objects. I guess SD cards are kinda like a sector erasable/ programmable flash chip. Where you can only manipulate certain sections at a time, instead of having total access, like i2c. If all seems so hard when you don't know what you are doing, but so easy when you do.·If you guys know any·info about these things, please tell me all about it. Thanks again for everything.

MagIO2

A sector is the smalles unit of data on a floppy/harddrive and nowadays on SD cards as well. Each sector has a number and by telling the SD card the number of a sector it will spit out it's data.

Building on sectors you have the file-system. It's called FAT (file allocation table). You don't have to know much about that because this part is already implemented by fsrwfemto. A FAT is build on 2 basic parts, the cluster-table and the directories.
One cluster combines a number of sectors. This way a list of used parts of the SD card/HD... can be smaller. The cluster table marks the used clusters and builds a chain of clusters that belong together (because these are used by one file).
The directory holds the name of the files and the number of the start cluster and the filesize in bytes.

When building a cluster-table you have to decide how big a cluster number can be ( how much bits you want to waste ;o). The possible cluster size (how many sectors belong to 1 cluster) depends on the size of the storage device and the max. cluster number.

Theoretically this system allows to use all sectors which are not needed for cluster table and directory can be used for one file.

The 8kB or 32kB you mention are not file-sizes, but cluster sizes. A 2GB or 4GB SD card usually is formatted with 32kB clusters. So, even if you have a file with only one byte, this will occupy 32kB.

Ravenkallen

Thank you magio2. So if i am understanding correctly. Sectors are single byte storage locations and clusters are a group of sectors. And then files are groups of clusters. HUH, amazing. So you said the fsrwfemto takes care of the low level stuff? SO you can theoretically have a file size as large as you want? Gosh, i didn't know it was so much work to interface one of these.

Cluso99

Yes Ravenkallen. A sector on the SD is fixed at 512 bytes. It is now usual to format the card in cluster sizes of 32KB. I am not sure of the theoretical maximum file size. There is also a limt to the number of files on a card under FAT16. Some later cards and code can support FAT32. You can find all this info on the wiki.

We use file sizes of 8MB and 32MB for ZiCog and it's derivatives. These files are used to simulate each disk drive under CPM.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Links to other interesting threads:

· Home of the MultiBladeProps: TriBlade,·RamBlade,·SixBlade, website
· Single Board Computer:·3 Propeller ICs·and a·TriBladeProp board (ZiCog Z80 Emulator)
· Prop Tools under Development or Completed (Index)
· Emulators: CPUs Z80 etc; Micros Altair etc;· Terminals·VT100 etc; (Index) ZiCog (Z80) , MoCog (6809)·
· Prop OS: SphinxOS·, PropDos , PropCmd··· Search the Propeller forums·(uses advanced Google search)
My cruising website is: ·www.bluemagic.biz·· MultiBlade Props: www.cluso.bluemagic.biz

Ravenkallen

So a sector is not a single byte storage location, but rather a 512 byte one. SO you use files that have mega bytes of memory? That is exactly what i need for my project. I am working on creating my own programming language and the hardware necessary to support it. The main advantage of it will be the ability to program the device without using a PC. The fast transfer rate and large storage capacity of sd cards, has me hooked. I was planing on using a i2c eeprom, but it can only have up to 64KB of memory, plus it only has a top speed of, like 400Kilohertz. Can one read and write to the memory completely randomly? Like ram, sort of. I need to be able to access a random byte, without changing files or stuff like that. I tried making my own programming language for a Picaxe system, but it failed miserably from lack of speed and program space. So now i decided to use the propeller. I even made my own resistive based keyboard, that has keys for all 26 letters (and when you press a shift button) decimals 0-9, punctuation marks, space and clear screen. The whole system is still in its infancy, but it has a lot of potential. I am going to make the language very simple, but not as easy to understand as BASIC. For example, i will use only four letter commands and most commands will not have more that two operands. It will have some easy to use stuff like, Reading adc's, serial commands, audio outputs, Maybe even specialized sensor commands. If you guys have some suggestions, please fire away.

Cluso99

No a sector is fixed at 512 bytes. If you want to change some bytes in the middle you have to re-read (if it was not the last sector read), modify what you want, and write it back. If you are doing this a lot then the perhaps ~100,000 write cycles may be a problem. The SD and microSd cards are accessed as SPI memory (serially) but I cannot tell you the speed other than it is quite (a relative term) fast. Look at the wiki for some specs.

I have a pcb that has 512KB of SRAM and a microSD called the RamBlade (see my signature). However, because I am after speed, there are only 2 prop pins spare for I/O. These can be for a serial pc connection or to another processor running as a terminal or to a PS2 Kyeboard and TV B&W using my 2-pin drivers (1 for TV, 1 for keyboard). The RamBlade runs ZiCog and CPM as well as a mini operating system of PropDos, PropCmd and Sphinx.

There are other pcbs with SRAM such as my TriBlade, the DracBlade, and Bill's Morpheus products. It really depends on your precise requirements.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Links to other interesting threads:

· Home of the MultiBladeProps: TriBlade,·RamBlade,·SixBlade, website
· Single Board Computer:·3 Propeller ICs·and a·TriBladeProp board (ZiCog Z80 Emulator)
· Prop Tools under Development or Completed (Index)
· Emulators: CPUs Z80 etc; Micros Altair etc;· Terminals·VT100 etc; (Index) ZiCog (Z80) , MoCog (6809)·
· Prop OS: SphinxOS·, PropDos , PropCmd··· Search the Propeller forums·(uses advanced Google search)
My cruising website is: ·www.bluemagic.biz·· MultiBlade Props: www.cluso.bluemagic.biz

MagIO2

For modifying single bytes you can use what I called vMem (virtual memory). You no longer need to know in which sector a byte is located. Say you have a file which has 128MB you can simply use the address of each byte inside of the file to read/write it. vMem keeps the sector in memory as long as you don't access a byte in a different sector. So, if you write bytes 0-511 in a sequence the sector will not be written back with each change but when you access byte 512.

As clusoe already mentioned, it depends on your application whether it's a good idea to use SD or not. For high frequency write it's not a good idea, as the SD will very soon malfunction then.

Have a look at this thread: http://forums.parallax.com/showthread.php?p=856170

Please read the comments in the driver and the demo very carefully!

Ravenkallen

No, i am going to use the SD card as non volatile program storage. And for ram, i am going to use whatever RAM is left in the propeller that is not being used by the interpreter. Say, do you guys know of any good SRAM modules. I have used a ton of different parallel access memories over the years, so i know how a lot of them work, but i never did find a good SRAM. I have tried battery backed NVSRAM, but i don't need the system to retain memory upon power up. I wonder if there is a serial access ram of some sort. Wait, so with your program, you can really access A WHOLE file(128MB) completely randomly? Like, you can use a long to address any part of the memory?

Ravenkallen

I don't know. The SD card seems kinda of lame now. Maybe i will have to go back to my original i2c eeprom plan. I guess i could stick several of them on the same bus and just change the slave address for each one. If the system could fit 9 eeproms on the same line that would be like 9x 64KB. I don't know what to do, really. I could use I2c or i could use the SD card. I don't want to have to write back 511 bytes, just to change one byte. That is nuts. I guess the SD card really is not a good choice for my application. Bummer, huh. Thank you guys. Your help means a lot. Sorry, if it seems like i don't "get" things very well. I am kind of a slow learner, but once i know something, I got it in my noggin forever.

MagIO2

Using the propeller a SD card can read up to 1.6MB/sec in SPI mode. And as far as I remember writing is not to much slower. Maybe you should tell us a bit more about what you wanna do.
If you really need random access the speed will of course go down as you read/write 512 bytes just for picking/updating 1 byte. Maybe something in logic/datastructure can be changed.

Post Edited (MagIO2) : 4/25/2010 5:17:04 PM GMT

Ravenkallen

I just need a system where i can READ from the WHOLE file completely at random. Like something so simple, where you can use a long variable as the address pointer and read data from it. All though writing would be nice to, i just need to read for now. I don't even know how i would do that, seeing as the demo only covers reading/writing strings. I need complete random access to the file. Does that program you mentioned do that? I really want to use the sd card method( Instead of i2c), but i just have no clue on how to start. I mostly just want to know if it is possible to read completely radomly from the card. I am not to concerned about speed right now, i just need something functional. If that Virtual memory thing works like you say, then that should be what i need. Thanks for your time.

MagIO2

Yes, vMem can do that!

You need the file you wanna read from as the first file on the SD card. Simply format the SD card, copy that file to it (per default it has to be names SWAP.SYS, but that can be changed in your program if you need). After that you can copy whatever you want. The file can have any size up to the maximum free bytes of the SD card.

For real random access you'd use the vPeek vPoke functions for longs or vbPeek and vbPoke for bytes. Please note that addresses for longs are automatically alligned to long-addresses. This means that address 0, 1, 2 and 3 will access the same long.

Ravenkallen

So do you have a link to the vMem thing? So you need to reformat the card? You need to copy what file to it?. So you just create a file called swap.sys and then put stuff in the file? And you can read and write from it randomly?

Cluso99

Also take a look at ZiCog. The older versions use sdspifemto. The start of a file(s) is found and then we just use the direct sector for accessing the file. It is very simple and easy to do. Another possibility is to use a SPI Flash as they are much larger than the I2C eeprom. There are also SPI ram, fram, etc.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Links to other interesting threads:

· Home of the MultiBladeProps: TriBlade,·RamBlade,·SixBlade, website
· Single Board Computer:·3 Propeller ICs·and a·TriBladeProp board (ZiCog Z80 Emulator)
· Prop Tools under Development or Completed (Index)
· Emulators: CPUs Z80 etc; Micros Altair etc;· Terminals·VT100 etc; (Index) ZiCog (Z80) , MoCog (6809)·
· Prop OS: SphinxOS·, PropDos , PropCmd··· Search the Propeller forums·(uses advanced Google search)
My cruising website is: ·www.bluemagic.biz·· MultiBlade Props: www.cluso.bluemagic.biz

MagIO2

Ravenkallen said...
So do you have a link to the vMem thing?

Yep ... 6 posts above the post you asked this question. Somewhere in the middle of the thread you find a archive called vMem - Archive .....

Ravenkallen said...
So you need to reformat the card?

Yep ... as stated before ... and you'll find an additional description in vMem object (which is only an addon to FSRW). Please read!

Ravenkallen said...
You need to copy what file to it?.

Any file that is big enough to store your data. If you only want to do random reads I suppose you already have a file containing some data, don't you? Take it.

Ravenkallen said...
So you just create a file called swap.sys and then put stuff in the file?

See answer above ... it depends on what you want to do. If you already have a file containing the data that you need you can take that file. If you are happy with an empty file you can create an empty file. vMem will open a file called SWAP.SYS per default. If you want to use another file you can call vOpenSwap with another filename. Important thing is that you never write to an address which will not fit into the file! Which means, if the file has a size of·32768·bytes you should only write to byte 0 - 32767. Any address above that will overwrite whatever is stored in the next sector/cluster of the SD card.

Ravenkallen said...
And you can read and write from it randomly?

Yep ... but as discussed before, if you really do random read/write an SD card is only good in a write sometimes/read many application. According to clusos MTBF of ~100000 write cycles to the same sector, if you do 1000 writes per day per sector, you'd wear out the SD card in 100 days. If you only have 10 writes per sector a day, you have 10000 days.

Leon

The card could be made to last a lot longer by using software that detects when a segment is faulty, locks it out, and uses another. A similar technique is sometimes used with flash devices that have a limited number of cycles.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Leon Heller
Amateur radio callsign: G1HSM

Ravenkallen

I think i am going to stick with i2c for now, although the spi flash/RAM sounds kinda cool.... Thanks magio2 for your time( Sorry i wasted it).

Ravenkallen

So, i tried looking at the Vmem thing. I didn't really understand it. It sounded like you could only access 512 bytes at a time. I just need to read from the blasted thing at random, Gosh it seems so difficult. I don't really want to use the i2c eeprom if i don't have to, so if you know of anything else, please feel free to share. The sd card is good for the huge memory space and faster speed, but it is diabolical trying to get it to work the way i need it to. I apologize for my indecision.

Mike Green

Reading from either an SD card or EEPROM doesn't wear anything out. If you want to use an SD card file, it can be many megabytes in size. You'd have a routine that's given a 24 bit address. The routine takes the most significant 15 bits, uses that as a sector address, and reads in the 512 byte sector that's there. The least significant 9 bits is the byte address of a byte in that sector which your routine will return. You could write your routine to be smart about remembering what sector is in the buffer so it doesn't have to read it in again if, say, the next byte in the sector is accessed on the next call.

If you want to use EEPROMs, remember that they come in sizes up to 128K bytes and you can have four of those on a pair of I/O pins for a total of 512K bytes. I have part of a compiler that reads a file from an SD card into a 32K block of EEPROM. This is the source program. The 2nd 32K block is the dictionary. The 3rd 32K block is the object file which gets written to the SD card if the compilation is successful. The main advantage of this scheme is that it's really easy to make small "random" changes in the dictionary and object file as the compilation progresses. If the files had to be on the SD card, there would be a lot of 512 byte blocks read and written for a series of 1 or 2 byte changes and/or accesses scattered over the whole 32K area.

MagIO2

vMem has a local copy of 512 bytes in HUB RAM because a sector has 512 bytes and if you want to write back you have to write 512 bytes. But vMem takes care of the sector-handling. If you do a vbPeek or a vbPoke you can access each single byte of the whole file.
Let's say your file is 1MB in size = 1024kB. As one sector exactly has 0.5kB you need 2048 sectors to store this file. If you give each byte in this file an address, the address range is from 0 - 1048575. If you want to access a random byte inside of this file you have to find out in which sector it is. This is an easy operation, as 512 is 2^9 which means that it perfectly fits to bit-representations. Addresses of the first sector ( 0 - 511 ) have a bit representation of %...000_000000000 - %...000_111111111. The next address will be located in the second sector and have the address %...001_000000000. So, on the right side of the underscore you find the relative address inside of a sector and the left side gives you the number of the sector to load.
Now let's assume you want to access byte 128, 196, 246 and 1033 in this sequence. vbPeek( ... 128 ) will load the first sector and gives you the content of the byte with address 128. vbPeek( ... 196 ) will not load, because it's in the same sector which is already loaded into a local buffer. It simply returns the value of byte 196. Same for byte 246. When you now do vbPeek( .. 1033 ) vMem recognizes that another sector is now needed. First it checks whether something has been written in the old sector and writes it back if so, then it loads the sector needed to find byte 1033.
So, if you do really jump from one end of the file to the other with each access the only thing that can happen is, that vMem has to read a sector with each access. But you don't have to take care of that!

Ravenkallen

Okay, i think i am getting it. So the vpeek and vpoke commands can accept any address, as long as the MSB's of the address represents the sector to read/write? so you can't write to a byte(In the same sector) that has already been writen to in the same sector. But you can read from any byte in the buffer in the same sector without having to load a new one? So in order to write to random bytes in the same sector, you would have to load the sector, write your byte and then you would have to load a whole differnt sector and then reload the one you wanted to write in. I bet the delay to read in the 512 byte buffer would be substanial. It sounds a little more clear now, but it is still pretty fuzzy...... Thanks for the idea mike. I was thinking about a similiar system, have the sdcard write itself to the eeprom, which can then provide random read access. Only thing is, i would need quite a few eeproms to get the size i needed. Thanks for the help guys!!!