@Toby "The only thing I have a quiry on is, When the first SD> prompt comes up, I always get a "Unknown command" message. The wanted cammand is happy on the second and all other times though."
Is that using the serial driver or the local keyboard? I had the same problem on the serial driver, as the buffer has one or two bytes in it - I think from when it is connected or from the terminal program at the other end. The problem seemed to go away when I changed to local keyboard. It might be a matter of clearing the buffer.
@OBC, TV would be the next logical thing. Do you have some code?
The first prompt not working problem I have is from local kbd, I have not tried the remote way yet. Although the '232 was connected all the time to the PC. I will check that out when Clunky gets back on the bench (playing with AVRs, sorry, sorry, ... )
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Why did I think a new, more challenging, job was a good idea ??
Kye, Can you look into this for supporting long filenames? en.wikipedia.org/wiki/Rock_Ridge
@potatohead mentions it in another thread. Without long filenames your work is not useful for my next Java project.
Mmm, I sorry but I really can't be the one to add support for Rock_Ridge... I do not have time. This driver has already taken me a year to complete using most of my extra free time. I will not be changing the structure for a while.
But, you can just store the long name of the file inside the file as the first 256 bytes. Then make the actual names of the files on disk just hash codes that should match the long name of the file when asked for... or something like that.
>_ Running command: test
Creating test file "testfile" ... Success
Wrote 131,072 bytes at 0000042484 bytes per second
Running byte stride write test...
Wrote 32,768 bytes at 0000004192 bytes per second
Running byte stride read test...
Read 32,768 bytes at 0000004297 bytes per second
Running word stride write test...
Wrote 65,5536 bytes at 0000007895 bytes per second
Running word stride read test...
Read 65,5536 bytes at 0000008489 bytes per second
Running long stride write test...
Wrote 131,072 bytes at 0000014289 bytes per second
Running long stride read test...
Read 131,072 bytes at 0000016804 bytes per second
Running speed writing test (512 bytes at a time)...
Wrote 131,072 bytes at 0000062608 bytes per second
Running speed reading test (512 bytes at a time)...
Read 131,072 bytes at 0000250432 bytes per second
Running append test... Success
Running seek test... Success
Deleting test file "testfile" ... Success
This time there is no "0" bytes per second read. I guess the one in my first post was due to some overrun?
Kye said...
Mmm, I sorry but I really can't be the one to add support for Rock_Ridge... I do not have time. This driver has already taken me a year to complete using most of my extra free time. I will not be changing the structure for a while.
But, you can just store the long name of the file inside the file as the first 256 bytes. Then make the actual names of the files on disk just hash codes that should match the long name of the file when asked for... or something like that.
I believe I can make it work for what I need to do ... something like that. Thanks.
Not every card works with FSRW as I've heard also. Mmm, I really have no clue. Apparently though there are differences in the card timing betwen thick and thin SD cards. It could be alot of things. I would need to have the said card in hand to debug with.
Kye said...
Not every card works with FSRW as I've heard also...
Hmm. Do you know any specific cases? I was under the impression that the latest FSRW 2.6 version, using the default safe_spi block driver, hadn't had any issues. I do know that with some cards you can still get to a state that requires a power cycle if the prop was reset during a transfer, but that's because the MMC and SD specs don't specifically mandate how the cards handle a reset if they think they are in the middle of a multiblock read or write. And we use the multiblock mode because SDHC cards tend to suck, performance wise, using singleblock mode. Without the multiblock mode we wouldn't be able to get our current transfer rates of > 1MB/s.
Jonathan
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
lonesock
Piranha are people too.
They say the only dumb question is those that go un-asked, so to prevent this from staying a dumb question, here it is:
What are the capabilities of the WAV play and record functions in the SD2.0 object? What's the best "quality" I can record and what's the best "quality" that I can convert on my PC to have the propeller playback?
A friend of mine used my VMusic2 in a project for his autistic son that I believe could be easily done with this new object, but the audio needs to be clean on a 2" speaker. Obviously, with a 2" speaker I don't need CD quality, but telephone quality wouldn't be desirable as a finished project. The dis-advantages of the current design is that the audio files (mostly voice) have to be pre-recorded as MP3s which makes it less user friendly and requires a PC. It would be nice to be able to record the audio clips directly on the device.
The WAV play function handles all wav files that comform to https://ccrma.stanford.edu/courses/422/projects/WaveFormat/ that layout and can hit play back speeds of 44100 Khz. The DACEngine can go all the way up to 80Khz, but the block driver read speeds will only get you to about 60Khz max.
The record feature records wav files that also conform to the layout on the posted link. The only limiting factor with it is that some SD cards with my driver have very bad write speeds. So... I set the recording feature down to 8Khz with 8 bit sampling.
However, the ADCEngine can go up to 80Khz at 16 bits if the cards transfer speed can keep up with it.
Kye said...
The FAT file system spec mandates that you do so when allocating a new cluster so that all remaining bytes are zeros. You must do this when making a new cluster for directories since a zero byte at the start of a dir entry indicates the end of the directory. For files it is not so neccesary... but I am using the same function for both. I could add in an option for the function so that it does not do that for files. This would increase write speed.
Kye could you please add this functionality as the write speed I am experiencing with this new snazzy release is considerably lower* than the fsrw predecessor. This lower write speed is forcing me to continue to use the fsrw version on my current project as SD write speed will become the system throughput bottleneck. Every bit helps but >= 2X speed increase would remove the bottleneck.
*Card being used...4GB Sandisk micro HD ....fsrw write speed ~250,000 bytes per second, SD2.0 27,000 bytes per second...both preallocation mode, same propeller system @ 40 Mhz (marginal increases in write speeds when bumping up to 80 Mhz). Similar results with other cards also...really a bummer as the new functionality in this library is outstanding.
Kye, your FAT work is amazing, and is very well commented, which is really great.
Have you ever considered dividing the filesystem management code and the device sector access code into separate modules?
If you were to do that, then your FAT access code could be run on top of different devices, such as hard disks or even external RAM, as long as the device-driver modules provided initialization and shutdown routines and addressed-sector read and write routines.
Also, with a separate module, the low level SD card sector access module could be used directly by those who really need the fastest write speeds, such as for raw data loggers.
Or, lonesock's block driver could simply be plugged in to your FAT module, yielding the best of both worlds!
What do you think about that idea?
It's great how your SD device access code is so well commented, and it's also great that your code reads the SD size from the SD CSD register, which a raw data logger could find useful when it sets up its legal logging write addresses.
And a separate question in regards to attaining the fastest write speeds - I heard somewhere that if SD card raw sectors are explicitely pre-erased, then its data sector write operation will detect that condition and the actual data writes will go much faster, because they can then skip the internal erase step of the write process. I don't know if that's just a urban legend from the early days of SD development, or if it's still true.
Kye, during all your SD work, have you heard anything like that?
To test it, the raw SD erase command would have to be implemented, which neither your code nor the mb_raw_spi module currently does. Also, it looks like the erase command needs to be configured using some CSD register values, so it's not a trivial command to implement.
Kye said...
Yeah, I know, I wish I had lonesock to help me with the block driver.
Okay, I'll see what I can do.
Sorry for not responding earlier. In my safe driver (assuming 80MHz operation) I have the output at 20MHz, using one counter to drive a clock pin, and the second counter as a shift-out-register. You don't get to vary the clock pin speed, though, it must always be 1/4 the propeller's clock speed, that way each clock can line up to a single PASM instruction (usually a ROL on my shift register, PHSA). For safe input I'm using 10MHz (or, more correctly, prop frequency / 8). The fast version uses a 20MHz clock pin as well, but it doesn't work with all hardware at all times, so we default to the 10MHz for input. (btw, MMC init code can't happen at 20MHz, according to the spec, so I use a Spin version, which is plenty slow, and not too large.)
A second main trick for speed is the read-ahead and write-behind code. I have a single buffer (128 longs in the cog, = 512 bytes). When you transfer a block to be written, it get's copied into the cog buffer, then control returns to the caller, and the writing to the SD card actually happens in the background without tying up the calling cog. When you request a read block, the block is read, passed to the caller, then the next sequential block is read and stored in the buffer. At the next read call, the buffer is checked to see if the requested block actuall was the next one in the sequence...if so, it's already in the buffer, which is copied out, and the next sequential block is read. This takes advantage of the normal usage pattern of the reads and writes. You just have to be aware of the state of the driver, and you may have to block calls until the driver isn't busy.
Finally, I use the multiblock mode because most SDHC cards are terribly slow in single-block mode. YOur code may not be the bottleneck. Sometimes the wait for the card to signal ready after requesting a single block read or write is loooooong. You may want to add some profiling to see if this is the case.
Jonathan
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
lonesock
Piranha are people too.
Okay, so here's a version where I hooked in the FSRW block drivers...
The 512 bytes at a time read speed gain is about 150%. And the write speed gain is about 10%. So... I have no idea why I can't get faster write speeds. The cluster size for the test card was 128 sectors, so this is the most optimum senario for multiblock read and write.
Anyway, look through this version for clues on how to make it faster.
@David B - Yes I have heard that pre erasing will increase speed. But I have no space for that in my code.
While everyone wants really fast read and write speeds... my goal is to have safe/compatible/ and reliable operation while providing alot of features. So, for now. use FSRW when you need massive speed. I don't think my file system will be able to offer it currently.
I've tested your code on a few uSD cards & they all worked without problems (attached are the results), fantastic. Just for fun, I've done a few formats on a single 4GB card in 2k, 4k, 8k & 16k allocated units. The speed seems to go up but at the expenses of many things like space & the life-span of the uSD. Appreciates your great work.
@All - Okay I think the reason I receive no speed gain using the FSRW drivers for write mode is because I read in every sector before I write it out. This makes multiblock mode write useless.
Hmm, just been trying this - which seems excellent BTW - and got it working with a Kingston 1GB card, but not with a Canon 16M card. I tried FSRW2.6 to see if it's the card, but that fails when trying to write to the filesystem too.
I'm guessing that the card's not formatted correctly - although I formatted both in the same camera. Does anyone know what "format" command I should use (from Windows7) please?
I used Win7 to format the card as well. Right-click on the card & select format (uncheck Quick Format). The default is 4k allocation unit, I think. I've did tried 2k, 4k, 8k & 16k and all worked but it is better to stick to win's default 4k.
Comments
Ken
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,
Is that using the serial driver or the local keyboard? I had the same problem on the serial driver, as the buffer has one or two bytes in it - I think from when it is connected or from the terminal program at the other end. The problem seemed to go away when I changed to local keyboard. It might be a matter of clearing the buffer.
@OBC, TV would be the next logical thing. Do you have some code?
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.smarthome.viviti.com/propeller
Thanks for the great code you wrote!!!!!
>_ Running command: test
Creating test file "testfile" ... Success
Wrote 131,072 bytes at 0000056563 bytes per second
Running byte stride write test...
Wrote 32,768 bytes at 0000004227 bytes per second
Running byte stride read test...
Read 32,768 bytes at 0000004297 bytes per second
Running word stride write test...
Wrote 65,5536 bytes at 0000008105 bytes per second
Running word stride read test...
Read 65,5536 bytes at 0000008454 bytes per second
Running long stride write test...
Wrote 131,072 bytes at 0000014883 bytes per second
Running long stride read test...
Read 131,072 bytes at 0000016804 bytes per second
Running speed writing test (512 bytes at a time)...
Wrote 131,072 bytes at 0000080496 bytes per second
Running speed reading test (512 bytes at a time)...
Read 131,072 bytes at 0000250432 bytes per second
Running append test... Success
Running seek test... Success
Deleting test file "testfile" ... Success
Micoro SD HC 4GB
The first prompt not working problem I have is from local kbd, I have not tried the remote way yet. Although the '232 was connected all the time to the PC. I will check that out when Clunky gets back on the bench (playing with AVRs, sorry, sorry, ... )
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Why did I think a new, more challenging, job was a good idea ??
I can jump over from my current project to connecting Doug's 8x8 driver to this sometime this week.
(Unless you've already started.)
BTW, Your Dracoboards arrived today.. Not sure if I can get the parts quick enough before I go to UPEW, but I'm going to try.
(Nice Boards!!!!)
OBC
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Propeller Feature Projects: PropellerPowered.com
Visit the: PROPELLERPOWERED SIG forum kindly hosted by Savage Circuits.
@potatohead mentions it in another thread. Without long filenames your work is not useful for my next Java project.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Propeller Pages: Propeller JVM
But, you can just store the long name of the file inside the file as the first 256 bytes. Then make the actual names of the files on disk just hash codes that should match the long name of the file when asked for... or something like that.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,
This time there is no "0" bytes per second read. I guess the one in my first post was due to some overrun?
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Pullmoll's Propeller Projects
>_ mount
Running command: mount
Disk 0x00000000 with volume ID 0xFC303DA9 a.k.a ready
>_ test
Running command: test
Creating test file "testfile" ... Success
Wrote 131,072 bytes at 0000056983 bytes per second
Running byte stride write test...
Wrote 32,768 bytes at 0000004262 bytes per second
Running byte stride read test...
Read 32,768 bytes at 0000004297 bytes per second
Running word stride write test...
Wrote 65,5536 bytes at 0000008105 bytes per second
Running word stride read test...
Read 65,5536 bytes at 0000008489 bytes per second
Running long stride write test...
Wrote 131,072 bytes at 0000014988 bytes per second
Running long stride read test...
Read 131,072 bytes at 0000016874 bytes per second
Running speed writing test (512 bytes at a time)...
Wrote 131,072 bytes at 0000080496 bytes per second
Running speed reading test (512 bytes at a time)...
Read 131,072 bytes at 0000268320 bytes per second
Running append test... Success
Running seek test... Success
Deleting test file "testfile" ... Success
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Propeller Pages: Propeller JVM
Perhaps that's a very easy answer Jazzed.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Propeller Wiki: Share the coolness!
8x8 color 80 Column NTSC Text Object
Wondering how to set tile colors in the graphics_demo.spin?
Safety Tip: Life is as good as YOU think it is!
I think I'll have to give it a try.
I'm a little worried though because some people have reported that some cards didn't work...
Do you think that is just because they are older cards?
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
My Prop Info&Apps: ·http://www.rayslogic.com/propeller/propeller.htm
My Prop Products:· http://www.rayslogic.com/Propeller/Products/Products.htm
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,
Jonathan
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
lonesock
Piranha are people too.
Mabye I'm wrong.
Also, How did you reach those speeds? Man I tried but I could not get the 20Mhz clock stuff to work.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,
What are the capabilities of the WAV play and record functions in the SD2.0 object? What's the best "quality" I can record and what's the best "quality" that I can convert on my PC to have the propeller playback?
A friend of mine used my VMusic2 in a project for his autistic son that I believe could be easily done with this new object, but the audio needs to be clean on a 2" speaker. Obviously, with a 2" speaker I don't need CD quality, but telephone quality wouldn't be desirable as a finished project. The dis-advantages of the current design is that the audio files (mostly voice) have to be pre-recorded as MP3s which makes it less user friendly and requires a PC. It would be nice to be able to record the audio clips directly on the device.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Andrew Williams
WBA Consulting
WBA-TH1M Sensirion SHT11 Module
My Prop projects: Reverse Geo-Cache Box, Custom Metronome, Micro Plunge Logger
The record feature records wav files that also conform to the layout on the posted link. The only limiting factor with it is that some SD cards with my driver have very bad write speeds. So... I set the recording feature down to 8Khz with 8 bit sampling.
However, the ADCEngine can go up to 80Khz at 16 bits if the cards transfer speed can keep up with it.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,
Kye could you please add this functionality as the write speed I am experiencing with this new snazzy release is considerably lower* than the fsrw predecessor. This lower write speed is forcing me to continue to use the fsrw version on my current project as SD write speed will become the system throughput bottleneck. Every bit helps but >= 2X speed increase would remove the bottleneck.
*Card being used...4GB Sandisk micro HD ....fsrw write speed ~250,000 bytes per second, SD2.0 27,000 bytes per second...both preallocation mode, same propeller system @ 40 Mhz (marginal increases in write speeds when bumping up to 80 Mhz). Similar results with other cards also...really a bummer as the new functionality in this library is outstanding.
Okay, I'll see what I can do.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,
Have you ever considered dividing the filesystem management code and the device sector access code into separate modules?
If you were to do that, then your FAT access code could be run on top of different devices, such as hard disks or even external RAM, as long as the device-driver modules provided initialization and shutdown routines and addressed-sector read and write routines.
Also, with a separate module, the low level SD card sector access module could be used directly by those who really need the fastest write speeds, such as for raw data loggers.
Or, lonesock's block driver could simply be plugged in to your FAT module, yielding the best of both worlds!
What do you think about that idea?
It's great how your SD device access code is so well commented, and it's also great that your code reads the SD size from the SD CSD register, which a raw data logger could find useful when it sets up its legal logging write addresses.
And a separate question in regards to attaining the fastest write speeds - I heard somewhere that if SD card raw sectors are explicitely pre-erased, then its data sector write operation will detect that condition and the actual data writes will go much faster, because they can then skip the internal erase step of the write process. I don't know if that's just a urban legend from the early days of SD development, or if it's still true.
Kye, during all your SD work, have you heard anything like that?
To test it, the raw SD erase command would have to be implemented, which neither your code nor the mb_raw_spi module currently does. Also, it looks like the erase command needs to be configured using some CSD register values, so it's not a trivial command to implement.
A second main trick for speed is the read-ahead and write-behind code. I have a single buffer (128 longs in the cog, = 512 bytes). When you transfer a block to be written, it get's copied into the cog buffer, then control returns to the caller, and the writing to the SD card actually happens in the background without tying up the calling cog. When you request a read block, the block is read, passed to the caller, then the next sequential block is read and stored in the buffer. At the next read call, the buffer is checked to see if the requested block actuall was the next one in the sequence...if so, it's already in the buffer, which is copied out, and the next sequential block is read. This takes advantage of the normal usage pattern of the reads and writes. You just have to be aware of the state of the driver, and you may have to block calls until the driver isn't busy.
Finally, I use the multiblock mode because most SDHC cards are terribly slow in single-block mode. YOur code may not be the bottleneck. Sometimes the wait for the card to signal ready after requesting a single block read or write is loooooong. You may want to add some profiling to see if this is the case.
Jonathan
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
lonesock
Piranha are people too.
The 512 bytes at a time read speed gain is about 150%. And the write speed gain is about 10%. So... I have no idea why I can't get faster write speeds. The cluster size for the test card was 128 sectors, so this is the most optimum senario for multiblock read and write.
Anyway, look through this version for clues on how to make it faster.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,
While everyone wants really fast read and write speeds... my goal is to have safe/compatible/ and reliable operation while providing alot of features. So, for now. use FSRW when you need massive speed. I don't think my file system will be able to offer it currently.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,
I've tested your code on a few uSD cards & they all worked without problems (attached are the results), fantastic. Just for fun, I've done a few formats on a single 4GB card in 2k, 4k, 8k & 16k allocated units. The speed seems to go up but at the expenses of many things like space & the life-span of the uSD. Appreciates your great work.
@All - Okay I think the reason I receive no speed gain using the FSRW drivers for write mode is because I read in every sector before I write it out. This makes multiblock mode write useless.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,
If this works as advertised it could solve several problems that we have.
Russ
I'm guessing that the card's not formatted correctly - although I formatted both in the same camera. Does anyone know what "format" command I should use (from Windows7) please?
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Cheers,
Simon
www.norfolkhelicopterclub.com
Announcement: To cut costs in the current economic climate, we have switched-off the light at the end of the tunnel.
I used Win7 to format the card as well. Right-click on the card & select format (uncheck Quick Format). The default is 4k allocation unit, I think. I've did tried 2k, 4k, 8k & 16k and all worked but it is better to stick to win's default 4k.
The Disk I/O error is not fixable if it keeps happening. The file system unsupported error means that the card is formated uncompatibly.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,