FSRW (uSD card read/write) for P2 with 2.4 MB/s read speed

Rayman

Here is a version of FSRW that works with both FastSpin and the Parallax Spin compiler (AKA PNut and soon to be PropTool for P2).

Thanks to @cheezus for providing the base from which this came from as discussed here: http://forums.parallax.com/discussion/169786/
Thanks to input from @Cluso99 and @"Peter Jakacki" for showing me how to improve read speed.
And, of course thanks to Tomas Rokicki and Jonathan Dummer for providing the original P1 FSRW 2.6 from where this all came from...

This driver works very much like the P1 version, with some small differences:
- Mount() used to return 0 on success and negative on failure. Now, it returns 1 or 2 on success to tell you card type (1==SD and 2==SDHC).
- There are two versions of the low level SPI driver, sdspi, to select from in Fsrw.spin2. The "bashed" version does not use a separate cog, while the other, "ASM" one does. The "ASM" version is about 10% or so faster in some tests.
Note: If you choose the assembly version, edit the "sdspi_asm_mb.spin2" driver if you want to change the cog# being used. This version has "SPI_COG=6".
- There are two new functions FastBlocksRead() and FasterBlocksRead() that blindly read a series of blocks without following the FAT chain. This can be useful for maximizing read speed of large files on freshly formatted disks. Both of these options require the use of the new remount() function to switch files.

The read speed can be tested using the "fsrw_test2.spin2" file in the attached. There are three different read options to choose from.
This code repeatedly loads and displays the file bitmap2.bmp over VGA and sends diagnostic info over the USB serial connection. This is useful for checking for read errors.
With the FasterBlocksRead() option, I'm seeing about 2000 kB/s at 250 MHz clock and 2400 kB/s at 300 MHz clock.
This is fast enough to do QVGA video demonstrated by the included file, "P2VideoPlayer.spin2", as described here: http://www.rayslogic.com/Propeller2/P2Video/P2Video.html

This is by no means complete, there are still a lot of improvements that can be made. Any improvements or suggestions are welcome.
I don't know if I'll be the one doing any updates or not as it seems to fill my needs as is now.
Also, I haven't really tested the writing part of this very much, mostly just been focused on reading.

25May20: Fixed a timeout issue with the bashed version that would cause opening a new file to fail when a long time passed since opening the first file. Also, added in a basic timeout to the assembly version.
29May20: removed a PINL(56) and a PINL(57) in fsrw.spin2 that was used for debugging.

Rayman

Note that one big advantage of this FSRW, like the original, is that it is tiny.
One disadvantage, also like the original, is that it only works with 8.3 DOS style filenames and also only in the root directory.

However, you can still open files with long filenames if you can figure out the 8.3 version of the filename.
In Windows, you can bring up the cmd DOS window and do "dir /x" to see both long and short filenames.
I just googled this to remember how to do it and got this from here: https://digitalsupport.ge.com/en_US/Article/Is-Windows-8-3-File-Naming-Enabled-How-Do-I-Enable-8-3-File-Naming-If-It-Is-Not

Tried this on the attached and it showed up as "LARGEF~1.bmp". Quick test shows this does work (see screenshot of fsrw_test2.spin with the filename changed to this).

Peter Jakacki

Rayman wrote: »

Note that one big advantage of this FSRW, like the original, is that it is tiny.
One disadvantage, also like the original, is that it only works with 8.3 DOS style filenames and also only in the root directory.

Other directories in the root can be opened just like a file except the first sector points to the new directory. So it is rather simple to navigate directories if you use a pointer to the directory which is initially loaded with the root. 8.3 file names are very efficient for embedded systems although I wish they came up with almost anything else but the mangled horrible mess that LFN is.

Here is a quick look at TAQOZ that stores the current working directory sector in the cwdir variable, and checking the sector where it is pointing.

TAQOZ# cwdir @ .l --- $0000_F70C ok
TAQOZ# $F70C OPEN-SECTOR ---  ok
TAQOZ# 0 $80 SD DUMP --- 
00000: 52 4F 4F 54  44 49 52 20  20 20 20 08  00 00 00 00     'ROOTDIR    .....'
00010: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
00020: 54 41 51 4F  5A 20 20 20  52 4F 4D 20  00 00 00 00     'TAQOZ   ROM ....'
00030: 21 00 AE 50  00 00 00 00  21 00 03 00  00 00 02 00     '!..P....!.......'
00040: 35 31 32 4B  20 20 20 20  52 4F 4D 20  00 00 00 00     '512K    ROM ....'
00050: 00 00 00 00  00 00 00 00  00 00 07 00  00 00 08 00     '................'
00060: 45 41 53 59  4E 45 54 20  46 54 48 20  00 00 00 00     'EASYNET FTH ....'
00070: 00 00 00 00  00 00 00 00  00 00 17 00  00 00 01 00     '................' ok

whicker

It's not a mangled horrible mess. It had to be that way because of forward and backward compatibility with DOS. There was never a good time to start over on LFN in a way that caused older PC's to be unable to read the files.

Wuerfel_21

whicker wrote: »

It's not a mangled horrible mess. It had to be that way because of forward and backward compatibility with DOS. There was never a good time to start over on LFN in a way that caused older PC's to be unable to read the files.

Well, there was. It's called NTFS. Except that by default, an NTFS volume will inexplicably still keep track of 8.3 names for everything for DOS compatibility. (even though the DOS emulation in Windows NT and 9x has always had LFN, anyways?)

This means you can do this:

C:\temp>echo Hello World > testabcde.txt

C:\temp>type testabcde.txt
Hello World

C:\temp>type testab~1.txt
Hello World

Highly amusing.

Cluso99

I've long known that the setup to read or write a sector is significant, and also depends on the SD card being used.
Today I did a couple of tests and the results were interesting...

The card is a SanDisk Ultra 16GB micro SDHC I C10.

These are the times to setup a read of sector 0 of the SD card after it had been initialised with commands 0,8,55,A41,58,16. The CSD & CID have not been read. The setup is in system clocks from after the SPI pins were set as output, but before any command was sent.

Timer starts, then Command 17 with sector=00000000 was sent to the card along with the CRC byte. Once the card had replied following the busy bytes (ie $FE received) the timer was stopped. This time excludes the actual read of the 512 bytes and crc16 of 2 bytes.

This was then repeated a second time to show the first time and subsequent time to access the same sector. These two times are significantly different as expected.

For P2 system clock frequencies of 100MHz, 200MHz, 300MHz and 360MHz the times were essentially identical.
First read sector setup time 4.18ms
Second read sector setup time 2.7ms

From this I deduce that the method of coding for the setup is almost immaterial. Only the actual code reading or writing the sector data bytes has any bearing on the throughput. The setup time is determined by the SD Card being used.

Wuerfel_21

Cluso99 wrote: »

I've long known that the setup to read or write a sector is significant, and also depends on the SD card being used.
Today I did a couple of tests and the results were interesting...

The card is a SanDisk Ultra 16GB micro SDHC I C10.

These are the times to setup a read of sector 0 of the SD card after it had been initialised with commands 0,8,55,A41,58,16. The CSD & CID have not been read. The setup is in system clocks from after the SPI pins were set as output, but before any command was sent.

Timer starts, then Command 17 with sector=00000000 was sent to the card along with the CRC byte. Once the card had replied following the busy bytes (ie $FE received) the timer was stopped. This time excludes the actual read of the 512 bytes and crc16 of 2 bytes.

This was then repeated a second time to show the first time and subsequent time to access the same sector. These two times are significantly different as expected.

For P2 system clock frequencies of 100MHz, 200MHz, 300MHz and 360MHz the times were essentially identical.
First read sector setup time 4.18ms
Second read sector setup time 2.7ms

From this I deduce that the method of coding for the setup is almost immaterial. Only the actual code reading or writing the sector data bytes has any bearing on the throughput. The setup time is determined by the SD Card being used.

I've also got a 16 GB SanDisk Ultra - does yours have the little A1 symbol? (A-class is more relevant to setup time (and speed in general - to actually write at the specced speed a special "speed class recording" mode is required) than the old speed classes. A1 also implies Class 10)

Anyways, some things I've noticed:
- If the card is left idle for a while (is less than one NTSC field, so maybe 10 ms?) the first access afterwards has much longer setup time.
- Enabling performance enhancement features (notably on-card write-behind cache) is not possible in SPI mode because CMD49 is not allowed in SPI mode.

Peter Jakacki

I decided to test that observation about leaving it idle and repeating that exercise in various one-liners and it is true that if it has been idle for a while, it takes much longer. I measured "longer" as around 2.5ms and otherwise it was more like 270us. But how long was a while?
Well, if you can follow my code you will see the results say that without checking further yet, it is between 5ms and 10ms.

Here's the code and the delay is in us starting from 100 then 500, 5000, and 10000 as it rereads the same sector 3 times.

TAQOZ# .CARD --- SANDISK   SD SC64G REV$80 #35190404 DATE:2018 /10 ok
TAQOZ# pub ?LATENCY ( sector usdly -- ) 3 FOR OVER CRLF .LATENCY DUP us NEXT 2DROP ; ---  ok
TAQOZ# $100.0000 100 ?LATENCY --- 
691,960 cycles= 2,306,533ns @300MHz
81,704 cycles= 272,346ns @300MHz
81,704 cycles= 272,346ns @300MHz ok
TAQOZ# $100.0000 500 ?LATENCY --- 
690,928 cycles= 2,303,093ns @300MHz
81,016 cycles= 270,053ns @300MHz
81,016 cycles= 270,053ns @300MHz ok
TAQOZ# $100.0000 5000 ?LATENCY --- 
691,616 cycles= 2,305,386ns @300MHz
81,704 cycles= 272,346ns @300MHz
81,704 cycles= 272,346ns @300MHz ok
TAQOZ# $100.0000 10000 ?LATENCY --- 
691,960 cycles= 2,306,533ns @300MHz
691,960 cycles= 2,306,533ns @300MHz
691,960 cycles= 2,306,533ns @300MHz ok

I tried sectors from all over and they responded in similar manner and the latency figure is not technically correct since it also includes the sector read time, but the actual sector transfer time is measured as 150us.
The point at which it did dither was around the 6ms mark. I'd say that the SD control chip is saving power perhaps? I wonder if we always kept it busy whether it would exhibit the same behavior. More tests.

Here's the cards CSD report:

*** CSD *** 
    CARD TYPE....................... SDHC
    LATENCY......................... 1ms+1400 clocks 
    SPEED........................... 50MHz 
    CLASSES......................... 0 1 0 1 1 0 1 1 0 1 0 1 
    BLKLEN.......................... 512
    SIZE............................ 62,367MB
    Iread Vmin...................... 10ma
    Iread Vmax...................... 25ma
    Iwrite Vmin..................... 60ma
    Iwrite Vmax..................... 35ma

whicker

Wuerfel_21 wrote: »

whicker wrote: »

It's not a mangled horrible mess...

Well, there was. It's called NTFS.
...
This means you can do this:

C:\temp>echo Hello World > testabcde.txt

C:\temp>type testabcde.txt
Hello World

C:\temp>type testab~1.txt
Hello World

Highly amusing.

Yes, that's how it's supposed to work! And the squiggly means you're dealing with a truncated file name.

You can mix FAT and NTFS volumes on the same PC. So installshield scripts, BAT files, rundll, open and save dialog boxes, etc. All work as expected.

Yes you can think of ways to break it (use a Linux tool to write invalid characters and symbols to the filename), but it worked great for decades.

Electrodude

whicker wrote: »

Wuerfel_21 wrote: »

whicker wrote: »

It's not a mangled horrible mess...

Well, there was. It's called NTFS.
...
This means you can do this:

C:\temp>echo Hello World > testabcde.txt

C:\temp>type testabcde.txt
Hello World

C:\temp>type testab~1.txt
Hello World

Highly amusing.

Yes, that's how it's supposed to work! And the squiggly means you're dealing with a truncated file name.

You can mix FAT and NTFS volumes on the same PC. So installshield scripts, BAT files, rundll, open and save dialog boxes, etc. All work as expected.

Yes you can think of ways to break it (use a Linux tool to write invalid characters and symbols to the filename), but it worked great for decades.

The fact that they had to do it that way for compatibility reasons doesn't somehow imply that it isn't a mangled horrible mess. Unfortunately, attempts at compatibility turn into horrible messes all too often.

Rayman

I just found some issues with the timeout functionality...

It appears to be disabled in the assembly version and missing a start time initialization in the bashed version that can cause opening a new file to fail...
This looks easy to fix, so I'll work on that now.

Rayman

Think I fixed the bashed and added a basic timeout to the assembly...

Rayman

Shoot. Just when I thought I was done with this...
I found a PINL(56) and a PINL(57) in there that I was using for troubleshooting.
Can't let that stand...

Found whilst turning into a eMMC driver...

This one has that removed.

dMajo

Rayman wrote: »

Note that one big advantage of this FSRW, like the original, is that it is tiny.
One disadvantage, also like the original, is that it only works with 8.3 DOS style filenames and also only in the root directory.

However, you can still open files with long filenames if you can figure out the 8.3 version of the filename.
In Windows, you can bring up the cmd DOS window and do "dir /x" to see both long and short filenames.
I just googled this to remember how to do it and got this from here: https://digitalsupport.ge.com/en_US/Article/Is-Windows-8-3-File-Naming-Enabled-How-Do-I-Enable-8-3-File-Naming-If-It-Is-Not

Tried this on the attached and it showed up as "LARGEF~1.bmp". Quick test shows this does work (see screenshot of fsrw_test2.spin with the filename changed to this).

I always set the DIRCMD environment (system not user) variable to /OGN /X to have always this switches active. This is also deployed with GPO on all the clients machines where I administer the IT

Peter Jakacki

I always wish there was a simpler not-so-long-and-awkward file names for FAT32. 8.3 is a bit on the short side, but a length of 31 would be fine and could hide in a directory entry slot IMO. Anyway, this thing about only working with the root directory is easily fixed because the software already needs a pointer to the root sector anyway, and directory entries are just "files" that I find the start sector for and change the directory pointer to point to this instead. Mind you, even if you don't keep a copy of the root sector to make it easy to switch to, it is still easy to find it in the FAT info.

Dave Hein

The version of FSRW that was used in p2gcc handles subdirectories. The code is on GitHub at https://github.com/davehein/p2gcc/blob/master/lib/fsrw.c . It uses the C version of FSRW that was included with FSRW 2.6. The new code that I added starts after line 1000 in fsrw.c. In addition to the subdirectory code I also added support for file handles so that I could have many open files.

If you prefer using Spin, the CLib object in the OBEX contains a Spin version of FSRW that support subdirectories and handles. It is located at https://github.com/parallaxinc/propeller/tree/master/libraries/community/p1/All/CLib .

Peter Jakacki

I think you can modiy the Spin version easily too. In my filesystem I use a "cwdir" pointer to the directory sector. When I "cd" into another directory all it does is find the starting sector of that "file" and change the cwdir pointer. Have a look at my console output, especially the DIR listing which includes the starting sector in the listing. I even examine the cwdir at each point too for some clarity.

TAQOZ# MOUNT ---  CARD: SANDISK   SD SC64G REV$80 #35190404 DATE:2018 /10 ok                            
TAQOZ# DIR ---                                                                                          
  0: ROOTDIR      08                                                                                    
  1: TAQOZ   .ROM 20 $0000_F74C   1980-01-24 00:00        131,072 /       131,072                       
  2: 512K    .ROM 20 $0000_F84C   1980-00-24 00:00        524,288 /       524,288                                                             
<snip>                     
 40: PBCANON .WAV 20 $0001_C94C   2006-01-24 12:20      3,318,188 /     3,342,336                       
 44: TAQOZ        10 $0001_E2CC   2020-05-24 01:42              0 /        32,768  ok                   
TAQOZ# cwdir @ .L --- $0000_F70C ok                                                                     
TAQOZ# CD TAQOZ Opened @123596 ---  ok                                                                  
TAQOZ# DIR ---                                                                                          
  0: .            10 $0001_E2CC   2020-05-24 01:40              0 /        32,768                       
  1: ..           10 $0000_F68C   2020-05-24 01:40              0 /        98,304                       
  5: FORTH        10 $0001_E30C   2020-05-24 01:41              0 /        32,768                       
  6: BINARIES     10 $0001_E88C   2020-05-24 01:47              0 /        32,768  ok                   
TAQOZ# cwdir @ .L --- $0001_E2CC ok                                                                     
TAQOZ# CD FORTH Opened @123660 ---  ok                                                                  
TAQOZ# DIR ---                                                                                          
  0: .            10 $0001_E30C   2020-05-24 01:41              0 /        32,768                       
  1: ..           10 $0001_E2CC   2020-05-24 01:41              0 /        32,768                       
  2: EASYFILE.FTH 20 $0001_E34C   2020-05-24 00:54         42,292 /        65,536                       
  3: TAQOZ   .FTH 20 $0001_E3CC   2020-05-24 00:17         46,531 /        65,536                       
  4: EASYNET .FTH 20 $0001_E44C   2020-05-24 13:21         40,492 /        65,536                       
  5: TIM     .FTH 20 $0001_E4CC   2020-05-24 12:26         50,165 /        65,536     
<snip>                     
 17: PLEXLED4.FTH 20 $0001_E80C   2020-04-24 05:44          2,113 /        32,768                       
 18: MORSER  .FTH 20 $0001_E84C   2020-04-24 05:18          2,603 /        32,768  ok                   
TAQOZ# cwdir @ .L --- $0001_E30C ok                                                                     
TAQOZ#

BTW, the "ls -l" command in TAQOZ prints out a more standard listing which indicates the names that are directories, I should update DIR to do the same.

-rwxrwxrwx 1 502     500    3318188 Jan 12  12:20 PBCANON.WAV                                           
-rwxrwxrwx 1 502     500          0 May 30  01:42 [TAQOZ]

Dave Hein

Peter Jakacki wrote: »

I think you can modiy the Spin version easily too.

Peter, the CLib version is written in Spin.

Rayman

It seems the pwrite function is broken in the pure assembly version, but not the inline assembly version.

Trying to figure out what's going on and if it ever worked...

Seems I switched Cheezus's code from Spartpin mode to bit bang mode and broke writing in the process. Guess I was so focused on read speed, didn't pay attention...

Cheezus's assembly code version still works for writing though....