P2 SD Wrangling Tools

TubularTubular Posts: 3,804
edited 2019-12-03 - 02:40:11 in Propeller 2
I'm developing a P2 product that I hope can just work from an SD card, without the boot flash. Turns out there's a bit of work involved in getting all this to be nice and reliable.

Windows 10's formatter only offers exFAT for SD cards above 32 GB, so I thought I'd start a thread to cover SD wrangling tools people use to get by. Of course many cards already come in FAT32 out of the packet, but I've found sometimes a reformat is necessary to restore things.

Feel free to chime in with tools you've used (eg with rasp Pi), and what their caveats are. I'm curious what people on Mac/Linux use

SD Card Formatting
* Taqoz - P2! - Peter's extended Taqoz has a FAT32 formatter.
* "Format" - Windows 10) - Right click on drive from in explorer etc. 32 GB cards are the biggest it'll format with FAT32
* GUIFORMAT - Windows - runs under windows without installation. Also known as Fat32format?
* SD formatter - Windows/Mac - recommended to me by Jeff Martin. Sandisk/SD Association tool.

SD Image Burning
* Win32DiskImager - Windows - Used with Rasp Pi images. Whole drive image so takes a while to run. Don't like the proximity of 'device' to the image filename, could potentially confuse drives

Comments

  • Peter JakackiPeter Jakacki Posts: 8,795
    edited 2019-12-03 - 03:31:22
    Yes, there certainly is a problem with the Window's format utility not only in that you cannot specify that it format a >32GB card to FAT32 (or any other format), but also in that it does not comply with the format method recommended by the SD card association where several MB is left free before the start of the partition for improved SD card operation. Most format tools just format a Flash device the same as a hard-drive without taking this into consideration.

    TAQOZ has the FAT32 formatter which I have tested on cards up to 128GB (so far) and can specify the cluster size etc. You can even format just one part such as the MBR for instance which has pulled me out of a sticky jam a few times already. Also as part of the formatter it includes disk tools that can not only report details about the card, but also run tests and handle the card as one great big memory chip for the first 4GB, or as a sector device, or individual files as virtual memory up to 4GB which can be handled as any other memory (read, write, dump, list etc).

    Here is a terminal dump where I format a 64GB card, first without permissions, and then with read/write/system permissions after which I dump the first sector as if it were an open file, manipulate it and dump it again.
    TAQOZ# FORMAT ---  FAILED! ok
    TAQOZ#  ---  ok
    TAQOZ# RWS FORMAT ---  CARD: SANDISK   SD SC64G REV$80 #38874969 DATE:2019/8 CARD: SANDISK   SD SC64G REV$80 #38874969 DATE:2019/8
    
                       *** OCR *** 
        VALUE........................... $C0FF_8000
        RANGE........................... 2.7V to 3.6V
    
                       *** CSD *** 
        CARD TYPE....................... SDHC
        LATENCY......................... 1ms+1400 clocks 
        SPEED........................... 50MHz 
        CLASSES......................... 010110110101
        BLKLEN.......................... 512
        SIZE............................ 62,367MB
        Iread Vmin...................... 10ma
        Iread Vmax...................... 25ma
        Iwrite Vmin..................... 60ma
        Iwrite Vmax..................... 35ma
    
                     *** SPEEDS *** 
        LATENCY......................... 446us,131us,131us,131us,132us,130us,131us,131us,
        SECTOR.......................... 355us,287us,287us,286us,287us,287us,286us,287us,
        BLOCKS.......................... 3,148kB/s @300MHz
    
                       *** MBR *** 
        PARTITION....................... 0 00 INACTIVE
        FILE SYSTEM..................... FAT32 LBA
        CHS START....................... 1023,254,63
        CHS END......................... 0,0,0
        FIRST SECTOR.................... $0000_8000
        TOTAL SECTORS................... 124,702,720 = 63,847MB
    
    00170: 0000_0000 0000_0000 0000_0000 0000_0000     '................'
    
                      *** FAT32 *** 
        OEM............................. TAQOZ P2
        Byte/Sect....................... 512
        Sect/Clust...................... 128 = 65MB
        FATs............................ 2
        Media........................... F8
        Sect/Track...................... $003F
        Heads........................... $00FF
        Hidden Sectors.................. 32,768 = 16MB
        Sect/Part....................... 124,702,720 = 63,847MB
        Sect/FAT........................ 7,611 = 3MB
        Flags........................... 0
        Ver............................. 00 00 
        ROOT Cluster.................... $0000_0002 SECTOR: $0000_BB96
        INFO Sector..................... $0001 = $0000_8001
        Backup Sector................... $0006 = $0000_8006
        res............................. 00 00 00 00 00 00 00 00 00 00 00 00 
        Drive#.......................... 128
        Ext sig......................... $29 OK!
        Part Serial#.................... $6269_0201 #1651048961
        Volume Name..................... P2 CARD    FAT32    ok
    TAQOZ# 0 OPEN-SECTOR ---  ok
    TAQOZ# 0 $200 SD DUMP --- 
    00000: 64 12 80 FD  50 32 44 32  72 33 20 20  03 00 00 00     'd...P2D2r3  ....'
    00010: 00 2D 31 01  00 A3 E1 11  FB 0E 00 01  00 C2 01 00     '.-1.............'
    00020: 30 31 32 33  34 00 00 00  4C 0C 00 00  00 F8 E6 11     '01234...L.......'
    00030: 35 36 00 00  C4 04 00 00  34 22 00 00  F2 22 00 00     '56......4"..."..'
    00040: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    00050: 00 00 00 00  00 00 00 00  00 00 00 00  8C 00 8C 00     '................'
    00060: 0D 00 78 7B  00 00 00 00  00 00 00 00  00 00 00 00     '..x{............'
    00070: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    00080: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    00090: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    000A0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    000B0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    000C0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    000D0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    000E0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    000F0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    00100: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    00110: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    00120: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    00130: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    00140: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    00150: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    00160: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    00170: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    00180: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    00190: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    001A0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    001B0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 FE     '................'
    001C0: FF FF 0C 00  00 00 00 80  00 00 00 D0  6E 07 00 00     '............n...'
    001D0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    001E0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
    001F0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 55 AA     '..............U.' ok
    TAQOZ# " HELLO WORLD" $100 SDADR $! FLUSH ---  ok
    TAQOZ# 0 $200 SD DUMPAW --- 
    00000: d...P2D2r3  .....-1.............01234...L.......56......4"..."....................................x{............................
    00080: ................................................................................................................................
    00100: HELLO WORLD.....................................................................................................................
    00180: ............................................................................n.................................................U. ok
    TAQOZ#
    


    btw, that funny little operation that I did to write a string could just as easily have had function for it with:
    pub SDWRITE$ ( str addr -- )  SDADR! $! ;
    
    The SDADR! also marks the sector as having been written to so there is no need to specifically flush it as it will be done automatically if needed.

    Tachyon Forth - compact, fast, forthwright and interactive
    useforthlogo-s.png
    P2 --- The LOT --- TAQOZ INTRO & LINKS --- P2 SHORTFORM DATASHEET --- TAQOZ RELOADED - 64kB binary with room to spare
    P1 --- Latest Tachyon with EASYFILE --- Tachyon Forth News Blog --- More
    paypal.png PayPal me
    Brisbane, Australia
    phone.png
  • Peter JakackiPeter Jakacki Posts: 8,795
    edited 2019-12-03 - 04:49:59
    I also have a few different methods for listing the directory, the short method, the long DIR which lists out helpful information about starting sector and clusters allocated etc, and the FTP friendly "ls -l" method.

    Forgive the long listing, I had to chop out (~~~) whole sections just to keep the forum happy:
    TAQOZ# DIRW --- 
    ROOTDIR      ANSI         CODE         HELP         MUSIC        
    VIDEOS       3X5     .TXT 500MILES.WAV ABOTTLE .WAV ALABAMA .WAV AMORE   .WAV ASCIIART.TXT AVATAR  .BMV 
    AVATAR  .WAV BABYLON .WAV BASS    .WAV BEACH   .BMP BEACH2  .BMP BELIEVER.WAV BENDER  .BMV BIRD    .BMP 
    WORDMIX .TXT _BOOT_P2.B1  _BOOT_P2.B2  _BOOT_P2.BIB _BOOT_P2.BI5 _BOOT_P2.BIZ _BOOT_P2.BY  _BOOT_P2.LZM 
    BOOTS   .WAV BROWNEYE.WAV BUZZ    .BMP C2PROG  .FTH CALCDEMO.FTH CANON   .WAV COMEAWAY.WAV COSMIC  .WAV 
    CRASHES .WAV CRUSH   .WAV DANNY   .WAV DAYO    .WAV DECOMP  .FTH DOTLED  .FTH DRAGON  .BMP DREAMIN .WAV 
    DRIFT   .WAV DRUM    .WAV EASYFILE.FTH EASYNET .FTH EATWORLD.WAV EXFAT   .TXT EXTEND  .FTH EXTRAS  .FTH 
    EYEGOD  .BMP EYEJOE  .WAV FACE    .BMP FALLLOVE.WAV FASTCAR .WAV FBSPI3  .FTH FIRE    .BMP FISH2   .VT  
    FLOWERS .WAV FOURWIND.WAV FRIENDS .WAV GAS     .WAV GEORGIA .WAV GRASS   .WAV GUANTAN .WAV GUNDI   .WAV 
    HERDOOR .WAV HOME    .HTM HOMEWARD.WAV HORIZON .WAV HTTP001 .HTM HTTP404 .HTM ILIAD   .TXT INLOVE  .WAV 
    ISLAND  .WAV ISTANBUL.WAV JANE    .BMV JANE    .WAV JANE1   .BMV KELLYST .WAV KJV     .TXT LEXICON .TXT 
    LIFE    .FTH LION    .WAV LMMS    .BMP LOGON   .HTM LOVE    .WAV LOVEISIN.WAV MANICMON.WAV MARIO   .BMP 
    MARIO   .PNG MCQUEEN .BMP MIDENG  .TXT MOONLIGH.WAV MOONRIVR.WAV MORNING .WAV MOUTH   .WAV P2      .ROM 
    P2D2    .BMP P2D2A   .BMP P8X32A  .PDF PARALLAX.PNG PERHAPS .WAV PODCAST .WAV POPCORN .WAV PRIDE   .TXT 
    PS2KEYB .FTH PUMP    .TXT REET    .WAV ROCKROLL.WAV ROUGES  .TXT SEE     .FTH SHARONA .WAV SHINE   .WAV 
    SOFTLY  .WAV SPIDEY  .BMP SPIDEY  .GIF SPIDEY  .LZM SPIDEY  .PNG SPLAT-V4.FTH STANDBY .WAV SUMMER  .WAV 
    SUMMER69.WAV SUNSET  .BMP SUNSHINE.WAV TAQOZ1V1.FTH TAQOZ2V0.FTH TEQUILA .WAV THISKISS.WAV TIGER   .BMP 
    TIGER   .GIF TIGER   .JPG TIGER   .PNG TIGER1  .BMP TONIGHT .WAV TRULY   .WAV TWIST   .WAV VGA     .FTH 
    VOCAB   .FTH VULGAR  .TXT W5500A  .FTH WARPEACE.LZM WARPEACE.TXT WARWORLD.TXT WAYWARD .WAV WEBSTERS.TXT 
    WISHIN  .WAV WONDER  .WAV _BOOT_P2.BI4 _BOOT_P2.BIX   ok
    TAQOZ# DIR --- 
      0: ROOTDIR      08 
      1: ANSI         10 $0002_B296   2019-03-24 16:04              0 /        65,536
      2: CODE         10 $0003_4416   2017-07-24 07:56              0 /        65,536
      3: HELP         10 $0003_4716   2017-12-24 12:34              0 /        65,536
      4: MUSIC        10 $0003_4E96   2016-06-24 02:37              0 /        65,536
      5: VIDEOS       10 $0007_1016   2016-08-24 02:50              0 /        65,536
      6: 3X5     .TXT 20 $000A_AF16   2017-11-24 15:06          3,968 /        65,536
      7: 500MILES.WAV 20 $000A_AF96   2019-04-24 05:00     17,892,994 /    17,956,864
      8: ABOTTLE .WAV 20 $000B_3896   2019-04-24 05:00     12,872,642 /    12,910,592
      9: ALABAMA .WAV 20 $000B_9B16   2019-04-24 05:00     26,440,884 /    26,476,544
     10: AMORE   .WAV 20 $000C_6516   2019-04-24 05:00     16,658,054 /    16,711,680
    ~~~
    152: WONDER  .WAV 20 $00DA_4996   2019-04-24 05:00     22,577,076 /    22,609,920
    155: _BOOT_P2.BI4 20 $0000_BC96   1980-00-24 00:00      1,048,576 /     1,048,576
    156: _BOOT_P2.BIX 20 $0000_C496   1980-01-24 00:00         65,536 /        65,536  ok
    TAQOZ# ls -l
    -rwxrwxrwx 1 502     500          0 Mar 12  16:04 [ANSI]
    -rwxrwxrwx 1 502     500          0 Jul 20  07:56 [code]
    -rwxrwxrwx 1 502     500          0 Dec  4  12:34 [HELP]
    -rwxrwxrwx 1 502     500          0 Jun 25  02:37 [MUSIC]
    -rwxrwxrwx 1 502     500          0 Aug  5  02:50 [VIDEOS]
    -rwxrwxrwx 1 502     500       3968 Nov 23  15:06 3X5.TXT
    -rwxrwxrwx 1 502     500   17892994 Apr  3  05:00 500MILES.WAV
    -rwxrwxrwx 1 502     500   12872642 Apr  3  05:00 ABOTTLE.WAV
    -rwxrwxrwx 1 502     500   26440884 Apr  3  05:00 ALABAMA.WAV
    -rwxrwxrwx 1 502     500   16658054 Apr  3  05:00 AMORE.WAV
    -rwxrwxrwx 1 502     500      55148 Nov 23  15:05 ASCIIART.TXT
    -rwxrwxrwx 1 502     500 3404325888 Apr  2  02:31 AVATAR.BMV
    -rwxrwxrwx 1 502     500  139198622 Apr  2  05:37 AVATAR.WAV
    ~~~
    -rwxrwxrwx 1 502     500   15644386 Apr  3  05:00 WAYWARD.WAV
    -rwxrwxrwx 1 502     500   28956348 Nov 23  12:59 WEBSTERS.TXT
    -rwxrwxrwx 1 502     500   15443946 Apr  3  05:00 WISHIN.WAV
    -rwxrwxrwx 1 502     500   22577076 Apr  3  05:00 WONDER.WAV
    -rwxrwxrwx 1 502     500    1048576 ...  0  00:00 _BOOT_P2.BI4
    -rwxrwxrwx 1 502     500      65536 Jan  1  00:00 _BOOT_P2.BIX
     ---  ok
    TAQOZ#
    

    Tachyon Forth - compact, fast, forthwright and interactive
    useforthlogo-s.png
    P2 --- The LOT --- TAQOZ INTRO & LINKS --- P2 SHORTFORM DATASHEET --- TAQOZ RELOADED - 64kB binary with room to spare
    P1 --- Latest Tachyon with EASYFILE --- Tachyon Forth News Blog --- More
    paypal.png PayPal me
    Brisbane, Australia
    phone.png
  • Cluso99Cluso99 Posts: 15,549
    edited 2019-12-03 - 05:17:46
    MS doesn't get any royalties out of FAT32, but they do with exFAT :(

    BTW the ROM will try to load"_BOOT_P2.BIX" and if not found, it will try "_BOOT_P2.BIY"
    Of course it will try sector 0, then the VOL sector first, looking for the signature to load.
    The Sector 0 mechanism works for exFAT formatted SD cards.
    My Prop boards: P8XBlade2 , RamBlade , CpuBlade , TriBlade
    P1 Prop OS (also see Sphinx, PropDos, PropCmd, Spinix)
    Website: www.clusos.com
    P1: Tools (Index) , Emulators (Index) , ZiCog (Z80)
    P2: Tools & Code , Tricks & Traps
  • Thanks for the exFAT reminder Ray. I haven't tried those methods yet, but should do for completeness

    There has been a bit of movement on exFAT recently. They published the spec and opened it up to some
    https://www.zdnet.com/article/microsoft-readies-exfat-patents-for-linux-and-open-source/

  • I'll try and post an example flashing the leds on the P2-EVAL when I get home tonight
    My Prop boards: P8XBlade2 , RamBlade , CpuBlade , TriBlade
    P1 Prop OS (also see Sphinx, PropDos, PropCmd, Spinix)
    Website: www.clusos.com
    P1: Tools (Index) , Emulators (Index) , ZiCog (Z80)
    P2: Tools & Code , Tricks & Traps
  • In Linux I use the standard mkfs tools (mkfs.vfat, specifically) for formatting, and GNU ddrescue for imaging/backing up cards (works very similarly to the UNIX/Linux "builtin" dd, but with extra features)
    --
    WIP Spin/PASM drivers: (P1) (P2) P1 & P2: LSM9DS1, US2066, MLX90621, SHT3x, SSD1306, TCS347x, MAX31856, nRF24L01+, MLX90614, DS28CM00, TSL2591, INA219, SX1276/LoRa, VEML6075 | P1 only: TMC2130, MAX9744, CC1101, SX1231, LM75, DS18B20, Si7021, BT81x, MB85RC | P2 only: W25Qxxx, INA260 | WIP: L3G4200D, IL3820, MPU9250, MLX90640 | Stalled: SI446x, BMP280, BME680, BNO055
  • I'd suggest using Rufus on windows.
    https://rufus.ie/

    Definitely one of the best SD Burners I've used, and handles FAT filesystem generation parameters very well.
Sign In or Register to comment.