Shop OBEX P1 Docs P2 Docs Learn Events
P2 SD Wrangling Tools — Parallax Forums

P2 SD Wrangling Tools

TubularTubular Posts: 4,702
edited 2019-12-03 02:40 in Propeller 2
I'm developing a P2 product that I hope can just work from an SD card, without the boot flash. Turns out there's a bit of work involved in getting all this to be nice and reliable.

Windows 10's formatter only offers exFAT for SD cards above 32 GB, so I thought I'd start a thread to cover SD wrangling tools people use to get by. Of course many cards already come in FAT32 out of the packet, but I've found sometimes a reformat is necessary to restore things.

Feel free to chime in with tools you've used (eg with rasp Pi), and what their caveats are. I'm curious what people on Mac/Linux use

SD Card Formatting
* Taqoz - P2! - Peter's extended Taqoz has a FAT32 formatter.
* "Format" - Windows 10) - Right click on drive from in explorer etc. 32 GB cards are the biggest it'll format with FAT32
* GUIFORMAT - Windows - runs under windows without installation. Also known as Fat32format?
* SD formatter - Windows/Mac - recommended to me by Jeff Martin. Sandisk/SD Association tool.

SD Image Burning
* Win32DiskImager - Windows - Used with Rasp Pi images. Whole drive image so takes a while to run. Don't like the proximity of 'device' to the image filename, could potentially confuse drives

Comments

  • Peter JakackiPeter Jakacki Posts: 10,193
    edited 2019-12-03 03:31
    Yes, there certainly is a problem with the Window's format utility not only in that you cannot specify that it format a >32GB card to FAT32 (or any other format), but also in that it does not comply with the format method recommended by the SD card association where several MB is left free before the start of the partition for improved SD card operation. Most format tools just format a Flash device the same as a hard-drive without taking this into consideration.

    TAQOZ has the FAT32 formatter which I have tested on cards up to 128GB (so far) and can specify the cluster size etc. You can even format just one part such as the MBR for instance which has pulled me out of a sticky jam a few times already. Also as part of the formatter it includes disk tools that can not only report details about the card, but also run tests and handle the card as one great big memory chip for the first 4GB, or as a sector device, or individual files as virtual memory up to 4GB which can be handled as any other memory (read, write, dump, list etc).

    Here is a terminal dump where I format a 64GB card, first without permissions, and then with read/write/system permissions after which I dump the first sector as if it were an open file, manipulate it and dump it again. 
    TAQOZ# FORMAT ---  FAILED! ok
TAQOZ#  ---  ok
TAQOZ# RWS FORMAT ---  CARD: SANDISK   SD SC64G REV$80 #38874969 DATE:2019/8 CARD: SANDISK   SD SC64G REV$80 #38874969 DATE:2019/8

                   *** OCR *** 
    VALUE........................... $C0FF_8000
    RANGE........................... 2.7V to 3.6V

                   *** CSD *** 
    CARD TYPE....................... SDHC
    LATENCY......................... 1ms+1400 clocks 
    SPEED........................... 50MHz 
    CLASSES......................... 010110110101
    BLKLEN.......................... 512
    SIZE............................ 62,367MB
    Iread Vmin...................... 10ma
    Iread Vmax...................... 25ma
    Iwrite Vmin..................... 60ma
    Iwrite Vmax..................... 35ma

                 *** SPEEDS *** 
    LATENCY......................... 446us,131us,131us,131us,132us,130us,131us,131us,
    SECTOR.......................... 355us,287us,287us,286us,287us,287us,286us,287us,
    BLOCKS.......................... 3,148kB/s @300MHz

                   *** MBR *** 
    PARTITION....................... 0 00 INACTIVE
    FILE SYSTEM..................... FAT32 LBA
    CHS START....................... 1023,254,63
    CHS END......................... 0,0,0
    FIRST SECTOR.................... $0000_8000
    TOTAL SECTORS................... 124,702,720 = 63,847MB

00170: 0000_0000 0000_0000 0000_0000 0000_0000     '................'

                  *** FAT32 *** 
    OEM............................. TAQOZ P2
    Byte/Sect....................... 512
    Sect/Clust...................... 128 = 65MB
    FATs............................ 2
    Media........................... F8
    Sect/Track...................... $003F
    Heads........................... $00FF
    Hidden Sectors.................. 32,768 = 16MB
    Sect/Part....................... 124,702,720 = 63,847MB
    Sect/FAT........................ 7,611 = 3MB
    Flags........................... 0
    Ver............................. 00 00 
    ROOT Cluster.................... $0000_0002 SECTOR: $0000_BB96
    INFO Sector..................... $0001 = $0000_8001
    Backup Sector................... $0006 = $0000_8006
    res............................. 00 00 00 00 00 00 00 00 00 00 00 00 
    Drive#.......................... 128
    Ext sig......................... $29 OK!
    Part Serial#.................... $6269_0201 #1651048961
    Volume Name..................... P2 CARD    FAT32    ok
TAQOZ# 0 OPEN-SECTOR ---  ok
TAQOZ# 0 $200 SD DUMP --- 
00000: 64 12 80 FD  50 32 44 32  72 33 20 20  03 00 00 00     'd...P2D2r3  ....'
00010: 00 2D 31 01  00 A3 E1 11  FB 0E 00 01  00 C2 01 00     '.-1.............'
00020: 30 31 32 33  34 00 00 00  4C 0C 00 00  00 F8 E6 11     '01234...L.......'
00030: 35 36 00 00  C4 04 00 00  34 22 00 00  F2 22 00 00     '56......4"..."..'
00040: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
00050: 00 00 00 00  00 00 00 00  00 00 00 00  8C 00 8C 00     '................'
00060: 0D 00 78 7B  00 00 00 00  00 00 00 00  00 00 00 00     '..x{............'
00070: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
00080: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
00090: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
000A0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
000B0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
000C0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
000D0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
000E0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
000F0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
00100: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
00110: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
00120: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
00130: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
00140: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
00150: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
00160: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
00170: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
00180: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
00190: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
001A0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
001B0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 FE     '................'
001C0: FF FF 0C 00  00 00 00 80  00 00 00 D0  6E 07 00 00     '............n...'
001D0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
001E0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00     '................'
001F0: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 55 AA     '..............U.' ok
TAQOZ# " HELLO WORLD" $100 SDADR $! FLUSH ---  ok
TAQOZ# 0 $200 SD DUMPAW --- 
00000: d...P2D2r3  .....-1.............01234...L.......56......4"..."....................................x{............................
00080: ................................................................................................................................
00100: HELLO WORLD.....................................................................................................................
00180: ............................................................................n.................................................U. ok
TAQOZ#


    btw, that funny little operation that I did to write a string could just as easily have had function for it with: 
    pub SDWRITE$ ( str addr -- )  SDADR! $! ;
    The SDADR! also marks the sector as having been written to so there is no need to specifically flush it as it will be done automatically if needed.
  • Peter JakackiPeter Jakacki Posts: 10,193
    edited 2019-12-03 04:49
    I also have a few different methods for listing the directory, the short method, the long DIR which lists out helpful information about starting sector and clusters allocated etc, and the FTP friendly "ls -l" method.

    Forgive the long listing, I had to chop out (~~~) whole sections just to keep the forum happy: 
    TAQOZ# DIRW --- 
ROOTDIR      ANSI         CODE         HELP         MUSIC        
VIDEOS       3X5     .TXT 500MILES.WAV ABOTTLE .WAV ALABAMA .WAV AMORE   .WAV ASCIIART.TXT AVATAR  .BMV 
AVATAR  .WAV BABYLON .WAV BASS    .WAV BEACH   .BMP BEACH2  .BMP BELIEVER.WAV BENDER  .BMV BIRD    .BMP 
WORDMIX .TXT _BOOT_P2.B1  _BOOT_P2.B2  _BOOT_P2.BIB _BOOT_P2.BI5 _BOOT_P2.BIZ _BOOT_P2.BY  _BOOT_P2.LZM 
BOOTS   .WAV BROWNEYE.WAV BUZZ    .BMP C2PROG  .FTH CALCDEMO.FTH CANON   .WAV COMEAWAY.WAV COSMIC  .WAV 
CRASHES .WAV CRUSH   .WAV DANNY   .WAV DAYO    .WAV DECOMP  .FTH DOTLED  .FTH DRAGON  .BMP DREAMIN .WAV 
DRIFT   .WAV DRUM    .WAV EASYFILE.FTH EASYNET .FTH EATWORLD.WAV EXFAT   .TXT EXTEND  .FTH EXTRAS  .FTH 
EYEGOD  .BMP EYEJOE  .WAV FACE    .BMP FALLLOVE.WAV FASTCAR .WAV FBSPI3  .FTH FIRE    .BMP FISH2   .VT  
FLOWERS .WAV FOURWIND.WAV FRIENDS .WAV GAS     .WAV GEORGIA .WAV GRASS   .WAV GUANTAN .WAV GUNDI   .WAV 
HERDOOR .WAV HOME    .HTM HOMEWARD.WAV HORIZON .WAV HTTP001 .HTM HTTP404 .HTM ILIAD   .TXT INLOVE  .WAV 
ISLAND  .WAV ISTANBUL.WAV JANE    .BMV JANE    .WAV JANE1   .BMV KELLYST .WAV KJV     .TXT LEXICON .TXT 
LIFE    .FTH LION    .WAV LMMS    .BMP LOGON   .HTM LOVE    .WAV LOVEISIN.WAV MANICMON.WAV MARIO   .BMP 
MARIO   .PNG MCQUEEN .BMP MIDENG  .TXT MOONLIGH.WAV MOONRIVR.WAV MORNING .WAV MOUTH   .WAV P2      .ROM 
P2D2    .BMP P2D2A   .BMP P8X32A  .PDF PARALLAX.PNG PERHAPS .WAV PODCAST .WAV POPCORN .WAV PRIDE   .TXT 
PS2KEYB .FTH PUMP    .TXT REET    .WAV ROCKROLL.WAV ROUGES  .TXT SEE     .FTH SHARONA .WAV SHINE   .WAV 
SOFTLY  .WAV SPIDEY  .BMP SPIDEY  .GIF SPIDEY  .LZM SPIDEY  .PNG SPLAT-V4.FTH STANDBY .WAV SUMMER  .WAV 
SUMMER69.WAV SUNSET  .BMP SUNSHINE.WAV TAQOZ1V1.FTH TAQOZ2V0.FTH TEQUILA .WAV THISKISS.WAV TIGER   .BMP 
TIGER   .GIF TIGER   .JPG TIGER   .PNG TIGER1  .BMP TONIGHT .WAV TRULY   .WAV TWIST   .WAV VGA     .FTH 
VOCAB   .FTH VULGAR  .TXT W5500A  .FTH WARPEACE.LZM WARPEACE.TXT WARWORLD.TXT WAYWARD .WAV WEBSTERS.TXT 
WISHIN  .WAV WONDER  .WAV _BOOT_P2.BI4 _BOOT_P2.BIX   ok
TAQOZ# DIR --- 
  0: ROOTDIR      08 
  1: ANSI         10 $0002_B296   2019-03-24 16:04              0 /        65,536
  2: CODE         10 $0003_4416   2017-07-24 07:56              0 /        65,536
  3: HELP         10 $0003_4716   2017-12-24 12:34              0 /        65,536
  4: MUSIC        10 $0003_4E96   2016-06-24 02:37              0 /        65,536
  5: VIDEOS       10 $0007_1016   2016-08-24 02:50              0 /        65,536
  6: 3X5     .TXT 20 $000A_AF16   2017-11-24 15:06          3,968 /        65,536
  7: 500MILES.WAV 20 $000A_AF96   2019-04-24 05:00     17,892,994 /    17,956,864
  8: ABOTTLE .WAV 20 $000B_3896   2019-04-24 05:00     12,872,642 /    12,910,592
  9: ALABAMA .WAV 20 $000B_9B16   2019-04-24 05:00     26,440,884 /    26,476,544
 10: AMORE   .WAV 20 $000C_6516   2019-04-24 05:00     16,658,054 /    16,711,680
~~~
152: WONDER  .WAV 20 $00DA_4996   2019-04-24 05:00     22,577,076 /    22,609,920
155: _BOOT_P2.BI4 20 $0000_BC96   1980-00-24 00:00      1,048,576 /     1,048,576
156: _BOOT_P2.BIX 20 $0000_C496   1980-01-24 00:00         65,536 /        65,536  ok
TAQOZ# ls -l
-rwxrwxrwx 1 502     500          0 Mar 12  16:04 [ANSI]
-rwxrwxrwx 1 502     500          0 Jul 20  07:56 [code]
-rwxrwxrwx 1 502     500          0 Dec  4  12:34 [HELP]
-rwxrwxrwx 1 502     500          0 Jun 25  02:37 [MUSIC]
-rwxrwxrwx 1 502     500          0 Aug  5  02:50 [VIDEOS]
-rwxrwxrwx 1 502     500       3968 Nov 23  15:06 3X5.TXT
-rwxrwxrwx 1 502     500   17892994 Apr  3  05:00 500MILES.WAV
-rwxrwxrwx 1 502     500   12872642 Apr  3  05:00 ABOTTLE.WAV
-rwxrwxrwx 1 502     500   26440884 Apr  3  05:00 ALABAMA.WAV
-rwxrwxrwx 1 502     500   16658054 Apr  3  05:00 AMORE.WAV
-rwxrwxrwx 1 502     500      55148 Nov 23  15:05 ASCIIART.TXT
-rwxrwxrwx 1 502     500 3404325888 Apr  2  02:31 AVATAR.BMV
-rwxrwxrwx 1 502     500  139198622 Apr  2  05:37 AVATAR.WAV
~~~
-rwxrwxrwx 1 502     500   15644386 Apr  3  05:00 WAYWARD.WAV
-rwxrwxrwx 1 502     500   28956348 Nov 23  12:59 WEBSTERS.TXT
-rwxrwxrwx 1 502     500   15443946 Apr  3  05:00 WISHIN.WAV
-rwxrwxrwx 1 502     500   22577076 Apr  3  05:00 WONDER.WAV
-rwxrwxrwx 1 502     500    1048576 ...  0  00:00 _BOOT_P2.BI4
-rwxrwxrwx 1 502     500      65536 Jan  1  00:00 _BOOT_P2.BIX
 ---  ok
TAQOZ#
  • Cluso99Cluso99 Posts: 18,069
    edited 2019-12-03 05:17
    MS doesn't get any royalties out of FAT32, but they do with exFAT :(

    BTW the ROM will try to load"_BOOT_P2.BIX" and if not found, it will try "_BOOT_P2.BIY"
    Of course it will try sector 0, then the VOL sector first, looking for the signature to load.
    The Sector 0 mechanism works for exFAT formatted SD cards.
  • TubularTubular Posts: 4,702
    Thanks for the exFAT reminder Ray. I haven't tried those methods yet, but should do for completeness

    There has been a bit of movement on exFAT recently. They published the spec and opened it up to some
    https://www.zdnet.com/article/microsoft-readies-exfat-patents-for-linux-and-open-source/

  • Cluso99Cluso99 Posts: 18,069
    I'll try and post an example flashing the leds on the P2-EVAL when I get home tonight
  • avsa242avsa242 Posts: 452
    In Linux I use the standard mkfs tools (mkfs.vfat, specifically) for formatting, and GNU ddrescue for imaging/backing up cards (works very similarly to the UNIX/Linux "builtin" dd, but with extra features)
  • kamilionkamilion Posts: 20
    I'd suggest using Rufus on windows.
    https://rufus.ie/

    Definitely one of the best SD Burners I've used, and handles FAT filesystem generation parameters very well.
Sign In or Register to comment.