I mean I know once the boot is sorted then you can add files to SD to do a more complete FAT handler, it just seems an unnecessary step.
The presumption is there will be a tool to support this, so it's a pretty small bother imho. It's not like this will be the primary means of software testing/development/debugging.
That said, I'm easy if Chip feels a FAT filesystem hardwired in the Prop2 ROM is doable. But I do feel it's going too far.
For the minimal 1st stage SD boot using the pointer in the MBR I'm not thinking that this would usually be done on the PC but rather on the P2 itself so really there are no PC hoops to jump through as we have direct access. In Tachyon for instance the whole MBR bootsig write is only a few instructions that all fit on one line, super easy.
@Oz, if we boot the Spin compiler which then finds the Spin file and compiles it then I believe we could do it this way. So that would be a two stage "bootload" or should I say load and compile. Same applies to my Tachyon except of course I already have that capability since Tachyon is also the compiler as well as the runtime environment. I just say FLOAD <source file> and away it goes.
I have a "clean room" implementation of SD reading code that I came up with for SSD1921 image processor work. I didn't want to use the Microchip copyrighted version... It combines things from FSRW and a couple other open sources. It's at home, but I can dig it up tonight...
As I recall, there are several different ways to do it. FSRW seems to do it in a very unique way, couldn't find anyone else doing it that way.
Peter, most users will involve a pc. That is part of the reason some of us were advocating a partition as pointer. And that is a few instructions as well.
No tool needed on linux, and only dd required on Windows.
Chip, my head's totally into SD and FAT32 at present, just been going over all the specs and protocols in a revision so I believe I could write a very compact PASM level bootloader that would find the file in the root directory as quickly and cleanly as possible. I'd rather that you didn't muddle and corrupt the sensitivities of your brain with FAT32 as I have to somehow not go mad or laugh or cry when I look at how it is all implemented.
For testing I will load the "bootloader" in via PNut although if you are inclined you could implement the serial Flash bootloader into the FPGA, then we could all test the SD part of it more easily.
I just looked at my code and it's really not much different that FRSW.
FRSW already works and is in PASM, so maybe that's actually the best way...
Anyway, this driver has the minimal stuff needed to low level init SD card.
What's missing is the bit-banging SPI that Safe_spi.spin has and the upper level stuff that FSRW.spin has.
But, some of this init is specific to SSD1921...
Anyway, I think this is the important part, the slow SPI initialization:
SendCommand(CMD0,%0000_0000,0) 'Send CMD0 to reset and enter SPI mode
'Checking Voltage Compatibility with High Capacity Cards
SendCommand(CMD8,%0000_0010,$1AA) 'type 1 returns 0, type 2 returns $1AA
i:=GetResponse32
if (i==$1AA)
CardType:=type_SDHC 'or maybe SD
else
CardType:=type_SD 'or maybe MMC
'Get card into ready state
repeat
SendCommand(ACMD41, %0000_0010, $40200000)
i:=GetResponse32
if (i&$8000_0000)<>0 'Note: >0 doesn't work here because sign bit!
quit 'Ready!
delayms(100)
SendCommand(CMD58,%0000_0010, 0) 'see if SD or SDHC
i:=GetResponse32
if i&|<30
CardType:=type_SDHC
adrShift:=0 'don't need to shift address for HC cards!
else
CardType:=type_SD
adrShift:=9
' card is mounted, make sure the CRC is turned off
SendCommand( CMD59, %0000_0010, 0)
'Have to ask card ID, but don't need it
SendCommand(CMD2,%0000_0001,0)
i:=GetResponse32
'Need relative card address (RCA)
SendCommand(CMD3,%0010_0010,$0001_0000)
RCA:=GetResponse32
'Select card using RCA
SendCommand(CMD7,%0010_0010,RCA)
i:=GetResponse32
'Set clock to high speed
For SSD1921, I used FSRW.spin with safe_spi.spin replaced with the attached...
Anyway, I think this is the important part, the slow SPI initialization:
I think you will probably find that this was really meant for MMC compatibility plus also the smallest card you can barely get is 4GB and SDHC. The more typical card you would buy would be 8 or 16GB and we never ever have to worry about standard SD or FAT16 as proper 2GB cards are so hard to find and more expensive then what I can 8GB SanDisk for.
My low-level SPI routines work very reliably with a wide range of brands and different sized cards I have been testing with the P2.
BTW, 64GB cards are preformatted with MS proprietary exFAT, the ex being short for excessive. How much did MS pay the SD card association to recommend this monstrosity and tightly controlled proprietary format that MS can withhold at a whim from "unsavoury" types (bad bad Linux ppl :> )
This sounds like an ideal candidate for a two stage process.
a) Chip does the lowest Simple HW boot, up to SPI, and includes that in a near term FPGA build.
I think the core of that is known already - 50% clock duty cycle and broad Tco tolerance.
This also defines exactly how much ROM space is available for SD.
b) That simple HW boot [UART.i2c?.SPI] read-only is used as a testing base for SD loaders, and those candidates can be field tested before one is selected and loaded into the spare ROM.
It sounds like Chip is much better spending his time on Smart Pins, than on SD details.
...BTW, 64GB cards are preformatted with MS proprietary exFAT, the ex being short for excessive. How much did MS pay the SD card association to recommend this monstrosity and tightly controlled proprietary format that MS can withhold at a whim from "unsavoury" types (bad bad Linux ppl :> )
Can those be reformatted to get rid of the excessive FAT. Or is this stubborn FAT?
...BTW, 64GB cards are preformatted with MS proprietary exFAT, the ex being short for excessive. How much did MS pay the SD card association to recommend this monstrosity and tightly controlled proprietary format that MS can withhold at a whim from "unsavoury" types (bad bad Linux ppl :> )
Can those be reformatted to get rid of the excessive FAT. Or is this stubborn FAT?
Yes, I just reformat them with fat32, why didn't they just stick with that in the first place? ($$$)
I believe it's the windows format utility that is stubborn about not formatting as fat32, funny about that, but à third party formatter does the trick as does Linux ☺
Wikipedia says they switched to exFAT for HDXC cards >32GB because HD camcorders can exceed the 4 GB file size limit in an hour... exFAT lets you keep your long recordings in one file...
Also made a table of hashed filenames to speed file finding..
Wikipedia says they switched to exFAT for HDXC cards >32GB because HD camcorders can exceed the 4 GB file size limit in an hour... exFAT lets you keep your long recordings in one file...
Also made a table of hashed filenames to speed file finding..
Interesting - thinking ahead to a time when 32GB is considered small, and you cannot buy anything less, where does that leave P2 ROM support ?
There are also gong to be logging cases using P2, where "keep your long recordings in one file..." would also appeal.
Seems to me, a block level procedure is future proofing. No FAT, just put the image on the card.
We can always use a partition for that and or a FAT filesystem.
So a person formats the card, or runs a program, or an existing Propeller system does it. They will be right there along with a ton of other people formatting for use on pre-exFAT devices, and will be there for many years to come.
I've a few of these higher end, new cards, and all of them accepted formatting of various kinds. Even really brutal, like just writing the MBR from a small card to a larger one. That worked too, with the obvious unused space in play, of course.
For the dataloggers, boot with something bootable and provide big filesystem support there, just as we do on P1 with an EEPROM and FAT code.
It's going to look a lot like hard disks do, and IMHO, that's all just fine. Disk prep is a known thing.
Now that you mention these things, I think I'll make sure my P2 systems have two SD card slots. One will contain all the tools, the other is storage, prep, etc...
Wikipedia says they switched to exFAT for HDXC cards >32GB because HD camcorders can exceed the 4 GB file size limit in an hour... exFAT lets you keep your long recordings in one file...
Also made a table of hashed filenames to speed file finding..
So ext2 or ext4 etc would be far more logical but of course that is an "open" format and not something that MS will support although third-party Windows drivers are available AFAIK. What a stranglehold MS has proven on the computer industry that it has kept the industry back by decades. When the day comes when laptop and peripheral manufacturers aren't effectively blackmailed into supporting Windows exclusively then you may find the likes of MS going the way of Big Blue did although the latter have always invested in researching advanced technologies.
Hashing file directories might only good for runtime operation but a firmware update via a PC may well move things around.
UDF might be an open filesystem that's usable and can handle the big files. It's certainly a supported one. But don't count this as a vote for baking a filesystem in!
Seems to me, a block level procedure is future proofing. No FAT, just put the image on the card.
I think you may be right. There's so much FAT in the way, it's hard to tell.
Could any of you draw up some pseudo code or a flowchart to explain what the simplest options are?
If we can nail it down, we can have something in several days from now.
Ah, back to what I was saying from the beginning, just point to a sector but don't even be dependant upon the partition table either, a simple header in the first 16 bytes of sector 0 which many call the MBR can point to an absolute start sector for the image. That start sector may be in some partition or an actual file etc or there may be another format or none at all, it doesn't matter.
My favorite laptop has finally fritzed and I am just pulling it apart to see if it is repairable or order a new mb but I may have to use another in the meantime. So then I can get back to writing a strictly PASM SD block loader and then even put in FAT32 support for the final product later on. I don't know why anyone would be worried about having to write to the MBR, it's child's play and there is no reason why a simple bit of temporary P2 code can't locate the file for you and set the pointer in the 16-byte header so that thereafter the basic block loader can just does its thing that it does so well.
Chip, I can write workable PASM code that you can modify to suit.
Seems to me, a block level procedure is future proofing. No FAT, just put the image on the card.
I think you may be right. There's so much FAT in the way, it's hard to tell.
Could any of you draw up some pseudo code or a flowchart to explain what the simplest options are?
If we can nail it down, we can have something in several days from now.
Ah, back to what I was saying from the beginning, just point to a sector but don't even be dependant upon the partition table either, a simple header in the first 16 bytes of sector 0 which many call the MBR can point to an absolute start sector for the image. That start sector may be in some partition or an actual file etc or there may be another format or none at all, it doesn't matter.
My favorite laptop has finally fritzed and I am just pulling it apart to see if it is repairable or order a new mb but I may have to use another in the meantime. So then I can get back to writing a strictly PASM SD block loader and then even put in FAT32 support for the final product later on. I don't know why anyone would be worried about having to write to the MBR, it's child's play and there is no reason why a simple bit of temporary P2 code can't locate the file for you and set the pointer in the 16-byte header so that thereafter the basic block loader can just does its thing that it does so well.
Chip, I can write workable PASM code that you can modify to suit.
Peter, that would be perfect. I'll adapt in into the ROM booter, then.
Done, I will try to get it done today so others can start testing it too. I have timing diagrams in my last post in the Tachyon P2 thread which gives a good picture of what is required for reading one sector although since the card takes time until it has data ready I may just have an option for multiple block read to improve the load time even more.
Chip,
IMHO the simplest would be Peters method of the MBR pointing to a block of code. This would always work and the simplest to start with while we build a FAT version for you.
For a refresher, there is a quirk that leaves some SD cards (all?) in an incorrect state outputting on DO ??? It was necessary to give the SD a number of clocks to clear the SD card from this state. I patched fsrw to solve this (see earlier post of my modified code - marked) and its in Kyes code too (cannot recall if I did it or got Kye to do it).
fsrw - the pasm driver handles both I2C and SPI. Its a little intermixed probably due to cog space shortage. It can be stripped down to a basic SPI driver.
As for the FAT driver, Kye did a great job, but it does too many checks, so some cards that aren't properly formatted don't work even though those cards work fine on PCs. This code is also over the top with capabilities but misses a few requirements such as direct addressing which I have added.
Currently I don't have the time to get right into the FAT driver to trim it down. FWIW I use Kyes FAT drivers in my PropOS so the code is posted on that thread for anyone who is interested.
Comments
The presumption is there will be a tool to support this, so it's a pretty small bother imho. It's not like this will be the primary means of software testing/development/debugging.
That said, I'm easy if Chip feels a FAT filesystem hardwired in the Prop2 ROM is doable. But I do feel it's going too far.
@Oz, if we boot the Spin compiler which then finds the Spin file and compiles it then I believe we could do it this way. So that would be a two stage "bootload" or should I say load and compile. Same applies to my Tachyon except of course I already have that capability since Tachyon is also the compiler as well as the runtime environment. I just say FLOAD <source file> and away it goes.
But, we can do read only, and perhaps take other shortcuts.
http://codeandlife.com/2012/04/02/simple-fat-and-sd-tutorial-part-1/
That's the reference I started reading.
We can always offer a fall back, to a low level block scheme.
And we can always opt to not do it too, but attempting FAT makes sense.
As I recall, there are several different ways to do it. FSRW seems to do it in a very unique way, couldn't find anyone else doing it that way.
Yeah, I want to read some options. Looking forward to it.
Me, too.
No tool needed on linux, and only dd required on Windows.
For testing I will load the "bootloader" in via PNut although if you are inclined you could implement the serial Flash bootloader into the FPGA, then we could all test the SD part of it more easily.
FRSW already works and is in PASM, so maybe that's actually the best way...
Anyway, this driver has the minimal stuff needed to low level init SD card.
What's missing is the bit-banging SPI that Safe_spi.spin has and the upper level stuff that FSRW.spin has.
But, some of this init is specific to SSD1921...
Anyway, I think this is the important part, the slow SPI initialization:
SendCommand(CMD0,%0000_0000,0) 'Send CMD0 to reset and enter SPI mode 'Checking Voltage Compatibility with High Capacity Cards SendCommand(CMD8,%0000_0010,$1AA) 'type 1 returns 0, type 2 returns $1AA i:=GetResponse32 if (i==$1AA) CardType:=type_SDHC 'or maybe SD else CardType:=type_SD 'or maybe MMC 'Get card into ready state repeat SendCommand(ACMD41, %0000_0010, $40200000) i:=GetResponse32 if (i&$8000_0000)<>0 'Note: >0 doesn't work here because sign bit! quit 'Ready! delayms(100) SendCommand(CMD58,%0000_0010, 0) 'see if SD or SDHC i:=GetResponse32 if i&|<30 CardType:=type_SDHC adrShift:=0 'don't need to shift address for HC cards! else CardType:=type_SD adrShift:=9 ' card is mounted, make sure the CRC is turned off SendCommand( CMD59, %0000_0010, 0) 'Have to ask card ID, but don't need it SendCommand(CMD2,%0000_0001,0) i:=GetResponse32 'Need relative card address (RCA) SendCommand(CMD3,%0010_0010,$0001_0000) RCA:=GetResponse32 'Select card using RCA SendCommand(CMD7,%0010_0010,RCA) i:=GetResponse32 'Set clock to high speedFor SSD1921, I used FSRW.spin with safe_spi.spin replaced with the attached...
I think you will probably find that this was really meant for MMC compatibility plus also the smallest card you can barely get is 4GB and SDHC. The more typical card you would buy would be 8 or 16GB and we never ever have to worry about standard SD or FAT16 as proper 2GB cards are so hard to find and more expensive then what I can 8GB SanDisk for.
My low-level SPI routines work very reliably with a wide range of brands and different sized cards I have been testing with the P2.
BTW, 64GB cards are preformatted with MS proprietary exFAT, the ex being short for excessive. How much did MS pay the SD card association to recommend this monstrosity and tightly controlled proprietary format that MS can withhold at a whim from "unsavoury" types (bad bad Linux ppl :> )
a) Chip does the lowest Simple HW boot, up to SPI, and includes that in a near term FPGA build.
I think the core of that is known already - 50% clock duty cycle and broad Tco tolerance.
This also defines exactly how much ROM space is available for SD.
b) That simple HW boot [UART.i2c?.SPI] read-only is used as a testing base for SD loaders, and those candidates can be field tested before one is selected and loaded into the spare ROM.
It sounds like Chip is much better spending his time on Smart Pins, than on SD details.
Can those be reformatted to get rid of the excessive FAT. Or is this stubborn FAT?
Yes, I just reformat them with fat32, why didn't they just stick with that in the first place? ($$$)
I believe it's the windows format utility that is stubborn about not formatting as fat32, funny about that, but à third party formatter does the trick as does Linux ☺
Also made a table of hashed filenames to speed file finding..
There are also gong to be logging cases using P2, where "keep your long recordings in one file..." would also appeal.
We can always use a partition for that and or a FAT filesystem.
So a person formats the card, or runs a program, or an existing Propeller system does it. They will be right there along with a ton of other people formatting for use on pre-exFAT devices, and will be there for many years to come.
I've a few of these higher end, new cards, and all of them accepted formatting of various kinds. Even really brutal, like just writing the MBR from a small card to a larger one. That worked too, with the obvious unused space in play, of course.
For the dataloggers, boot with something bootable and provide big filesystem support there, just as we do on P1 with an EEPROM and FAT code.
It's going to look a lot like hard disks do, and IMHO, that's all just fine. Disk prep is a known thing.
Now that you mention these things, I think I'll make sure my P2 systems have two SD card slots. One will contain all the tools, the other is storage, prep, etc...
So ext2 or ext4 etc would be far more logical but of course that is an "open" format and not something that MS will support although third-party Windows drivers are available AFAIK. What a stranglehold MS has proven on the computer industry that it has kept the industry back by decades. When the day comes when laptop and peripheral manufacturers aren't effectively blackmailed into supporting Windows exclusively then you may find the likes of MS going the way of Big Blue did although the latter have always invested in researching advanced technologies.
Hashing file directories might only good for runtime operation but a firmware update via a PC may well move things around.
I think you may be right. There's so much FAT in the way, it's hard to tell.
Could any of you draw up some pseudo code or a flowchart to explain what the simplest options are?
If we can nail it down, we can have something in several days from now.
Ah, back to what I was saying from the beginning, just point to a sector but don't even be dependant upon the partition table either, a simple header in the first 16 bytes of sector 0 which many call the MBR can point to an absolute start sector for the image. That start sector may be in some partition or an actual file etc or there may be another format or none at all, it doesn't matter.
My favorite laptop has finally fritzed and I am just pulling it apart to see if it is repairable or order a new mb but I may have to use another in the meantime. So then I can get back to writing a strictly PASM SD block loader and then even put in FAT32 support for the final product later on. I don't know why anyone would be worried about having to write to the MBR, it's child's play and there is no reason why a simple bit of temporary P2 code can't locate the file for you and set the pointer in the 16-byte header so that thereafter the basic block loader can just does its thing that it does so well.
Chip, I can write workable PASM code that you can modify to suit.
Peter, that would be perfect. I'll adapt in into the ROM booter, then.
That will let us start playing with SD support.
IMHO the simplest would be Peters method of the MBR pointing to a block of code. This would always work and the simplest to start with while we build a FAT version for you.
For a refresher, there is a quirk that leaves some SD cards (all?) in an incorrect state outputting on DO ??? It was necessary to give the SD a number of clocks to clear the SD card from this state. I patched fsrw to solve this (see earlier post of my modified code - marked) and its in Kyes code too (cannot recall if I did it or got Kye to do it).
fsrw - the pasm driver handles both I2C and SPI. Its a little intermixed probably due to cog space shortage. It can be stripped down to a basic SPI driver.
As for the FAT driver, Kye did a great job, but it does too many checks, so some cards that aren't properly formatted don't work even though those cards work fine on PCs. This code is also over the top with capabilities but misses a few requirements such as direct addressing which I have added.
Currently I don't have the time to get right into the FAT driver to trim it down. FWIW I use Kyes FAT drivers in my PropOS so the code is posted on that thread for anyone who is interested.