Reboot from SD?
JonnyMac
Posts: 9,105
I'm working on a P1 project that allows me to reload the EEPROM with a binary image stored on the SD. The process essentially follows what the P1 bootloader does: 1) move binary image into EE, 2) clear rest of EE to zeroes, 3) write initial stack frame values, and 4) run checksum.
Some time ago, Chip mentioned to me in a conversation that the P2 would be able to boot from SD. What I'm hoping is that P2 Spin will allow me to pass a filename to the bootloader and have it take care of loading and running that image for me.
Some time ago, Chip mentioned to me in a conversation that the P2 would be able to boot from SD. What I'm hoping is that P2 Spin will allow me to pass a filename to the bootloader and have it take care of loading and running that image for me.
Comments
I don't think anyone concluded the Prop2 would be able to directly boot from SD. There was much discussing it but it's not a simple process like the serial ROMs. Not only that, but any filesystem implementations would be post boot anyway.
The general thread of discussion went along the lines of the Prop2 having the ability to cold load one or two fixed location blocks, called a boot loader, from the SD card. Typically such blocks are at the start of a drive and are used for partitioning and the likes. It would need a tool to manage it. This boot loader can be customised to pull in further loader code for an OS load up or it could do something else like simply booting into a single file via a pre-calculated block list written into it by the manager tool.
Another variant was the fixed blocks instead contain just a list of pre-calculated blocks to load. This way the first loader on the SD card can be much larger and can even be a written as a normal file. Except, to make this file the one to boot from, still requires a manager tool to built the block list with.
It would seem simple enough to exploit that to divert to a default boot file that would be read as a binary image to Propeller RAM. But the Propeller would require some sort of BIOS/boot loader with a bit of knowledge of FAT16 to succeed.
I am presuming that FAT16 is public domain. The rest of the SD card's file system access would depend entirely on what was in the first 32K image.
That external booter can be anything the user chooses.
A possible 'rough' solution for SD cards, would be to use what I think the FPGA vendors do, which is have a start preamble or key. (32 or 64b) ( I guess they have more confidence their bit-streams do not contain that) but the approach is to clock until that is seen, so the number of clocks is open, and could be quite high.
Typical Manual user place-first on the SD card would see this trundle over the FAT area, and then hit the valid image, and start loading. Not as fancy as FAT, but much smaller and may be workable in many applications.
Update would be by careful file replacement.
What if the file is fragmented? Or does data fragmentation not happen in FAT16? Do these FPGAs cache the image start position? They should.
You kind of hinted at this, but that method doesn't work with multiple images (unless each image has its own unique identifier).
The Prop1 operates at that level already and the thinking is that the Prop2 will end up being the same with any SD card support being loaded from EEPROM. BTW, I would be happy to proven wrong and see the Prop2 come with hard-coded SD booting feature.
The very simple approach, of course, has many caveats.
If you want to use all FAT features, like fragmented files and multiple images, then you will need a full FAT aware loader.
Some systems just want to use SD cards as a convenient packaged SPI chip, and do not mind limitations like 'one file, not fragmented'.
IIRC the loader includes a Size field, so the first load stage can be quite fast.(ie not a full image load)
The way Kye was talking there were a lot of unresolved issues with the configuration of the card's interface to make sure the SD card is in the expected mode. And/or having multiple handlers so the Prop2 can adapt to the variations in cards.
Aside from the potential bugs, before we even get to block access level, "smallest" is not a very compatible scenario when considering SD cards.
Yes, I may have been a little terse in my 'two step loading' - that means the Full FAT SD is not in ROM, but the ROM is only a SPI-Flash-Memory (Winbond?) reader, with a choice of how many bytes.
Once that small piece of SPI flash is loaded (step 1), then that is run for SD load, (step 2) with whatever level of handling the user decides to operate.
ie designs should always plan to have a small, cheap SPI-Flash part. I think they can come 2x3mm ?
Digikey shows ~18c/1k for 4Mbit SPI Flash, cheapest SO8, and that climbs to ~ 46c/1000 for cheapest (2x3) 8USON tiny package = 0.5Mb.
I think what Electrodude was asking was how can you assure that the file you copied over to the SD card wasn't fragmented during the copy procedure. AFAIK, you can't using common OSes (win/linux/osx) by default, but perhaps only with special tools. Of course, fragmenting should only occur if the file size exceeds the cluster size of the file system.
As for that whole debate a while back concerning native SD boot capability, I wonder just how big of a problem card incompatibility really is, and what can be done to minimize it. You would think that with the HUGE prevalence of SD cards in the removable storage markets, compatibility by adherence to standards would be of utmost importance. I could see how there may have been problems for a while early on (so, using those cards with capacity in the MB range might be a bad idea), but what about newer cards such as SDHC? Can we expect less compatibility issues with them? Since cards with over 2GB capacity are so cheap, if you prefer to go SD instead of SPI flash, there's no good reason to use that crummy old 8MB card in the back of your drawer. Granted, this point is moot if there exists compatibility issues among SDHC cards too, but can anyone verify that?
If those problems can be overcome, then loading bins from SD cards, even ones with files systems, should be easy enough. SD cards can be partitioned, and then formatted with sectors large enough to contain bin files without possibly getting fragmented. The file location can either be added to the MBR, or the partition can be scanned for files that contain identification markers.
You need root access to write to the MBR on Linux (and OSX as well), so MBR might not be the best choice. A special boot partition would probably also need root permission to write to, but you could probably get around that with a raw fs driver kernel module.
1) The cluster sizes can be much larger than ideal for the sake of preventing the .bin from getting fragmented during the copy procedure. This makes special software tools for writing the file unnecessary.
2) An MBR entry can be avoided by just scanning a certain address range looking for identification markers denoting the .bin file. Once again, this avoids he need for special tools to copy the program over.
Besides avoiding the need for special tools, it also avoids the need for a filesystem aware bootloader.
If the users don't have root access to perform these functions, then it's probably best to provide them with cards already partitioned. Though this only solves the problem where an MBR entry is not required.
I have an idea. Tell your mount daemon or udev or whatever to change the owner of any raw partitions it finds to your user instead of root. It does require root access once, for installation, but you need root access for global installations anyway.
Ummm... Aren't the SDcards formated in FAT16 or FAT32? The whole UNIX/Linux file system scheme doesn't apply in such cases, whereas the MBR does.
I may be wrong, but I just suspect that a lot of us are a bit fuzzy on how the Propeller uses the SDcard and how other OSes adapt to it.
Yes, I do suppose that OS X and Linux users would have to shift to Superuser to rewrite the MBR, but that is not exactly a huge burden or some sort of great security sin. People actually might learn something. And that knowledge might even enhance their understand of how to retain security.
Uncertain that the filesystem aware bootloader argument is valid. There are other pros that may outweigh the cons. Standard stable use of SDcards expects a filesystem aware firmware. SDcards with file systems greatly enhance the Propeller's project scale. Dave Hein has already done wonderful things in Psft that create a mini OS.
It just may take some adaptation to do it outside of the Forth language.
http://dev.bitquell.de/docs/HIVE%2Fspinix-hive.git;jsessionid=0430C8C0435D349ABE8413B948588BFE
http://forums.parallax.com/showthread.php/143472-pfth-An-ANS-Forth-Interpreter-for-the-Prop
http://forums.parallax.com/showthread.php/149632-A-Linux-like-Propeller-OS-pfth-1.00-with-SDcard/page7
And then there is SPHINX
http://www.learn.parallax.com/inspiration/sphinx-standalone-propeller-development-system
But, I don't think Dave actually implimented an MBR read, just a chained EEPROM rewrite from a binary file on the SDcard.
Following the multi-step loader the same can work also from SD.
Since the SD supports the MBR and its partition table you can have a custom partition (partition type, offset 4 of the partition table entry, eg 50h) so that other OSs understand that this is not fat/ntfs/ext/... filesystem.
The propeller boot partition will be without filesystem and the partition start-end information prevent that other OS write over it.
To boot you can simply scan bytes 446, 462, 478 and 494 for which is defined as active or, if as custom partition is always used partition 1, simply read absolute bytes 447 to 449 (446 is offset of partition table entry 1, +1+2+3 are the offset of the first absolute sector of the partition) and start booting (transferring data) from there, sequentially.
This don't need a filesystem aware code and the code loaded in this way (second stage loader) can then continue its boot process from partition 2 that can be formatted in FAT32 or any other FS
Partition 1 can hold the full image (raw sequential data from the partition space) or only the second stage loader. In the latter case there is no needs for any admin(win)/root(lnx) access because the final image will be in partition 2 and can be also fragmented because the second stage loader in this case will know the filesystem. The real firmware can in this way be upgraded from any PC with a simple file copy operation, you will need some special tool (IIRC eg partition wizard has partition2file and file2partition functions) and root rights only to update the second stage loader stored in partition1.
Someone can in this way use the whole SD like a flash, writing and reading the data in raw if he/she don't need to read then this data on the pc, important is (for boot) that in sector 0 there is the MBR.
In any case it will be nice to have basic spi (or 4bit) hardware support in order to avoid bit-banging the data from sw.
I also think that with modern SD cards it shouldn't be so many compatibility issues between different manufacturers.
I certainly haven't tried solving this one myself but experience with cheap Asian imports tells me to expect the worst. Kye warned that it is a nightmare and I believe he's one of the few here that has the real experience.
In other words, if you want even minimal SD support hardwired into the Prop2, I suggest there is a strong need to prove the code beforehand. Otherwise, I'm confident we're looking at always needing a boot EEPROM to provide an updatable SD firmware.
Hi Dave,
From what I looked at, it seemed possible for pfth to load a program in Forth that could actually overwrite the EEPROM with another binary. Of course, that would loose the Forth environment when a reboot starts the new binary.
But Forth could be used to select and swap the EEPROM binary with one of many alternative binaries stored in files on an SDcard. It would only be a one-shot affair as I presume the new binary would be something other than a Forth environment.
If the Propeller could actually toggle between two binaries on a 64K EEPROM, Forth could be reinstated to allow selection again of another binary to the EEPROM.
I suppose what I am saying is that it may be possible to have Forth provide the Bootloader with SDcard access. But something in the actual Propeller has to be expanded to having a choice between two full binaries on the EEPROM.
My gut feeling is that Forth would provide a useful entry to FAT16/32 any needed services. People that don't want Forth, could simply use it in a limited edition as a boot-loader with specific documented procedures. It would be up to the end-user to compile useful binaries from any resources or language he prefers.
The Forth would just provide the serial port interface and ability to select a binary from SDcard directory to be loaded.
A lot of people here want to extend Propeller use deep into the SDcard, but don't realize how nicely Forth is able to do so. One can actually rotate Forth Dictionaries designed for specific tasks in and out of an SDcard in a rather seamless fashion. So instead of being limited to a mere 32K or less Forth dictionary space, the SDcard becomes part of the Forth dictionary space. A big portion of Hubram is just used to rotate definitions as the tasks require.
Now if we're talking about the P2, then the boot program should probably just be written entirely in PASM. I don't see much reason to use any other language for writing a boot program.
I understand. But it does seem that the Bootloader (either in Forth or PASM) would have to be overwritten, or provided with an alternative page of EEPROM.
I'd be surprised if someone hadn't already done that on the Prop1. All it needs is a physical toggle switch on the extra address line. Toggle between 0v and 3v for selecting the bank to boot from. Next time it's powered up it uses which ever bank has been selected.
If programmatic bank selection is also desired then a bit of external logic along side the switch would be in order.
I went looking, and need to look some more, but I did some SD card testing to vet the whole MBR, fixed block, partition compatible idea.
It is possible using standard Linux disk tools to make an SD card with a boot partition and a FAT partition. Actually, more than one, if you want.
The boot partition could be any multiple of the block size, and it can be first, insuring it's block offset and number of blocks are consistent.
The FAT partition is offset beyond the boot partition, and could be accessible with a post-boot driver.
At the time, we discussed something like 4K being loaded, and that 4K being enough to potentially go fetch an entire image off the SD card, or perhaps do something filesystem aware too. None of this was resolved, due to the idea of just booting off the SD itself being problematic. Kye investigated a lot of SD cards and came to the conclusion that a universal, basic, fast means of reading data, like we do an EEPROM, didn't exist, due to variations in the SD cards themselves.
On a LINUX system you would not need any tools to build a properly partitioned SD card containing a boot partition consisting of a sequential number of blocks, starting at a given block offset. Additional partitions would be optional, and easily supported. I didn't fully explore the Mac, though it has nice disk formatting GUI tools, likely to do the job.
A boot partition can be marked with a type that made it easy to spot, should code actually get smart enough to parse the MBR. This can also be done to minimize the corruption possibility by making it difficult to mount, and on windows, invisible. User just sees an SD card with a filesystem.
Once the partitions are done, a simple use of the "dd" command would move a binary file into the boot partition easily enough. Again, that's just UNIX.
The Windows FDISK program does write an MBR, and it does allow for setting partition types, etc... and would likely work in the same way, meaning a windows user would also be able to prepare an SD card too. I didn't find anything that worked like the "dd" command for filling the boot partition easily however.
It's my own personal opinion a bootable SD, should have a boot partition, due to it's lack of proper filesystem support.
Various other schemes, including fresh format, copy file, etc... to arrive at fixed offsets were discussed. So long as the partition type was FAT, this appeared to work also, and did close the gap in windows missing the "dd" type functionality.
On one of those threads, I believe I did upload an SD card image and some commentary about what I learned.
A P1, or P2 could very easily take a file and prep an SD card as an alternative to having it done on some operating system.
That's what I remember.
Frankly, I really liked this idea, but I found Kye's skepticism compelling. Did the exploration work anyway, just sort of hoping an overall answer would be found.
A tool to populate the SD card on a P1 or P2 wouldn't be a big deal, nor would a windows tool. Just needs block access to the SD, and either one could work from an initial image too.
There was some discussion about larger sized SD cards using something other than FAT. I found I was able to format them as one would do a hard disk. Just use the smaller filesystem and give up the space, or make multiple partitions.
It also is educational in that it expands awareness of what a boot partition is and the ability to have more partitions with different file systems.
Something less just represents a hack of the SDcard rather than really exploring and learning.
The MBR contains a small amount of 8086 code that is used to bootstrap a DOS or Windows OS. I believe later versions of Windows and other OSes, such as Linux use code in a boot partition to boot up the OS. From what I can tell, the boot partition is normally the first partition on the device. At least that's my understanding from what I've read.
Since the current Prop SD drivers use the first partition for the file system we probably want to avoid using that for the boot code. The boot code could either reside in another partition, or it could just be a file in the first partition. Making it a file in the first partition would make it easier to write the boot program to an SD card. The file will consist of sequential sectors if it is smaller than the cluster size, or if it is written to a freshly formatted SD card.