There is a simpler option to adding an extra partition and that is to use the reserved space at the start of the partition.
I see the separate partition idea as being used for when filesystem support is not wanted at all.
Just to repeat: All three options are open with the current idea of the pointer in block 0. The ROM code won't care which is used on a particular SD card. And that includes partitionless configs.
Yes, I was saying TWO partitions. One for the Prop boot image. Personally, I don't like the hacks in ROM. They are great, don't get me wrong. I think they would work too. It's not that.
It's all about using the device properly and doing so in a standards based way. If we don't have filesystem support, the right answer is a partition. That is why they exist. Doing it in the *right* way, standard in other words, is part of a pro-quality device, IMHO. Supporting filesystem hacks is different from supporting a partition too. Once again, great hacks, I'm not using that in a negative way here. Just would prefer standards compliant operation, that's all. I'll not say any more about it, other than I really don't care whether or not it boots SD at all. The other storage solution can very easily support an SD card, much like we do now on P1, and it could do so with a far more robust process, and can be updated / changed easily too.
You keep saying that using the Master Boot Record to boot with is a hack, but isn't booting what it was meant for? Is there some spec that says you cant use those 440Bytes how you want to boot? Either way you go about it, separate partition, or modify the MBR with sector numbers pointing to a boot file, both are not end user friendly. I find the partition stuff harder actually, because I have done it a few times for my ZipitZ2's.
With any SD, even one that has a bunch of stuff on it, you can modify the MBR. I would recommend, however that a contiguous space not be required as there is no need for this. There is plenty of space for cluster numbers or block numbers n the MBR. Of course, this may be a problem if there simply space in the ROM for the logic to load the different segments. If it can be loaded from multiple segments, I would like the ability to load a full 126KB load.
I'd be happy to throw in my 2 cents worth on this whole matter of booting but I'm sure with all you cooks in the kitchen that we will eventually have something edible
Here's quick question in another direction altogether. Some mention has been made about FFTs on the current Prop which I am not interested in for this purpose but I am interested in knowing how well an FFT could perform on the Prop II in the audio spectrum. Could P2 be used in place of a DSP for equalizing and compression etc. Any ideas?
Peter,
You were summoned, not to comment on how many cooks are in the kitchen, but rather to write us a small Kick *** forth system to go into the ROM in the P2 ;^)
Since this was brought up, I can say that I am torn between a small forth monitor and SDcard booting. With a forth monitor, we would have the ability to boot via serial so it could be done over cell phone modem, zigbee, bluetooth, etc, and for us hacks, even a dumb terminal! I REALLY like forth, for the little I have been exposed to it.
Booting on the SD card would probably be more used by more people.
Why You need it to have possibility to modify..
As I know none of systems can that without reformatting
This can be done. Just keep writing the file over and over until you get a contiguous one (by inspecting the fat). Then remove the previous copies.
I would recommend, however the the file not be required to be contiguous...if the logic can fit in the ROM.
1. Load the first pointer and the first runlength from the MBR. Read runlength blocks from the SD starting at pointer.
2. Load the 2nd pointer and the second runlength from the MBR. Read runlength blocks from the SD starting at the ponter.
Worst case, is 32 pairs since the smallest cluster size is 4KB, or 8 blocks(Is it safe to assume 512byte blocks?)
Runlength only needs to be a byte. Pointer, does it need to be over 24bits? If so, you could fit in 128 Bytes. Even if you were wasteful, and used 32bit pointers and 32bit runlengths, it would only take up 256Bytes of the 440 free in the MBR.
Just been trying to work out how modern Linuxes get booted ... not that easy to identify, most documentation, particularly webpages like Wikipedia, just talks about the general process as a concept and completely miss the technical detail of how the main loader is loaded itself.
Finally found it on page 41 of the Grub2 manual under the heading of "GRUB image files". The critical file being diskboot.img. It's a single 512 byte block and contains a list of block numbers and counts for loading the rest of the loader images that contain the filesystem driver for the root partition and basic console/shell that then loads Linux itself.
So, Linux needs a tool to build booting just like everyone else using the PC, but it's done away with needing contiguous files or any reserved space.
EDIT: I should clarify by saying that that's just the norm for your average BIOS based PC. It's certainly not the only way Grub can operate, hence the vagueness of the general documentation.
EXACTLY If we can fit it, let's load this way. We have 440 bytes in the MBR, let's use them.
You could then have many boot files on your SD, and then just have a utility to modify the MBR to point to another list of block numbers and runlengths, like Installboot used to do for us back in 1989 on the SunOS. We could have several kernel's sitting there and just run installboot when we wanted to try out a different one.
Another idea is to make the MBR itself as executable COG code so it can have the logic and list of blocks to load.
As executable code, maybe someone clever can even implement a boot menu with a few boot options.
Having part of the SD boot code on the SD MBR itself means that you could put less of it in the Cog, and it is upgradable.
Of course any SD boot option can lead you to another stage to implement a boot menu, or load a file from the FAT, but fewer stages=less confusion, less time booting, etc.
Any way it is done, I would HIGHLY recommend not requiring contiguous files.
This can be done. Just keep writing the file over and over until you get a contiguous one (by inspecting the fat). Then remove the previous copies.
I would recommend, however the the file not be required to be contiguous...if the logic can fit in the ROM.
1. Load the first pointer and the first runlength from the MBR. Read runlength blocks from the SD starting at pointer.
2. Load the 2nd pointer and the second runlength from the MBR. Read runlength blocks from the SD starting at the ponter.
Worst case, is 32 pairs since the smallest cluster size is 4KB, or 8 blocks(Is it safe to assume 512byte blocks?)
Runlength only needs to be a byte. Pointer, does it need to be over 24bits? If so, you could fit in 128 Bytes. Even if you were wasteful, and used 32bit pointers and 32bit runlengths, it would only take up 256Bytes of the 440 free in the MBR.
From what we've all read, there is enough room to mount the SD, identify where the boot image is, and fetch it as blocks. There isn't enough room to actually provide filesystem support, and that's a key point here in favor of the partition.
The hack part is making a lot of assumptions about what other tools may or may not do to the boot data.
From what we've all read, there is enough room to mount the SD, identify where the boot image is, and fetch it as blocks. There isn't enough room to actually provide filesystem support, and that's a key point here in favor of the partition.
The hack part is making a lot of assumptions about what other tools may or may not do to the boot data. For example, if a utility is used to deal with the MBR, it might overwrite the bytes needed to point to where some boot data blocks are. Different versions of operating systems store different amounts of executable code in the MBR as well. Compare older MBR records to newer ones and significant differences can be seen. I think the Win7 MBR actually does stomp on the bytes we are discussing, where the older DOS / Win 98 MBR record doesn't, for example. I'm quite sure there are others. In any case, the MBR is used for booting, but HOW it's used differs from how a Prop boots. In particular, the space in the MBR is for CPU executable code and partition info, and some meta-data about the disk, such as the serial number. We aren't putting executable code in there, we are putting pointers in there instead. That's just not what that portion of the MBR is for. I'm not saying it won't work, or even that it won't work well. I am saying, that's not best practice use of the MBR.
So since there are several different ways MicroSoft has used the MBR in the past it is a hack not to do it one of those ways? That would only make sense if you wanted the same SD card bootable for something MicroSoft as well as propeller.
Actually, I think putting executable code in the MBR may be a good option for the P2.
On what basis are you saying: "that's not best practice use of the MBR"?
Right from http://en.wikipedia.org/wiki/Master_Boot_Record "Other MBR code contains a list of disk locations...requires that the embedded list of disk locations be updated when changes are made that would relocate the remainder of the code."
Is this not EXACTLY what I am talking about?
On the filesystem level, there are assumptions about where the file is, how it's structured, etc... We need a sequential set of blocks, due to ROM limitations and we need a simple pointer to them, due to ROM limitations,
Not exactly. You don't have to access the FAT, nor do you even care if it is not a fat as long as you have a list of the starting block numbers and runlengths of each segment, it doesn't need to be a contiguous either.
and those limitations mean we don't actually use the file system, but are proposing to just kind of operate along with it in a way that will work in a lot of cases, depending.... That's a hack too, as is writing data to the lead in space to the filesystem, because the assumption is those things are constant. In any case, it's not best practice at all.
Actually, with a list of starting block numbers and run lengths, you can make it work in almost all cases, without depending on a contiguous file, without even depending on there being a FAT. It could even be XFS. If you have the list of block numbers and run lengths, it just doesn't matter.
When a region of the disk is needed, and there isn't filesystem support, or said filesystem is unique, partitions are used to allocate the storage blocks on the disk. Writing a partition insures that the necessary region of the disk is accounted for, and that other filesystems / uses of the disk won't conflict with that necessary region. This is best practice. Partitions are the way we identify a portion of a disk for our purposes and provide a pointer to said portion. Anything higher level than that requires filesystem support, which we won't have.
Is there something that uses the MBR, but not to boot that we need to concern ourselves with? If not, it just doesn't matter. The process would be write your file to the SDcard, and then run "installboot /dev/SDdevice propboot.img" or something like it. Of course, installboot would have to know about the filesystem. When you write your file on a pc(assuming fat), the fat gets updated, the file is visible and readable, and should not be trounced by anything else.
If we define a partition, and we size it to contain our boot image, it's going to work on those storage devices that have an MBR, and it's going to work, even when those devices also contain a filesystem or systems, and it's going to work no matter what one does with said file systems too. The ideas proposed here that don't actually incorporate a storage partition definition will work, if... and there are a lot of dependencies and assumptions in that "if" that really don't have to be there, and aren't best practice. A.K.A. "hacks"
Then you have a file that is not visible to some other operating system without special tools. On winders you couldn't just mount up the card and see what files you could be booting.
If you had an exact size replacement for the boot file, I believe you could even overwrite it without updating the MBR because the blocks used would be the same.
On UNIX, that means being able to prepare an SD card for booting on a Propeller with standard tools, again best practice.
Just because we have dd on linux, it doesn't mean it is best practice to use it. I don't have to tell you about the nasty things you can do with a typo using dd.
If you are not referring to dd, what "standard tools" are you referring to?
Windows is a different beast that will require some help no matter what. It's not a big deal to write a program that does it right, or use a Propeller to do it right, both of which the Windows user will be both capable of using and or in possession of.
So, we are doing a professional device right? Let's do the SD handling correctly then, meaning we define our boot partition so that it operates consistently in tandem with both the Propeller and other devices that may need to interact with the disk, and let's do that to best practices, because that's what professionals do. That's it really.
Once again, you are trying to say that not using the Master Boot Record for booting is not correct, and using a separate partition is. Using the MBR for booting, and having a list of blocks and runlengths in the MBR will operate correctly with propeller2, linux, winders of many varieties, freebsd, etc. Obviously, you can't make it bootable for more than one of those, but as far as reading and writing files, there is no hindrance. There is a hitch: sometimes when you go to install winders on a system, if you have any media out there, it might just wipe out the MBR of some media that you aren't even trying to install too. For this case, don't have your P2 boot SD card in a system while you are installing winders.
Any other caveats you know of potatohead?
Edit: That also tends to define which SD cards are appropriate. Either they can accept a partition or not. If not, they aren't supported, end of story. That's clean and easy. Given the large number of devices out there that utilize boot partitions, I suspect this isn't ever really going to be an issue beyond insuring that one does in fact source supported SD cards.
Another Edit: I just saw Sapieha's last post about freshly formatted media. That's yet another file system assumption. If we had filesystem support in the ROM, this assumption would not be on the table, because the Prop could simply get the file from the file system. But we don't have file system support.
When partitions are used, there is no requirement for freshly formatted media.
Once again, we don't need file system support to boot a non contiguous file from the media as long as we have a list of blocks & runlengths in the MBR....where it belongs.
We don't need freshly formatted media, we don't need to know if it is FAT or something else, we don't need a contiguous file.
The partition will be sequential and in the place it's supposed to be no matter what the state of the file system partition(s) may be in. Additionally, if there isn't room for the tiny Propeller boot partition, the user can choose to format the card with the partition, or resize existing ones, etc... The end result of that is a consistent, robust, best practice boot area to boot from.
Re: Tiny Forth Monitor / or just monitor.
I honestly would just want a nice monitor. A forth can be sent over, and actually anything the user wanted could be sent over. From there, execute it, and carry on. Very nice. A monitor could be really tiny too. I would take that over the SD boot capability, because the other storage solution would offer more robust SD card booting; namely, filesystem support, rendering a lot of discussion moot.
A simple monitor is really lean, and not obtuse. Starting with a Forth would be a bit more thick, and if it's going over the wire, simple, small, robust is best, IMHO
simple, small, robust......sound's like a good description for forth;^)
All boot systems that use RAW read mode of media
NEED have them contiguous.
It is why them need freshly formated media.
Nope. You are mistaken. Linux GrUB doesn't. LiLO doesn't. SunOS didn't 23 years ago! Many don't. The reason is you are limited to just 1 block number and 1 length. You can have a whole list of them. There are 440 bytes to use for this purpose in the MBR. The only limitation is putting in the loop to read in the blocks on the list. Hopefully there is enough room for it in the ROM because it would make the prepping of media to boot a P2 a whole lot easier. It would be. Put the file on the media with winders, linux, freebsd, kyedos, or whatever, and run a program like installboot....even on a different OS. Installboot would have to know the filesystem(in most cases, meaning reading the FAT).
How many block numbers are you planning on storing? If you are loading a 126KB image, potentially you could have 252 blocks.
I am not that familiar with clusters, but if you address clusters instead of blocks, and the clusters are only 4k, it could take 32 of them, and we cannot assume that they are one right after the other.
My method does not care... 2x512byte sectors are loaded and it it this that will contain the remaining code to boot the prop. Therefore this part is not in ROM. It may know about the SD format or not, and this is totally up to the user.
I have asked for consensus for the MBR offset so that at least we could interchange our code. No comments??
Don't SDHC cards respond to basic SD commands as well as their extended commands? If they don't then they are not backward compatible and all those older cameras cannot use them! Won't SDXC cards also be backward compatible?
AGAIN... MY METHOD DOES NOT RELY ON ANY SD CARD FORMATTING - I DON'T CARE ABOUT PARTITIONS, FAT16, FAT32, NTFS, OR ANY OTHER EXISTING OR NEW FORMAT.
Currently, heater & I use 32MB contiguous files to represent disks under CPM2.2. I locate the first sector for each file and use this as the start of the block of sectors. We then access this file as sectors relative to the base data sector, all via direct sector addressing. This way, CPM did not require changing. With a simple change, we could have used a partition(s) for this. This access is all soft so it can be simply changed. I just pointing this out to show I have no pre-disposed idea about what format should be used on the SD card. My proposed ROM boot code simply does not care - we can decide later to agree or disagree - it simply does not matter.
Postedit: BTW "Mount" is not performed in the ROM code. Specifically, it should not be in ROM, but in the boot code on the SD card and most likely second stage. The ROM should only perform minimum code (we also have size constraints) to get the booting started, and we also require that to conform to the Flash boot as well.
The MBR is specifically designed for booting purposes. There are other details that may or may not be contained on this record/sector. Using a few bytes to store an identification and pointer to 2 sectors containing 1024 bytes of first stage boot code seems to be the simplest, most reliable method. I have posted code to update the MBR sector using a P1. I use the FAT file system on the PC to update the boot code and this seems to be the easiest for me, rather than seperate partitions. But I am not going to dictate that way to you - you choose.
Hinv:
I believe the current plan is to just load 1kB using the ROM. Apparently this is what is done with the SPI flash loading. Control is past to it and it can decide how the rest is loaded. On that basis, one can be reasonably confident of two blocks being consecutive in an ordinary file. Certainly any FAT filesystem is guaranteed to be so.
That 1kB can then have a map of the rest of it's parts if it likes. Like the other ways, the ROM won't care.
Regarding the MBR, I don't see any advantage in using the MBR for storing code, just the pointer is fine - well, I suppose two pointers, one for each 512 byte block would work too. And we do get a bonus for not filling the MBR completely, we can freely do a partitionless boot (Into a FAT filesystem at the very least).
Don't SDHC cards respond to basic SD commands as well as their extended commands? If they don't then they are not backward compatible and all those older cameras cannot use them! Won't SDXC cards also be backward compatible?
Wikipedia says that all SD cards always power up in 3.3 volt SPI mode. Other modes have to be given a command to switch in.
Where did you see a partial load plan? That changes a lot of things.
Cluso said it right back at the start when Chip was still reading.
EDIT: A tool will still be needed to build the map with. Just doesn't have to do direct block writes, for forcing a contiguous file, which is a decent bonus of course.
I personally don't like long chain boots, because I have been bitten by them in the past....a machine saying Li............., or grub stage 1.5 failure.
I would rather have a separate partition load than add yet another chain link to boot.
I would rather load up to all 126KB at once at 20MHz, wouldn't this only take like 1/5th of a second?
@cluso: where are you getting these 2x512byte blocks again? Are they going to be allocated in the fat so that you don't trounce them?
So you are talking:
1 Load the MBR
2 Load the 2 blocks(1024 bytes total) that the MBR points to
3 Run the 1k that you just loaded to load something else.
Can you fit the filesystem code in that 1k? so that you can then boot a file.
If we aren't going to do a full load after the MBR, the chain is just getting to long for me. This is one thing that maid LiLO a PITA....and to some extent Grub too. I have wasted man-weeks on long chain loaders not doing what I expected. I have NEVER had a problem booting IRIX from the boot partition if it was set up correctly. When it was not set up correctly, I could tell with the dvhtool utility, not wonder why some part of the chain didn't get loaded. Now looking at it in that light, I would rather just fetch the target, the WHOLE thing, right out of a boot partition, as simple as it gets. Potatoehead, I'm with you. There are drawbacks, but a 3+ stage boot for me is too risky.
The ROM looks to the MBR as just a pointer, or two, to the first two blocks of the boot loader. No code in the MBR.
Those first two blocks would be the first two blocks of the boot loader. Ok, so it's a split loader but that's the price of not making it contiguous. It's still only two parts.
That is only one option ...
The main idea is to get a mostly non-changing boot loader that can read files after it's loaded. After that, the main program can be loaded. Same thing with the contiguous boot loader file except that really is a single file.
If you want a really large boot image that still reads files I'd go with reserved space instead. Reserved space works for small loader too. If you don't care about files at all then a custom partition type would work best.
EDIT: PS: The ROM doesn't even try to work out partitions. The pointers are raw device block numbers. Only the fully loaded filesystem handler needs to work out the partition stuff.
I personally don't like long chain boots, because I have been bitten by them in the past....a machine saying Li............., or grub stage 1.5 failure.
I would rather have a separate partition load than add yet another chain link to boot.
I would rather load up to all 126KB at once at 20MHz, wouldn't this only take like 1/5th of a second?
It is a requirement of the Flash loader and security to preload a small loader with encryption keys etc. I am just conforming to those requirements.
@cluso: where are you getting these 2x512byte blocks again? Are they going to be allocated in the fat so that you don't trounce them?
They can be anywhere you decide to put them...
* As the data in a file within a filesystem
* As data within a partition
* As raw data anywhere on the SD card
So, the location is not defined by me - it's your choice. My preference is a file, and that is what my test code does.
So you are talking:
1 Load the MBR
2 Load the 2 blocks(1024 bytes total) that the MBR points to
3 Run the 1k that you just loaded to load something else. 4 Can you fit the filesystem code in that 1k? so that you can then boot a file.
1 Yes
2 Yes
3 Yes
4 Probably not.
But for now, I actually load the full Prop1 32KB starting from the 2 blocks. These 2 blocks will contain the pass key for the security and enough code to tell the P2 what to do, and that is likely to say load a few more KB following these 2 sectors, and this will contain the filesystem code and a filename to boot the code from. The beauty of my proposal, is that it does not have to be finalised before Chip commits to ROM code. And if we decide later on some other method, it will still all work.
If we aren't going to do a full load after the MBR, the chain is just getting to long for me. This is one thing that maid LiLO a PITA....and to some extent Grub too. I have wasted man-weeks on long chain loaders not doing what I expected. I have NEVER had a problem booting IRIX from the boot partition if it was set up correctly. When it was not set up correctly, I could tell with the dvhtool utility, not wonder why some part of the chain didn't get loaded. Now looking at it in that light, I would rather just fetch the target, the WHOLE thing, right out of a boot partition, as simple as it gets. Potatoehead, I'm with you. There are drawbacks, but a 3+ stage boot for me is too risky.
As I have said above, there are other security requirements in play here. I have no control of this, and I have not even had the time to look since Chip posted the code a few days ago.
If Chip decides that more than 1K (2 sectors) needs to be loaded then this can be done. There will be a number of sectors in the cluster that will be contiguous.
We don't know that it isn't a single stage. I've not seen a commit to a partial load yet, have any of you?
See my post above, and all my other posts. There will be a 1K load initially. What happens thereafter isn't dictated by the ROM, but by this 1K of code, after security validation. So, apart from the security conforming part, it is up to you.
Every system on PC need in boot stage use BIOS raw read mode to load bootstrap before it can RUN its own more advanced Loader/Booter..
That You never come from.
Nope. You are mistaken. Linux GrUB doesn't. LiLO doesn't. SunOS didn't 23 years ago! Many don't. The reason is you are limited to just 1 block number and 1 length. You can have a whole list of them. There are 440 bytes to use for this purpose in the MBR. The only limitation is putting in the loop to read in the blocks on the list. Hopefully there is enough room for it in the ROM because it would make the prepping of media to boot a P2 a whole lot easier. It would be. Put the file on the media with winders, linux, freebsd, kyedos, or whatever, and run a program like installboot....even on a different OS. Installboot would have to know the filesystem(in most cases, meaning reading the FAT).
@Sapieha - My file system driver (the FATEngine) can boot form non-contiguous files located on the SD card. This is important for people who want stuff to just work. Not require caveats. So, you don't have to freshly reformat the disk to use the FATEngine to boot the propeller chip from an SD card. Mike Green specifically asked for this feature. It works by walking the FAT and building a list of sectors that the cog ASM driver then uses to fill memory with.
I don't believe I've been attacking you Sapieha. So, I'm not quite sure why your are calling me a know nothing. I've spent 3 years of my life understanding all the ins and outs of SD cards and the FAT file system. I know you are older than me and have way more knowledge about many things. But, I'm not ignorant of what I say on this thread at least.
The only reason I've been posting to this thread is just to appeal to people that whatever is done for the P2 needs to be fast, flexible, and works with caveats. I'm tired of having to defend the propeller as my choice uP. ARM processors do this stuff without caveats at the fastest speed possible. The propeller 2 needs to do the same.
So many people call me out for using the P1... it doesn't have C... now, I hear, it can't run C well. Or, I hear, the architecture sucks, or any other thing they don't like about it because its not an ARM or AVR (interestingly no-one I've listened to defends PICs). The P2 is a chance to get away from these problems. This is why I am posting here.
---
Anyway, I don't have anything more to contribute to this discussion. I'm out.
Sorry if I was to STRANGE to You.
BUT in some cases You write in terms that Yours answers NOT include others peoples questions only answers that You think are important to You..
(Many times Yours answers like --- MY are better -- without explaining many time why) In this Place at least I miss that You Say --- That Clusos else Ariba's not mention other people that posted theirs code spinets how them look on that SD-Dirver.
That make many people annoyed -- Inclusive me.
That is OK -- BUT what happen if I will don't have FAT system ---> Lets SAY I will build LINUX else other OS that have its own FILE System. And that function need be usable to all customers.
To that - That driver need even can read raw file from any position of SD ----> Usually at starting from Sector 1 (0 is always reserved)
As Propeller II will have power to build Advanced systems with its own File systems and so on.
Only thing that are important in that driver are --- That it can read RAW file format (if to that it even can read information by FAT positioning is one + to it) CAVEATS --- for many will be if them can't boot-(LOAD) THEIRS own FILE system handler to handle THEIRS own FILE system OS
Now even I'm are out from more discussion on subject --- Only if Chip ask any question.
@Sapieha - My file system driver (the FATEngine) can boot form non-contiguous files located on the SD card. This is important for people who want stuff to just work. Not require caveats. So, you don't have to freshly reformat the disk to use the FATEngine to boot the propeller chip from an SD card. Mike Green specifically asked for this feature. It works by walking the FAT and building a list of sectors that the cog ASM driver then uses to fill memory with.
I don't believe I've been attacking you Sapieha. So, I'm not quite sure why your are calling me a know nothing. I've spent 3 years of my life understanding all the ins and outs of SD cards and the FAT file system. I know you are older than me and have way more knowledge about many things. But, I'm not ignorant of what I say on this thread at least.
The only reason I've been posting to this thread is just to appeal to people that whatever is done for the P2 needs to be fast, flexible, and works with caveats. I'm tired of having to defend the propeller as my choice uP. ARM processors do this stuff without caveats at the fastest speed possible. The propeller 2 needs to do the same.
So many people call me out for using the P1... it doesn't have C... now, I hear, it can't run C well. Or, I hear, the architecture sucks, or any other thing they don't like about it because its not an ARM or AVR (interestingly no-one I've listened to defends PICs). The P2 is a chance to get away from these problems. This is why I am posting here.
---
Anyway, I don't have anything more to contribute to this discussion. I'm out.
I see, sorry Sapieha . I don't respond to everyone. I see how this can get frustrating.
---
Something to think about... whatever code that will be used for the SD card driver needs to be able to get the card out of multi block mode. No one's mentioned this but the SD card booting scheme will fail because of multi block mode if the boot loader does not support it.
This is because people will likely use the SD card in multi block mode during run time. But, if the boot loader code cannot get the card out of multi block mode your system will need to be power cycled to be reset.
This is a serious issue... I alerted Chip to this in my conversation with him. It's a hidden danger.
So many people call me out for using the P1... it doesn't have C... now, I hear, it can't run C well. Or, I hear, the architecture sucks, or any other thing they don't like about it because its not an ARM or AVR (interestingly no-one I've listened to defends PICs). The P2 is a chance to get away from these problems. This is why I am posting here.
Yeah. That is really what it all should be about. Good on you for that Kye.
This is because people will likely use the SD card in multi block mode during run time. But, if the boot loader code cannot get the card out of multi block mode your system will need to be power cycled to be reset.
I just started reading a piece of the spec and was noting how it kept saying certain modes are one way only. It's looking a bit like, officially, such reversals aren't supported ... Which leads to conclusion that software needs control of SD card power pin.
Or one of the caveats is: Only one mode supported on boot device.
When I was a budding SysAdmin at Motorola, we had over 1000 Sun3 machines that booted a boot block that loaded the kernel. In order to make a disk bootable, you had to run a utility called "installboot" which wrote the boot block telling where(what blocks) to find the kernel. I don't remember how big the kernel (SunOS version of unix) was, but it had to be in the 100s of KB at least. The way I figure it, if you are going for a cog ram only boot, that will take exactly 4 longs out of a 512Byte MBR for the block numbers. If you are going for a full hub load, it would take 252 longs, too much for a MBR(512 Bytes), but if you had block numbers followed by how many in a run(run length encoding), you could get it loaded in just about every case(except the most fragmented ones).
The beauty of this approach is it has already been done, works reliably(for hard disks) and doesn't mess with the filesystem.
Just throwing in my 10 bits
Doug
Where and When did you work for Motorola? I was also a sys admin at Mot!!
Comments
I see the separate partition idea as being used for when filesystem support is not wanted at all.
Just to repeat: All three options are open with the current idea of the pointer in block 0. The ROM code won't care which is used on a particular SD card. And that includes partitionless configs.
You keep saying that using the Master Boot Record to boot with is a hack, but isn't booting what it was meant for? Is there some spec that says you cant use those 440Bytes how you want to boot? Either way you go about it, separate partition, or modify the MBR with sector numbers pointing to a boot file, both are not end user friendly. I find the partition stuff harder actually, because I have done it a few times for my ZipitZ2's.
With any SD, even one that has a bunch of stuff on it, you can modify the MBR. I would recommend, however that a contiguous space not be required as there is no need for this. There is plenty of space for cluster numbers or block numbers n the MBR. Of course, this may be a problem if there simply space in the ROM for the logic to load the different segments. If it can be loaded from multiple segments, I would like the ability to load a full 126KB load.
Peter,
You were summoned, not to comment on how many cooks are in the kitchen, but rather to write us a small Kick *** forth system to go into the ROM in the P2 ;^)
Since this was brought up, I can say that I am torn between a small forth monitor and SDcard booting. With a forth monitor, we would have the ability to boot via serial so it could be done over cell phone modem, zigbee, bluetooth, etc, and for us hacks, even a dumb terminal! I REALLY like forth, for the little I have been exposed to it.
Booting on the SD card would probably be more used by more people.
This can be done. Just keep writing the file over and over until you get a contiguous one (by inspecting the fat). Then remove the previous copies.
I would recommend, however the the file not be required to be contiguous...if the logic can fit in the ROM.
1. Load the first pointer and the first runlength from the MBR. Read runlength blocks from the SD starting at pointer.
2. Load the 2nd pointer and the second runlength from the MBR. Read runlength blocks from the SD starting at the ponter.
Worst case, is 32 pairs since the smallest cluster size is 4KB, or 8 blocks(Is it safe to assume 512byte blocks?)
Runlength only needs to be a byte. Pointer, does it need to be over 24bits? If so, you could fit in 128 Bytes. Even if you were wasteful, and used 32bit pointers and 32bit runlengths, it would only take up 256Bytes of the 440 free in the MBR.
EXACTLY If we can fit it, let's load this way. We have 440 bytes in the MBR, let's use them.
You could then have many boot files on your SD, and then just have a utility to modify the MBR to point to another list of block numbers and runlengths, like Installboot used to do for us back in 1989 on the SunOS. We could have several kernel's sitting there and just run installboot when we wanted to try out a different one.
Another idea is to make the MBR itself as executable COG code so it can have the logic and list of blocks to load.
As executable code, maybe someone clever can even implement a boot menu with a few boot options.
Having part of the SD boot code on the SD MBR itself means that you could put less of it in the Cog, and it is upgradable.
Of course any SD boot option can lead you to another stage to implement a boot menu, or load a file from the FAT, but fewer stages=less confusion, less time booting, etc.
Any way it is done, I would HIGHLY recommend not requiring contiguous files.
All boot systems that use RAW read mode of media
NEED have them contiguous.
It is why them need freshly formated media.
From what we've all read, there is enough room to mount the SD, identify where the boot image is, and fetch it as blocks. There isn't enough room to actually provide filesystem support, and that's a key point here in favor of the partition.
The hack part is making a lot of assumptions about what other tools may or may not do to the boot data.
Actually, I think putting executable code in the MBR may be a good option for the P2.
On what basis are you saying: "that's not best practice use of the MBR"?
Right from http://en.wikipedia.org/wiki/Master_Boot_Record "Other MBR code contains a list of disk locations...requires that the embedded list of disk locations be updated when changes are made that would relocate the remainder of the code."
Is this not EXACTLY what I am talking about? Not exactly. You don't have to access the FAT, nor do you even care if it is not a fat as long as you have a list of the starting block numbers and runlengths of each segment, it doesn't need to be a contiguous either. Actually, with a list of starting block numbers and run lengths, you can make it work in almost all cases, without depending on a contiguous file, without even depending on there being a FAT. It could even be XFS. If you have the list of block numbers and run lengths, it just doesn't matter. Is there something that uses the MBR, but not to boot that we need to concern ourselves with? If not, it just doesn't matter. The process would be write your file to the SDcard, and then run "installboot /dev/SDdevice propboot.img" or something like it. Of course, installboot would have to know about the filesystem. When you write your file on a pc(assuming fat), the fat gets updated, the file is visible and readable, and should not be trounced by anything else. Then you have a file that is not visible to some other operating system without special tools. On winders you couldn't just mount up the card and see what files you could be booting.
If you had an exact size replacement for the boot file, I believe you could even overwrite it without updating the MBR because the blocks used would be the same. Once again, where are you getting your definitions of what is and what is not best practice? Just because we have dd on linux, it doesn't mean it is best practice to use it. I don't have to tell you about the nasty things you can do with a typo using dd.
If you are not referring to dd, what "standard tools" are you referring to? Once again, you are trying to say that not using the Master Boot Record for booting is not correct, and using a separate partition is. Using the MBR for booting, and having a list of blocks and runlengths in the MBR will operate correctly with propeller2, linux, winders of many varieties, freebsd, etc. Obviously, you can't make it bootable for more than one of those, but as far as reading and writing files, there is no hindrance. There is a hitch: sometimes when you go to install winders on a system, if you have any media out there, it might just wipe out the MBR of some media that you aren't even trying to install too. For this case, don't have your P2 boot SD card in a system while you are installing winders.
Any other caveats you know of potatohead? Once again, we don't need file system support to boot a non contiguous file from the media as long as we have a list of blocks & runlengths in the MBR....where it belongs.
We don't need freshly formatted media, we don't need to know if it is FAT or something else, we don't need a contiguous file. simple, small, robust......sound's like a good description for forth;^)
Best of luck with that over the longer haul, should it ship that way.
I have asked for consensus for the MBR offset so that at least we could interchange our code. No comments??
Don't SDHC cards respond to basic SD commands as well as their extended commands? If they don't then they are not backward compatible and all those older cameras cannot use them! Won't SDXC cards also be backward compatible?
AGAIN... MY METHOD DOES NOT RELY ON ANY SD CARD FORMATTING - I DON'T CARE ABOUT PARTITIONS, FAT16, FAT32, NTFS, OR ANY OTHER EXISTING OR NEW FORMAT.
Currently, heater & I use 32MB contiguous files to represent disks under CPM2.2. I locate the first sector for each file and use this as the start of the block of sectors. We then access this file as sectors relative to the base data sector, all via direct sector addressing. This way, CPM did not require changing. With a simple change, we could have used a partition(s) for this. This access is all soft so it can be simply changed. I just pointing this out to show I have no pre-disposed idea about what format should be used on the SD card. My proposed ROM boot code simply does not care - we can decide later to agree or disagree - it simply does not matter.
Postedit:
BTW "Mount" is not performed in the ROM code. Specifically, it should not be in ROM, but in the boot code on the SD card and most likely second stage. The ROM should only perform minimum code (we also have size constraints) to get the booting started, and we also require that to conform to the Flash boot as well.
The MBR is specifically designed for booting purposes. There are other details that may or may not be contained on this record/sector. Using a few bytes to store an identification and pointer to 2 sectors containing 1024 bytes of first stage boot code seems to be the simplest, most reliable method. I have posted code to update the MBR sector using a P1. I use the FAT file system on the PC to update the boot code and this seems to be the easiest for me, rather than seperate partitions. But I am not going to dictate that way to you - you choose.
Hinv:
I believe the current plan is to just load 1kB using the ROM. Apparently this is what is done with the SPI flash loading. Control is past to it and it can decide how the rest is loaded. On that basis, one can be reasonably confident of two blocks being consecutive in an ordinary file. Certainly any FAT filesystem is guaranteed to be so.
That 1kB can then have a map of the rest of it's parts if it likes. Like the other ways, the ROM won't care.
Regarding the MBR, I don't see any advantage in using the MBR for storing code, just the pointer is fine - well, I suppose two pointers, one for each 512 byte block would work too. And we do get a bonus for not filling the MBR completely, we can freely do a partitionless boot (Into a FAT filesystem at the very least).
I liked the $190 location, staying close to the MBR partition table means staying further away from other parameter tables in the VBR.
Wikipedia says that all SD cards always power up in 3.3 volt SPI mode. Other modes have to be given a command to switch in.
I've not read any of it but this may be of help -
http://www.sdcard.org/downloads/pls/simplified_specs/Part_1_Physical_Layer_Simplified_Specification_Ver_3.01_Final_100518.pdf
Cluso said it right back at the start when Chip was still reading.
EDIT: A tool will still be needed to build the map with. Just doesn't have to do direct block writes, for forcing a contiguous file, which is a decent bonus of course.
I would rather have a separate partition load than add yet another chain link to boot.
I would rather load up to all 126KB at once at 20MHz, wouldn't this only take like 1/5th of a second?
@cluso: where are you getting these 2x512byte blocks again? Are they going to be allocated in the fat so that you don't trounce them?
So you are talking:
1 Load the MBR
2 Load the 2 blocks(1024 bytes total) that the MBR points to
3 Run the 1k that you just loaded to load something else.
Can you fit the filesystem code in that 1k? so that you can then boot a file.
If we aren't going to do a full load after the MBR, the chain is just getting to long for me. This is one thing that maid LiLO a PITA....and to some extent Grub too. I have wasted man-weeks on long chain loaders not doing what I expected. I have NEVER had a problem booting IRIX from the boot partition if it was set up correctly. When it was not set up correctly, I could tell with the dvhtool utility, not wonder why some part of the chain didn't get loaded. Now looking at it in that light, I would rather just fetch the target, the WHOLE thing, right out of a boot partition, as simple as it gets. Potatoehead, I'm with you. There are drawbacks, but a 3+ stage boot for me is too risky.
Those first two blocks would be the first two blocks of the boot loader. Ok, so it's a split loader but that's the price of not making it contiguous. It's still only two parts.
That is only one option ...
The main idea is to get a mostly non-changing boot loader that can read files after it's loaded. After that, the main program can be loaded. Same thing with the contiguous boot loader file except that really is a single file.
If you want a really large boot image that still reads files I'd go with reserved space instead. Reserved space works for small loader too. If you don't care about files at all then a custom partition type would work best.
EDIT: PS: The ROM doesn't even try to work out partitions. The pointers are raw device block numbers. Only the fully loaded filesystem handler needs to work out the partition stuff.
I too was originally viewing the boot loader as a single bulk load using the ROM. But it's not that big a deal to drop back to a 1kB leader.
It is a requirement of the Flash loader and security to preload a small loader with encryption keys etc. I am just conforming to those requirements.
They can be anywhere you decide to put them...
* As the data in a file within a filesystem
* As data within a partition
* As raw data anywhere on the SD card
So, the location is not defined by me - it's your choice. My preference is a file, and that is what my test code does. 1 Yes
2 Yes
3 Yes
4 Probably not.
But for now, I actually load the full Prop1 32KB starting from the 2 blocks. These 2 blocks will contain the pass key for the security and enough code to tell the P2 what to do, and that is likely to say load a few more KB following these 2 sectors, and this will contain the filesystem code and a filename to boot the code from. The beauty of my proposal, is that it does not have to be finalised before Chip commits to ROM code. And if we decide later on some other method, it will still all work. As I have said above, there are other security requirements in play here. I have no control of this, and I have not even had the time to look since Chip posted the code a few days ago.
If Chip decides that more than 1K (2 sectors) needs to be loaded then this can be done. There will be a number of sectors in the cluster that will be contiguous.
I maybe said it in wrong way.
Every system on PC need in boot stage use BIOS raw read mode to load bootstrap before it can RUN its own more advanced Loader/Booter..
That You never come from.
I don't believe I've been attacking you Sapieha. So, I'm not quite sure why your are calling me a know nothing. I've spent 3 years of my life understanding all the ins and outs of SD cards and the FAT file system. I know you are older than me and have way more knowledge about many things. But, I'm not ignorant of what I say on this thread at least.
The only reason I've been posting to this thread is just to appeal to people that whatever is done for the P2 needs to be fast, flexible, and works with caveats. I'm tired of having to defend the propeller as my choice uP. ARM processors do this stuff without caveats at the fastest speed possible. The propeller 2 needs to do the same.
So many people call me out for using the P1... it doesn't have C... now, I hear, it can't run C well. Or, I hear, the architecture sucks, or any other thing they don't like about it because its not an ARM or AVR (interestingly no-one I've listened to defends PICs). The P2 is a chance to get away from these problems. This is why I am posting here.
---
Anyway, I don't have anything more to contribute to this discussion. I'm out.
Thanks,
Sorry if I was to STRANGE to You.
BUT in some cases You write in terms that Yours answers NOT include others peoples questions only answers that You think are important to You..
(Many times Yours answers like --- MY are better -- without explaining many time why) In this Place at least I miss that You Say --- That Clusos else Ariba's not mention other people that posted theirs code spinets how them look on that SD-Dirver.
That make many people annoyed -- Inclusive me.
That is OK -- BUT what happen if I will don't have FAT system ---> Lets SAY I will build LINUX else other OS that have its own FILE System.
And that function need be usable to all customers.
To that - That driver need even can read raw file from any position of SD ----> Usually at starting from Sector 1 (0 is always reserved)
As Propeller II will have power to build Advanced systems with its own File systems and so on.
Only thing that are important in that driver are --- That it can read RAW file format (if to that it even can read information by FAT positioning is one + to it)
CAVEATS --- for many will be if them can't boot-(LOAD) THEIRS own FILE system handler to handle THEIRS own FILE system OS
Now even I'm are out from more discussion on subject --- Only if Chip ask any question.
---
Something to think about... whatever code that will be used for the SD card driver needs to be able to get the card out of multi block mode. No one's mentioned this but the SD card booting scheme will fail because of multi block mode if the boot loader does not support it.
This is because people will likely use the SD card in multi block mode during run time. But, if the boot loader code cannot get the card out of multi block mode your system will need to be power cycled to be reset.
This is a serious issue... I alerted Chip to this in my conversation with him. It's a hidden danger.
Thanks,
Yeah. That is really what it all should be about. Good on you for that Kye.
look for Boot-sig in MBR and boot that. Else search thru sectors (slow) find sig and boot that.
loaded prog then has FAT (or whatever) support and write Boot-sig in MBR to boot faster next time.
My favorite is yet still a HUBMONITOR over serial.
You even could load a file with it!
copy propeller.bin com11:
Enjoy!
Mike
I just started reading a piece of the spec and was noting how it kept saying certain modes are one way only. It's looking a bit like, officially, such reversals aren't supported ... Which leads to conclusion that software needs control of SD card power pin.
Or one of the caveats is: Only one mode supported on boot device.
Where and When did you work for Motorola? I was also a sys admin at Mot!!