Page? As in 2kB? It's predicted to be under 150 longs so far.
PS: The pointers placed in code portion of first block is what Cluso advocated. Back then I was thinking of only supporting FAT filesystem. In hindsight now, my idea was silly because it required having knowledge in the ROM for all FAT partition types. Not to mention not everyone wants a FAT partition.
If SD gets in there at all, it's going to be above $7ff, from what I read so far. Maybe the latest ROM rewrite changes things, not sure. The booter as I understand it right now, includes the flash support, serial and monitor, chip init, authentication, etc... ending at $7ff.
The last thing we saw on the P2 thread was Chip saying there was a month to sort out SD card, and that a last minute ROM change would make sense, if the SD can boot from the existing pin definitions used to talk to the SPI flash, and if it is robust and I think that's the mode issue on warm reset with power applied, where prop resets but SD card does not.
Adding SD would just push the top of ROM up a bit above that, which also implies the SD code can be the size it needs to be, with the real issues being a robust boot, and the capability being worth the RAM, not so much the absolute size of the ROM boot code, which is bit changable at this point. IMHO, that means it's a straight overlay to the otherwise complete layout, no recomputing necessary, which is why he entertained it.
But that's just me. Maybe something else was said and I didn't pick up on it. And, if it's robust, maybe it's worth the page being partially ROM.
...and "page" could be 1Kb too, pages are just pages, and how they map out is up to the developer in many cases, so it's not a whole 2K, just the next power of two up from whatever the potential SD boot code size is.
This all is why I asked about having two locations so that compliant boots would be possible. I wouldn't have otherwise for code size reasons. As it stands now, it's just a cost, not some absolute fit requirement.
Finally, he asked for a flow chart of the boot process, which means coming to some consensus on what that process is.
Are any of the suggested SD boot systems going to work when I build my CP/M system with a raw CP/M hard disk image planted at the start of an otherwise blank SD card with no FAT or modern MBRs?
I mean I might want to xmodem a new Prop image to SD through CP/M:)
The pointers placed in code portion of first block is what Cluso advocated. Back then I was thinking of only supporting FAT filesystem. In hindsight now, my idea was silly because it required having knowledge in the ROM for all FAT partition types. Not to mention not everyone wants a FAT partition.
Why are people still talking about partitioning?
Even go as far as ip talk?
Ok people, reguardless of WHAT DATA STRUCTURE is in play on said storage medium, the prop2 still needs boot with basics.
If I choose to wipe out the format of the sdcard completely and code my own structure, and booter,
OR
if I choose to insert a small program that only tells the prop2: how to access the sd card in high speed mode with fat structure support.
Both methods only need the p2 to look for a pointer with a unique id.
For an even simpler coding technique, the p2 could just start at byte 0 on sd card and do linear search for id.
This makes location of pointer irrelevant, and more a "compatibility" recommendation.
If someone wants to make a pointer in a partition, in a file, they could.
But if that partition and file is deep in the data structure, the p2 might take a bit to do a zip line linear pattern search.
So then make sure the data is written near the beginning of the sd card. (inside your desired partition, or data structure)
Data structure is irrellivant. The P2 can do boot "build ups" to accomplish ANY data structure recognition. (up to variable index limits)
Are any of the suggested SD boot systems going to work when I build my CP/M system with a raw CP/M hard disk image planted at the start of an otherwise blank SD card with no FAT or modern MBRs?
I mean I might want to xmodem a new Prop image to SD through CP/M:)
This is why I am advocating the prop start searching at sector 0 for ID.
No need to obey any sd card structure. The timing difference between a direct jump to a certain low data location, and just searching from 0 would be null.
This is why I am advocating the prop start searching at sector 0 for ID.
No need to obey any sd card structure. The timing difference between a direct jump to a certain low data location, and just searching from 0 would be null.
That was my very first proposal in the Propeller II thread. Later I combined it with the pointer in sector 0.
So if there is no pointer it just searches for a sector with a certain signature, which takes some seconds to find. The bootcode can then write a shortcut in sector 0 to itself, so it takes only some milliseconds to start up next time.
This will work with just a file written to the SD card, no Tools needed. But it can work also with a separate partition, or even with a CP/M filesystem.
This search would only happen if the boot code finds no code in Flash and no Shortcut in sector 0. Instead that the Prop shuts down, it will start a search for a while and shuts down later if nothing found.
We can perhaps always use the location of the second partition in the MBR table for the pointer, also if no second partition exists. If the partition type is a hidden partition I think this will not confuse Windows or Linux. So if you really want you can also format the SD card with 2 partitions in a standard way and it should work also that way.
The code with looking for a pointer in sector 0 and then do a search through the sectors is only 142 instructions in size.
RE: about the SD card not resetting when already in high speed mode.
So if we reset the prop2 using the reset line, but don't power cycle the pcb, the sd card stays in a "high speed mode" humming along, like nothing happened?
A SMALL ROM fix for this condition could be this:
Perhaps the P2 can just assume sd card is in high speed mode, already, every time its boot ROM runs.
Then the p2, could just send a command to *reset* or put the sd card into low speed mode.
Then it can do normal low speed mode duty of initialization & booting.
And if the card is already in low speed mode (or uninitialized), will it just ignore that initial high speed command?
This could be tested for on a prop1 actually. I would do it, but I am pretty new to sd card i/o. give me a few years and i'll be guru.
or
If you want to suck up more ROM...
Is it possible to run two initial tests in p2 ROM to test for this condition?
Do all sdcards have some kinda "ACK" reply that you can try to get first in low speed mode, then high speed mode, to just SEE if we are in reset no power drop high speed sd mode?
I suppose i could look into the datasheet more. I just figured others would know quickly.
The only time this would ever happen is if the prop2 was reset using the reset line.
(or it reset its self into ROM boot mode using internal code)
BUT the fact that the sd card stays in high speed mode is an advantage actually.
This means we can run ROM initial boot code in low speed mode, get sd into high speed mode, find full p2 boot code, re-program, and the new program resets the prop2, without dropping power, then the new program knows that sd card is already in high speed mode.
So its a good/bad scenario.
If some kinda test can be run in ROM during first sd card access, to determine if already in high speed mode, then just use it?
And/or just brute force a high speed command to the sd card to tell it to reset or go into low speed mode.
So if there is no pointer it just searches for a sector with a certain signature, which takes some seconds to find. The bootcode can then write a shortcut in sector 0 to itself, so it takes only some milliseconds to start up next time.
This will work with just a file written to the SD card, no Tools needed. But it can work also with a separate partition, or even with a CP/M filesystem.
This search would only happen if the boot code finds no code in Flash and no Shortcut in sector 0. Instead that the Prop shuts down, it will start a search for a while and shuts down later if nothing found.
The code with looking for a pointer in sector 0 and then do a search through the sectors is only 142 instructions in size.
Andy
The desire for the P1 to boot from SD was discussed in 2007.
After reading this thread, I am a bit suprised that people still couldn't see past the FAT/partition/data structure, in that its not needed at all.
Strictly from a data point of view, once a hook is landed on the sd card, it can load structure code and then sd card structure can be utilized to load even bigger code.
This is how all modern computers and tech work when they have large storage systems.
They all use small bootloader programs, which only tell the cpu how and where to find full program code. Initially the boot code run on all modern pc's don't know anything about the structure of the file system or partitions.
After reading this thread, I am a bit suprised that people still couldn't see past the FAT/partition/data structure, in that its not needed at all.
Strictly from a data point of view, once a hook is landed on the sd card, it can load structure code and then sd card structure can be utilized to load even bigger code.
This is how all modern computers and tech work when they have large storage systems.
They all use small bootloader programs, which only tell the cpu how and where to find full program code. Initially the boot code run on all modern pc's don't know anything about the structure of the file system or partitions.
After reading this thread, I am a bit suprised that people still couldn't see past the FAT/partition/data structure, in that its not needed at all.
Strictly from a data point of view, once a hook is landed on the sd card, it can load structure code and then sd card structure can be utilized to load even bigger code.
This is how all modern computers and tech work when they have large storage systems.
They all use small bootloader programs, which only tell the cpu how and where to find full program code. Initially the boot code run on all modern pc's don't know anything about the structure of the file system or partitions.
A FAT file strucrure is not needed to boot the Propeller, but the SD card has it anyway when you buy it. And SD card booting is most useful if you use the SD card later in the application. All the SD drivers for Prop1 work with the FAT file system, so it makes a lot of sense to support FAT formated SD cards from the bootloader.
A FAT file strucrure is not needed to boot the Propeller, but the SD card has it anyway when you buy it. And SD card booting is most useful if you use the SD card later in the application. All the SD drivers for Prop1 work with the FAT file system, so it makes a lot of sense to support FAT formated SD cards from the bootloader.
Andy
When you say "bootloader" do you mean P2 ROM booter?, or do you mean the bootloader, that is located on SD card?
You can still do ALL this without ROM support for FAT. Putting FAT support in rom introduces even more potential "gotchas", and stops you from improving the FAT code.
It would also REQUIRE a fat format. What about people that want to use the SD card as they please??
If they wanted to write their own booter, and required FAT structure, they would need to "code around" all the fat structure, instead of using linear programming techniques.
For instance, the index offset used in the booter program could be directly tied to the sd card addressing, if no fat structure was required. The structure of the P2 program could actually be smaller if the P2 program started booting at byte 0, all code could access data using direct I/O, with simple indexing. If you force a fat structure (or any)to boot prop, you are forcing coders to have to work around the fat file system.
This condition cannot be undone, but the other way around, booting into dataspace, and then coding for structure, allows coder choice, with no structure limitations.
It should not be this way, the sd card needs to be viewed as any other data storage device, which has the potential for any desired format the user chooses.
Don't get me wrong, FAT is great and all, it allows one to insert the sd card into pc, and drop files. And this would be best, and could still be done without ROM fat support.
But if you forced ROM fat32 support, NTFS, FAT12/16, RAW, and many other formats are out and done. (you wanna Smile a linux coder off? require FAT in ROM)
Ok, im done trying to go into detail on why any data structure limitations, coded into ROM is a super bad idea.
The more code that you put into ROM, and the more requirements you put into ROM, means you narrow the uses of this feature, and increase the likelyhood of un-repairable bugs.
That was my very first proposal in the Propeller II thread. Later I combined it with the pointer in sector 0.
So if there is no pointer it just searches for a sector with a certain signature, which takes some seconds to find. The bootcode can then write a shortcut in sector 0 to itself, so it takes only some milliseconds to start up next time.
This will work with just a file written to the SD card, no Tools needed. But it can work also with a separate partition, or even with a CP/M filesystem.
This search would only happen if the boot code finds no code in Flash and no Shortcut in sector 0. Instead that the Prop shuts down, it will start a search for a while and shuts down later if nothing found.
The code with looking for a pointer in sector 0 and then do a search through the sectors is only 142 instructions in size.
Andy
You have it all right here. This would be gold, if implemented.
The part about the prop inserting the pointer into sector 0 should be part of YOUR program, NOT the prop ROM. (incase the pointer location needs to be changed later due to "gotchas")
(so once the prop found the pointer, it would run the code, and in that code, you tell it to place a shortcut pointer into sector 0 at offset X)
Less in ROM means less mucky muck.
Also include the sd card 1-bit/4-bit fix, like I mention above, this would obviously need to be in ROM.
Perhaps the P2 can just assume sd card is in high speed mode, already, every time its boot ROM runs.
Then the p2, could just send a command to *reset* or put the sd card into low speed mode.
Then it can do normal low speed mode duty of initialization & booting.
And if the card is already in low speed mode (or uninitialized), will it just ignore that initial high speed command?
This below could be done, even if the above was just implemented. It would do a pointer scan in sector 0, and then find the pointer when it reached the pointer marker in first/second/hidden partition, the prop2 would then insert a pointer into sector 0 to that partition/file. But the only thing in ROM should be the pointer scan.
The part about the prop inserting the pointer into sector 0 should be part of your program, NOT the prop ROM. (incase the pointer location needs to be changed later)
We can perhaps always use the location of the second partition in the MBR table for the pointer, also if no second partition exists. If the partition type is a hidden partition I think this will not confuse Windows or Linux. So if you really want you can also format the SD card with 2 partitions in a standard way and it should work also that way.
Are any of the suggested SD boot systems going to work when I build my CP/M system with a raw CP/M hard disk image planted at the start of an otherwise blank SD card with no FAT or modern MBRs?
What sort of allotted space does this CP/M volume have for boot code in it's first block?
This is exactly the sort of thing I'd want to support. The ID and pointer(s) go together. There shouldn't be any need to search beyond the first block.
Partitionless booting does has it's limits as there is no space defined outside of the volume itself, it all depends on the target filesystem/volume layout.
The ID and pointer(s) go together. There shouldn't be any need to search beyond the first block.
I'm a strong advocate of working from the first block rather than just scanning for first occurrence of some coincidental ID.
As an example of this working in a complex partitioning setup, take a GPT partitioned drive. It's partitioning structures start at the second block, not the first. But the first block contains an ordinary MBR with a single partition entry, of GPT type, that covers the whole drive.
So, the partitioning is an unknown until further down the boot process (even for PCs). But there is still space allotted for boot code in the first block, so, an ID and pointer can be placed there and can still point to anywhere in the whole drive. For the Prop, it becomes up to the loaded filesystem code to handle the complexities of how the partition is managed, not the ROM code.
When you say "bootloader" do you mean P2 ROM booter?, or do you mean the bootloader, that is located on SD card?
You can still do ALL this without ROM support for FAT. Putting FAT support in rom introduces even more potential "gotchas", and stops you from improving the FAT code.
It would also REQUIRE a fat format. What about people that want to use the SD card as they please??
If they wanted to write their own booter, and required FAT structure, they would need to "code around" all the fat structure, instead of using linear programming techniques.
For instance, the index offset used in the booter program could be directly tied to the sd card addressing, if no fat structure was required. The structure of the P2 program could actually be smaller if the P2 program started booting at byte 0, all code could access data using direct I/O, with simple indexing. If you force a fat structure (or any)to boot prop, you are forcing coders to have to work around the fat file system.
This condition cannot be undone, but the other way around, booting into dataspace, and then coding for structure, allows coder choice, with no structure limitations.
It should not be this way, the sd card needs to be viewed as any other data storage device, which has the potential for any desired format the user chooses.
Don't get me wrong, FAT is great and all, it allows one to insert the sd card into pc, and drop files. And this would be best, and could still be done without ROM fat support.
But if you forced ROM fat32 support, NTFS, FAT12/16, RAW, and many other formats are out and done. (you wanna Smile a linux coder off? require FAT in ROM)
Ok, im done trying to go into detail on why any data structure limitations, coded into ROM is a super bad idea.
The more code that you put into ROM, and the more requirements you put into ROM, means you narrow the uses of this feature, and increase the likelyhood of un-repairable bugs.
This thread is all about doing the ROM code without using the FAT, but make it so that it is compatible to a FAT filesystem. So you can use the PC to write your bootfile.
And for sure, the ROM code writes nothing (also no pointer) to the SD card, it just searches the bootfile and starts it. This started code can then write a pointer if desired.
What sort of allotted space does this CP/M volume have for boot code in it's first block?
This is exactly the sort of thing I'd want to support. The ID and pointer(s) go together. There shouldn't be any need to search beyond the first block.
Partitionless booting does has it's limits as there is no space defined outside of the volume itself, it all depends on the target filesystem/volume layout.
I'm a strong advocate of working from the first block rather than just scanning for first occurrence of some coincidental ID.
As an example of this working in a complex partitioning setup, take a GPT partitioned drive. It's partitioning structures start at the second block, not the first. But the first block contains an ordinary MBR with a single partition entry, of GPT type, that covers the whole drive.
So, the partitioning is an unknown until further down the boot process (even for PCs). But there is still space allotted for boot code in the first block, so, an ID and pointer can be placed there and can still point to anywhere in the whole drive. For the Prop, it becomes up to the loaded filesystem code to handle the complexities of how the partition is managed, not the ROM code.
This thread is all about doing the ROM code without using the FAT, but make it so that it is compatible to a FAT filesystem. So you can use the PC to write your bootfile.
And for sure, the ROM code writes nothing (also no pointer) to the SD card, it just searches the bootfile and starts it. This started code can then write a pointer if desired.
Andy
These all touch on the same point, having the P2 start at sector/byte 0 and search indefinitely to end of card size for ID/pointer, allows one to have any partition structure, and throw the program file with ID/pointer inside it, onto that sd card anywhere. Creating a partition close to beginning of card, and making it invisible, could be easily done.. IF you store it too deep in the sd card it will take a long time to search, but when it gets it, it never has to search again.
So the P2 can find its ID and pointer and program on any partition/format just so long as the P2 does complete sd card search at last attempt to boot sd. It will scan past all data structure, till it comes upon that one file that you dumped onto the sd card which had the MAGIC identifier and code. The pointer could be inserted after the file was written to the sd card, using a special tool, or even winhex.
If a search from 0 to max sd card size for pointer and ID isn't done, the P2 would REQUIRE a FORMAT/partition recognition system to be loaded BEFORE finding the program that you dropped onto the sd card.
If the prop just runs a full sd scan, it will eventually come upon the ID and pointer, boot the code, write its own "mbr" and never take that long to boot again. This is if the pointer, and full program file can be written sequentially to the sd card. Even then tools can write files sequentially if that is a problem.
Yes this is exactly what I have in mind. I will post a new test code soon.
I search now only in the first 20000 sectors for the MagicID, this takes about 40 seconds if I test every sector. I think we can recommend to write the bootfile as first file to an SD card, then it is at low sector numbers and is always contiguous.
I also found that the first sector of a file is always at a sector number which is multiple of 8 or one below (with min 4k clusters). So I have a faster search which tests only 2 out of 8 sectors, this makes it 4 times faster to find a file.
If there's going to be SD boot one way or another (even if it's just via flash), _please_ make its bootloader multi-stage! As has been said by others, that's how nearly all sane boot loader schemes work. You just don't put your filesystem handling (FAT or whatever) in the first stage bootloader. You just don't. Leave that part in the second stage bootloader which you load from the card, if you include filesystem handling at all (on Linux GRUB does, LILO doesn't, in their second stage boot loaders)
We can't know that we can even buy cards pre-formatted with FAT32 in a few years time. For all we know it may be that in three years from now all and every card will be exFAT formatted. Or something else.
Last year I had to find a source in China for old SD cards, because the board I worked with couldn't handle any of the cards I could buy in the local shop, which were all 2GB and upwards. Which surprised me because it felt almost like yesterday when a 512MB card was the largest and most expensive you could buy..
Let the bootloader be at least two-stage. Leave any filesystem handling out of the first stage, in-ROM or in-flash bootloader. Be sane!
These all touch on the same point, having the P2 start at sector/byte 0 and search indefinitely to end of card size for ID/pointer, allows one to have any partition structure, and throw the program file with ID/pointer inside it, onto that sd card anywhere. Creating a partition close to beginning of card, and making it invisible, could be easily done.. IF you store it too deep in the sd card it will take a long time to search, but when it gets it, it never has to search again.
So the P2 can find its ID and pointer and program on any partition/format just so long as the P2 does complete sd card search at last attempt to boot sd. It will scan past all data structure, till it comes upon that one file that you dumped onto the sd card which had the MAGIC identifier and code. The pointer could be inserted after the file was written to the sd card, using a special tool, or even winhex.
If a search from 0 to max sd card size for pointer and ID isn't done, the P2 would REQUIRE a FORMAT/partition recognition system to be loaded BEFORE finding the program that you dropped onto the sd card.
If the prop just runs a full sd scan, it will eventually come upon the ID and pointer, boot the code, write its own "mbr" and never take that long to boot again. This is if the pointer, and full program file can be written sequentially to the sd card. Even then tools can write files sequentially if that is a problem.
When you delete a file only it's FAT entry is marked as deleted while all the data on the disk is still there (this is the reason why all the undelete tools works). If, when you copy the new firmware, it will be allocated after the deleted one, your scan routine will still find the first.
Most of us agree on the pointer solution. For me the best place is to use an entry of the partition table (better the first so you have nothing to scan) but at the end, even if its in sector0/byte0, the only thing that matter is that the chosen solution haven't any relationship with any file system, is MBR based complying with its standards so that some other systems tool do not broke it (eg windows ask to sign the disks under certain conditions and this happens @440 offset), is the simplest, strong and safe, with other words: reliable.
Clock Loop,
Your scheme will fail. When you overwrite your Propeller file the existing one is marked deleted but not actually overwritten. The new data will be written at a possibly a different location. Then your Prop will perhaps find and boot the old copy that still exists somewhere.
A multi-stage boot is the sensible way to go, the code in the Props ROM should know nothing of file systems.
I'm still inclined to say "Boot out SD card booting" if it takes even one byte away from our precious RAM or one extra precious pin. And as I said before it effectively dictates what pins SD cards will be put on which rather goes against the grain.
Regarding the SD mode issue it could be resolved by powering the SD through a transistor driven by the reset line.
BTW it will be nice if the new P2 will drive low its reset pin in case of software reset thus allowing for the reset of other IC/Coprocessors that can be on the PCB
Many uP supervisors beside keepeng the reset low due to voltage/watchdog events recognize also a switch on the same line and after the switch pressure holds down the reset a little bit longer thus preventing also switch bounces.
Clock Loop,
Your scheme will fail. When you overwrite your Propeller file the existing one is marked deleted but not actually overwritten. The new data will be written at a possibly a different location. Then your Prop will perhaps find and boot the old copy that still exists somewhere.
A multi-stage boot is the sensible way to go, the code in the Props ROM should know nothing of file systems.
I'm still inclined to say "Boot out SD card booting" if it takes even one byte away from our precious RAM or one extra precious pin. And as I said before it effectively dictates what pins SD cards will be put on which rather goes against the grain.
Clock Loop,
Your scheme will fail. When you overwrite your Propeller file the existing one is marked deleted but not actually overwritten. The new data will be written at a possibly a different location. Then your Prop will perhaps find and boot the old copy that still exists somewhere.
A multi-stage boot is the sensible way to go, the code in the Props ROM should know nothing of file systems.
I thought if you overwrote a file it used same dataspace?
Even so, any style we use to boot the prop and do it from inside the structure, in a file, the prop still needs to know where that file is to jump to, from the pointer, where ever it is. The pointer still somehow needs to get there. So any way you look at it, the prop2 will need a utility to boot from sd, to put the proper data in the proper location.
If duplicate deleted (but not) data on SD is a problem.....well
Any way its done, a special program will be needed to put special data on the sd card with special calculated pointers to the full sd file.
Last i checked, its not hard to make a utility to scan the SD card for any old ID/POINTERS, and wipe them out.
I have used those undelete utilities and some of them had a way to permanently erase the deleted "undeleted" data.
Then a simple tool that scans for old ID/pointer is all that one would need.
Even if you just drop the binary file onto a fat formatted small partition at the beginning of the sd card, the binary file still needs its ID inserted and pointer calculated, this would require a utility.
If it was to be all done from windows.
Those things can be done with the P2, after someone gets a framework programmed and going for that purpose.
Any frame work chosen will probably work, so long as no other ID's were still on the dataspace, if near the beginning of the sd card, the file would be found, the P2 could insert its ID/and jump into MBR.
If that sd card then gets used in a pc to update its binary file, it will need a SD utility that does id scan and removal, before the new file gets put onto the card.
And also, somehow that file needs an ID and pointer to itself, which isn't that hard to do, but again a special utility would need to be made.
Or the P2 would need to have flags that it would know to use the dataspace directly proceeding the ID/pointer. I don't see how that could be done, but if we could tell the prop to use the dataspace directly after the ID/pointer, then we wouldn't need to calculate and insert the pointer.
I don't mind if the prop looks for this in specific locations, and THEN does a linear search from MBR/VBR byte 0 for id/pointer.
This would allow a specific land, or allow someone to use the sd card as they wanted, so long as they insert a id/pointer near the beginning of the sd card.
Comments
PS: The pointers placed in code portion of first block is what Cluso advocated. Back then I was thinking of only supporting FAT filesystem. In hindsight now, my idea was silly because it required having knowledge in the ROM for all FAT partition types. Not to mention not everyone wants a FAT partition.
The last thing we saw on the P2 thread was Chip saying there was a month to sort out SD card, and that a last minute ROM change would make sense, if the SD can boot from the existing pin definitions used to talk to the SPI flash, and if it is robust and I think that's the mode issue on warm reset with power applied, where prop resets but SD card does not.
Adding SD would just push the top of ROM up a bit above that, which also implies the SD code can be the size it needs to be, with the real issues being a robust boot, and the capability being worth the RAM, not so much the absolute size of the ROM boot code, which is bit changable at this point. IMHO, that means it's a straight overlay to the otherwise complete layout, no recomputing necessary, which is why he entertained it.
But that's just me. Maybe something else was said and I didn't pick up on it. And, if it's robust, maybe it's worth the page being partially ROM.
...and "page" could be 1Kb too, pages are just pages, and how they map out is up to the developer in many cases, so it's not a whole 2K, just the next power of two up from whatever the potential SD boot code size is.
This all is why I asked about having two locations so that compliant boots would be possible. I wouldn't have otherwise for code size reasons. As it stands now, it's just a cost, not some absolute fit requirement.
Finally, he asked for a flow chart of the boot process, which means coming to some consensus on what that process is.
Are any of the suggested SD boot systems going to work when I build my CP/M system with a raw CP/M hard disk image planted at the start of an otherwise blank SD card with no FAT or modern MBRs?
I mean I might want to xmodem a new Prop image to SD through CP/M:)
Why are people still talking about partitioning?
Even go as far as ip talk?
Ok people, reguardless of WHAT DATA STRUCTURE is in play on said storage medium, the prop2 still needs boot with basics.
If I choose to wipe out the format of the sdcard completely and code my own structure, and booter,
OR
if I choose to insert a small program that only tells the prop2: how to access the sd card in high speed mode with fat structure support.
Both methods only need the p2 to look for a pointer with a unique id.
For an even simpler coding technique, the p2 could just start at byte 0 on sd card and do linear search for id.
This makes location of pointer irrelevant, and more a "compatibility" recommendation.
If someone wants to make a pointer in a partition, in a file, they could.
But if that partition and file is deep in the data structure, the p2 might take a bit to do a zip line linear pattern search.
So then make sure the data is written near the beginning of the sd card. (inside your desired partition, or data structure)
Data structure is irrellivant. The P2 can do boot "build ups" to accomplish ANY data structure recognition. (up to variable index limits)
This is why I am advocating the prop start searching at sector 0 for ID.
No need to obey any sd card structure. The timing difference between a direct jump to a certain low data location, and just searching from 0 would be null.
I was kind of joking when I talk about a CP/M disk image on SD but it does amplify what you are saying.
I am a fan of the demo scene, less restriction on data structure works best.
The things these guys do with coding techniques... its amazing.
http://en.wikipedia.org/wiki/Demoscene
Plus, cmon, lets make this guys job, just a bit easier.
That was my very first proposal in the Propeller II thread. Later I combined it with the pointer in sector 0.
So if there is no pointer it just searches for a sector with a certain signature, which takes some seconds to find. The bootcode can then write a shortcut in sector 0 to itself, so it takes only some milliseconds to start up next time.
This will work with just a file written to the SD card, no Tools needed. But it can work also with a separate partition, or even with a CP/M filesystem.
This search would only happen if the boot code finds no code in Flash and no Shortcut in sector 0. Instead that the Prop shuts down, it will start a search for a while and shuts down later if nothing found.
We can perhaps always use the location of the second partition in the MBR table for the pointer, also if no second partition exists. If the partition type is a hidden partition I think this will not confuse Windows or Linux. So if you really want you can also format the SD card with 2 partitions in a standard way and it should work also that way.
The code with looking for a pointer in sector 0 and then do a search through the sectors is only 142 instructions in size.
Andy
So if we reset the prop2 using the reset line, but don't power cycle the pcb, the sd card stays in a "high speed mode" humming along, like nothing happened?
A SMALL ROM fix for this condition could be this:
Perhaps the P2 can just assume sd card is in high speed mode, already, every time its boot ROM runs.
Then the p2, could just send a command to *reset* or put the sd card into low speed mode.
Then it can do normal low speed mode duty of initialization & booting.
And if the card is already in low speed mode (or uninitialized), will it just ignore that initial high speed command?
This could be tested for on a prop1 actually. I would do it, but I am pretty new to sd card i/o. give me a few years and i'll be guru.
or
If you want to suck up more ROM...
Is it possible to run two initial tests in p2 ROM to test for this condition?
Do all sdcards have some kinda "ACK" reply that you can try to get first in low speed mode, then high speed mode, to just SEE if we are in reset no power drop high speed sd mode?
I suppose i could look into the datasheet more. I just figured others would know quickly.
The only time this would ever happen is if the prop2 was reset using the reset line.
(or it reset its self into ROM boot mode using internal code)
BUT the fact that the sd card stays in high speed mode is an advantage actually.
This means we can run ROM initial boot code in low speed mode, get sd into high speed mode, find full p2 boot code, re-program, and the new program resets the prop2, without dropping power, then the new program knows that sd card is already in high speed mode.
So its a good/bad scenario.
If some kinda test can be run in ROM during first sd card access, to determine if already in high speed mode, then just use it?
And/or just brute force a high speed command to the sd card to tell it to reset or go into low speed mode.
The desire for the P1 to boot from SD was discussed in 2007.
I hope the P2 doesn't miss this again.
http://forums.parallax.com/showthread.php?97895-boot-from-an-%B5SD-card-%28desired-hardware%29
After reading this thread, I am a bit suprised that people still couldn't see past the FAT/partition/data structure, in that its not needed at all.
Strictly from a data point of view, once a hook is landed on the sd card, it can load structure code and then sd card structure can be utilized to load even bigger code.
This is how all modern computers and tech work when they have large storage systems.
They all use small bootloader programs, which only tell the cpu how and where to find full program code.
Initially the boot code run on all modern pc's don't know anything about the structure of the file system or partitions.
It is what I talk on BootStrap in MBA -- But none listen.
A FAT file strucrure is not needed to boot the Propeller, but the SD card has it anyway when you buy it. And SD card booting is most useful if you use the SD card later in the application. All the SD drivers for Prop1 work with the FAT file system, so it makes a lot of sense to support FAT formated SD cards from the bootloader.
Andy
When you say "bootloader" do you mean P2 ROM booter?, or do you mean the bootloader, that is located on SD card?
You can still do ALL this without ROM support for FAT. Putting FAT support in rom introduces even more potential "gotchas", and stops you from improving the FAT code.
It would also REQUIRE a fat format. What about people that want to use the SD card as they please??
If they wanted to write their own booter, and required FAT structure, they would need to "code around" all the fat structure, instead of using linear programming techniques.
For instance, the index offset used in the booter program could be directly tied to the sd card addressing, if no fat structure was required. The structure of the P2 program could actually be smaller if the P2 program started booting at byte 0, all code could access data using direct I/O, with simple indexing. If you force a fat structure (or any)to boot prop, you are forcing coders to have to work around the fat file system.
This condition cannot be undone, but the other way around, booting into dataspace, and then coding for structure, allows coder choice, with no structure limitations.
It should not be this way, the sd card needs to be viewed as any other data storage device, which has the potential for any desired format the user chooses.
Don't get me wrong, FAT is great and all, it allows one to insert the sd card into pc, and drop files. And this would be best, and could still be done without ROM fat support.
But if you forced ROM fat32 support, NTFS, FAT12/16, RAW, and many other formats are out and done. (you wanna Smile a linux coder off? require FAT in ROM)
Ok, im done trying to go into detail on why any data structure limitations, coded into ROM is a super bad idea.
The more code that you put into ROM, and the more requirements you put into ROM, means you narrow the uses of this feature, and increase the likelyhood of un-repairable bugs.
You have it all right here. This would be gold, if implemented.
The part about the prop inserting the pointer into sector 0 should be part of YOUR program, NOT the prop ROM. (incase the pointer location needs to be changed later due to "gotchas")
(so once the prop found the pointer, it would run the code, and in that code, you tell it to place a shortcut pointer into sector 0 at offset X)
Less in ROM means less mucky muck.
Also include the sd card 1-bit/4-bit fix, like I mention above, this would obviously need to be in ROM.
This below could be done, even if the above was just implemented. It would do a pointer scan in sector 0, and then find the pointer when it reached the pointer marker in first/second/hidden partition, the prop2 would then insert a pointer into sector 0 to that partition/file. But the only thing in ROM should be the pointer scan.
The part about the prop inserting the pointer into sector 0 should be part of your program, NOT the prop ROM. (incase the pointer location needs to be changed later)
What sort of allotted space does this CP/M volume have for boot code in it's first block?
This is exactly the sort of thing I'd want to support. The ID and pointer(s) go together. There shouldn't be any need to search beyond the first block.
Partitionless booting does has it's limits as there is no space defined outside of the volume itself, it all depends on the target filesystem/volume layout.
I'm a strong advocate of working from the first block rather than just scanning for first occurrence of some coincidental ID.
As an example of this working in a complex partitioning setup, take a GPT partitioned drive. It's partitioning structures start at the second block, not the first. But the first block contains an ordinary MBR with a single partition entry, of GPT type, that covers the whole drive.
So, the partitioning is an unknown until further down the boot process (even for PCs). But there is still space allotted for boot code in the first block, so, an ID and pointer can be placed there and can still point to anywhere in the whole drive. For the Prop, it becomes up to the loaded filesystem code to handle the complexities of how the partition is managed, not the ROM code.
This thread is all about doing the ROM code without using the FAT, but make it so that it is compatible to a FAT filesystem. So you can use the PC to write your bootfile.
And for sure, the ROM code writes nothing (also no pointer) to the SD card, it just searches the bootfile and starts it. This started code can then write a pointer if desired.
Andy
These all touch on the same point, having the P2 start at sector/byte 0 and search indefinitely to end of card size for ID/pointer, allows one to have any partition structure, and throw the program file with ID/pointer inside it, onto that sd card anywhere. Creating a partition close to beginning of card, and making it invisible, could be easily done.. IF you store it too deep in the sd card it will take a long time to search, but when it gets it, it never has to search again.
So the P2 can find its ID and pointer and program on any partition/format just so long as the P2 does complete sd card search at last attempt to boot sd. It will scan past all data structure, till it comes upon that one file that you dumped onto the sd card which had the MAGIC identifier and code. The pointer could be inserted after the file was written to the sd card, using a special tool, or even winhex.
If a search from 0 to max sd card size for pointer and ID isn't done, the P2 would REQUIRE a FORMAT/partition recognition system to be loaded BEFORE finding the program that you dropped onto the sd card.
If the prop just runs a full sd scan, it will eventually come upon the ID and pointer, boot the code, write its own "mbr" and never take that long to boot again. This is if the pointer, and full program file can be written sequentially to the sd card. Even then tools can write files sequentially if that is a problem.
Yes this is exactly what I have in mind. I will post a new test code soon.
I search now only in the first 20000 sectors for the MagicID, this takes about 40 seconds if I test every sector. I think we can recommend to write the bootfile as first file to an SD card, then it is at low sector numbers and is always contiguous.
I also found that the first sector of a file is always at a sector number which is multiple of 8 or one below (with min 4k clusters). So I have a faster search which tests only 2 out of 8 sectors, this makes it 4 times faster to find a file.
Andy
We can't know that we can even buy cards pre-formatted with FAT32 in a few years time. For all we know it may be that in three years from now all and every card will be exFAT formatted. Or something else.
Last year I had to find a source in China for old SD cards, because the board I worked with couldn't handle any of the cards I could buy in the local shop, which were all 2GB and upwards. Which surprised me because it felt almost like yesterday when a 512MB card was the largest and most expensive you could buy..
Let the bootloader be at least two-stage. Leave any filesystem handling out of the first stage, in-ROM or in-flash bootloader. Be sane!
-Tor
When you delete a file only it's FAT entry is marked as deleted while all the data on the disk is still there (this is the reason why all the undelete tools works). If, when you copy the new firmware, it will be allocated after the deleted one, your scan routine will still find the first.
Most of us agree on the pointer solution. For me the best place is to use an entry of the partition table (better the first so you have nothing to scan) but at the end, even if its in sector0/byte0, the only thing that matter is that the chosen solution haven't any relationship with any file system, is MBR based complying with its standards so that some other systems tool do not broke it (eg windows ask to sign the disks under certain conditions and this happens @440 offset), is the simplest, strong and safe, with other words: reliable.
Your scheme will fail. When you overwrite your Propeller file the existing one is marked deleted but not actually overwritten. The new data will be written at a possibly a different location. Then your Prop will perhaps find and boot the old copy that still exists somewhere.
A multi-stage boot is the sensible way to go, the code in the Props ROM should know nothing of file systems.
I'm still inclined to say "Boot out SD card booting" if it takes even one byte away from our precious RAM or one extra precious pin. And as I said before it effectively dictates what pins SD cards will be put on which rather goes against the grain.
BTW it will be nice if the new P2 will drive low its reset pin in case of software reset thus allowing for the reset of other IC/Coprocessors that can be on the PCB
Many uP supervisors beside keepeng the reset low due to voltage/watchdog events recognize also a switch on the same line and after the switch pressure holds down the reset a little bit longer thus preventing also switch bounces.
It is why I proposed to have PIN's configurable in FUSES.
That's "Heater".
Pins configurable with fuses perhaps. But then I hate fuses as well. See my pile of "fused" AVRs:)
Sorry for extra -- h ---.
Even if I hate them -- It is only way to have it flexible.
I thought if you overwrote a file it used same dataspace?
Even so, any style we use to boot the prop and do it from inside the structure, in a file, the prop still needs to know where that file is to jump to, from the pointer, where ever it is. The pointer still somehow needs to get there. So any way you look at it, the prop2 will need a utility to boot from sd, to put the proper data in the proper location.
If duplicate deleted (but not) data on SD is a problem.....well
Any way its done, a special program will be needed to put special data on the sd card with special calculated pointers to the full sd file.
Last i checked, its not hard to make a utility to scan the SD card for any old ID/POINTERS, and wipe them out.
I have used those undelete utilities and some of them had a way to permanently erase the deleted "undeleted" data.
Then a simple tool that scans for old ID/pointer is all that one would need.
Even if you just drop the binary file onto a fat formatted small partition at the beginning of the sd card, the binary file still needs its ID inserted and pointer calculated, this would require a utility.
If it was to be all done from windows.
Those things can be done with the P2, after someone gets a framework programmed and going for that purpose.
Any frame work chosen will probably work, so long as no other ID's were still on the dataspace, if near the beginning of the sd card, the file would be found, the P2 could insert its ID/and jump into MBR.
If that sd card then gets used in a pc to update its binary file, it will need a SD utility that does id scan and removal, before the new file gets put onto the card.
And also, somehow that file needs an ID and pointer to itself, which isn't that hard to do, but again a special utility would need to be made.
Or the P2 would need to have flags that it would know to use the dataspace directly proceeding the ID/pointer. I don't see how that could be done, but if we could tell the prop to use the dataspace directly after the ID/pointer, then we wouldn't need to calculate and insert the pointer.
I don't mind if the prop looks for this in specific locations, and THEN does a linear search from MBR/VBR byte 0 for id/pointer.
This would allow a specific land, or allow someone to use the sd card as they wanted, so long as they insert a id/pointer near the beginning of the sd card.
On Windows...
Well, here's somebody who implemented a dd work alike:
http://www.chrysocome.net/dd
And FDISK on windows can very easily make the SD partition.
I think a bit of looking would reveal the tools are there too, just need to be found and documented.