Serial Bootloader
FredBlais
Posts: 370
For my next project, I would like to create a RS-485 networked prop module. Multiple of these modules could connect on a master Raspberry Pi. This would be for home automation.
I think it would be cool to have a kind of network command to reprogram a propeller node from a binary file. Has anyone already worked on this kind of stuff? All I find on the Obex is loader from SD card.
Fred
I think it would be cool to have a kind of network command to reprogram a propeller node from a binary file. Has anyone already worked on this kind of stuff? All I find on the Obex is loader from SD card.
Fred
Comments
The Propeller has what I would call a "Spin Loader" that requires a Reset as well as Tx and Rx. I worked on a RS422 full duplex with Rst interface last year and it works fine over distance of 1000 meters.
A 'serial bootloader' is something that an Arduino has. Instead of using the programming device to load software, it migrates the loading to an RS232 port. But the Propeller has no need to migrate, it is all there -- always was.
The real question is how are you going to manage multiple serial interfaces? The Propeller does NOT accept an address to separate out an individual Propeller from others on a shared serial line. All Propeller listen and load on a one-on-one basis with the computer providing the binary.
It would seem having the computer provide multiple RS232 or RS422 plus Reset ports would be the easiest way to manage this scheme. (And the Raspberry Pi cannot provide multiple RS232 ports at one time.. just one USART).
I suppose that you could withhold the Reset from all but one Propeller and share the Rx and Tx for loading, but then Pins 30 and 31 serial would likely be unavailable for other uses. This would be a bit of complex and messy wiring.
It you be easier to just have a network of Forth on Propeller devices. The interpreter would allow you to program any serial port that you can reach.
But even in Forth, I do not have clear idea how multiple Propellers are going to share one seral line. Each has to be given its own identity and just listen until it is properly addressed. Otherwise all will try to respond at the same time and the network crashes.
One could imagine that:
1) From reset the Prop loads a program from it's EEPROM as normal.
2) That program does nothing but maintain whatever protocol is required for the serial network and wait for a reprogramming command.
3) On getting the reprogramming command it receives a new program binary and stores it into some non-volatile storage. Perhaps the upper half of a 64KB EEPROM.
4) It then replaces itself in RAM with the downloaded program (read from the EEPROM) and runs it.
5) The new downloaded program also maintains the serial line protocol and does whatever it's function in life is to be.
6) However it also listens for reprogramming commands and can do step 3) upwards. (Or perhaps it just causes a reset and we start from step 1)
Details are left to the reader. At least we know it can be done because the loader in propgcc does something similar.
Edit: Addressing individual Props in a network can be handled by having some unique ID built into the hardware design that is used as an address.
What I want to do is be able to put a new program in the EEPROM and then issue a reset. The propeller would wait for certain commands on the serial link and one of them would be to replace the existing program.
In the propeller tool when I compile, I see that I have 2 options (save to a EEPROM or binary file), I suppose that I could use that to replace what is in the EEPROM?
Heater, I'm unsure about your step #3, why put the program in the upper half of the EEPROM?
p.s. the ID for RS-485 could easily be done with DIP switches.
You may want to read this Application Note from the Parallax Semiconductor site:
[h=1]AN007: Soft-loading an Application Image into the Propeller P8X32A via XBee Transceivers [/h]It's obviously about XBees for the communication part but the loader process and concepts should be similar, just swap out the XBee transport layer for RS-485.
The dilemma with a wired network solution is that all the Propellers have a 'bootloader' that offers no separate identity, that this code is built-in to the Propellers ROM (and unchangable, and the Propeller does the actual writting to a local external EEPROM.
I suspect you could do something similar to the XBee if you could have another microcontroller with a unique identity on a serial network take over the writting of a .spin binary to the EEPROM.
a. There are PICs that have a UART and an SPI or I2C interface that could do the job. But some additonal logic chips would be required to switch the EEPROM availablity between these two separate devices.
Alternatively, the PIC might just act as a gatekeeper and not mess with the Propeller's EEPROM writing.
b. Better yet, have the PIC do an identity check, and only pass communications that are appropritate for the unique identity until further notice. Then let the Propeller just manage writing to EEPROM as it normally does.
Of course this means that the Raspberry Pi must do some preliminary work before transmission of the binary. I suspect that could all be done in a script of some sort.
Either way, at the remote Propeller you need some sort of external chip to manage the identity before any serial communications take place.
+++++++++++
BTW, Heater's suggestion to put the program in the 'upper half' of a 64K EEPROM is likely doomed to failure. Anything above 32K is never looked at by the Propeller for loading a .spin binary. And it can't be modified to do so.
The soluyion to this will not use the Propeller serial loader. All the loader mechanisms must be part of the embedded application. It must carry code with it from generation to generation that can receive a new copy of itself, verify it, put it someplace safe, load it into the low 32k of eeprom and then reset itself.
You need to receive a new copy with your chosen protocol and stuff it into a safe place (upper eeprom)
Copy it to lower eeprom when you know it is good
Force a reset - the Propeller serial loader will fail and load from eeprom
Wharever it loads needs to have the smarts to go through the above scenarion when it's time to be replaced roles around
Looks like a great object I can hack to do what I want, thanks mindrobots
http://forums.parallax.com/showthread.php/124343-Parallax-Propeller-Programs-Propellers-in-ParallelV1.0-Tribute-to-the-BigBoy-4?highlight=bigboy
http://forums.parallax.com/showthread.php/127983-55-Parallax-Propeller-s-Parallells-Processing-of-Permanent-Perturbations
G
I am going to assume that:
a) The Prop is on the end of some network connection. This is implied by the mention of RS-485 in the opening post.
b) The normal Prop loader protocol is not workable over this network connection. Perhaps because a reset cannot be applied. Perhaps because of timing constraints. Perhaps because of unique device addressing issues.
The first thing that happens in this situation is that you have a some reprogramming code in you application that accepts a new binary from the net and reprograms the EEPROM.
That is fine until some error causes garbage to be written to the EEPROM. BOOM your system can no longer start properly and you can no longer get communication with it.
So the idea is:
a) Have a program, like a BIOS, in the EEPROM that is NEVER EVER changed.
b) That program on reset looks for code to run in the upper EEPROM. If there is such a program it loads it to the Prop and runs it.
c) If there is no such program in upper EEPROM it listens on the net for such an application which it stores in upper EEPROM, if received correctly, and then runs it.
d) You might want some hardware watchdog thing that resets the prop if things are not going well. Thus starting us again at a)
Point is, that if you cannot reset the Prop and talk to it via the normal Prop loader protocol then you need some other fixed program that you can talk to. That program is in the normal EEPROM space. The actual application code is somewhere else.
I'm tired, I'm not sure I am explaining this well. At the end of the day remote software loading and up date is hard. It has taken MicroSoft almost three decades to get it almost right!
Looks like mindrobots has the right idea. Except, never ever overwrite your lower EEPROM boot loader code. That is the only thing that can save you when all goes wrong.
Or, not.
You could. I'd rather be boiled alive in a flaming tub of pig fat but yes you could.
A funny story:
I was once working on an ARM based embedded system running Linux that needed exactly this kind of robust, remote, over the network, upgrade possibility.
The guys putting the OS and boot loader together arranged to split the available FLASH storage in two. One half ran the "live" system. The other half was available to download new systems in to. When an update was loaded the thing would boot to the new system. There was a hardware watchdog that would kick in if the new system was bad and reboot to the original. It was very clever and robust.
Only one problem, having split the FLASH in two there was no space left to put the application code my team was creating!
One way to "robust" is to keep the "application" off the hardware!!
...that much pig fat is probably bad for the heart!
I'll provide the flaming tub of pig fat it I can watch.
@Heater
You just don't seem to grasp how flexible Forth is on the Propeller. One can sit around waiting for a LMM or struggle with a good OpenAVR tool chain for Simple IDE on a this or a that and in the meanwhile, it is already up and running in Forth.
And while we do have Catalina and GCC to choose from, we even more version of Forth that are sucessful on the Propeller.
The OP seems to have jumped from RS485 to Xbee as it is EASY. Using RS422 and Forth might have kept this a wired project.
I do admit that there are a lot of advantages for an early home automation project to use wireless. You don't have to pull wire to locations that might change. But it is possible to program the Propeller over wire at 1000 meters with Forth.
I guess we need to figure out the definition of successful. (certainly the topic for another thread). There are bunches of Spin projects (some commercial, mostly hobbyist)publicized and in the works with active query threads, and a good number of C based projects (both commercial and hobbyist) and certainly a large population of Spin programmers and a growing number of C programmers (Catalna and PropGCC) in the audience but, I think I can count the people admittedly using Forth on four hands. There have been a few Forth project publicized and I know Peter has quite the commercial portfolio of successful projects but when you split these 20 or so people across 3 different Forths, I'm not sure that supports "successful". (and I'm on of the mistfits who tried to Forth regularly!)
It might go back to what out old buddy, OBC says, "It's Projects Not Platforms!" Well, the same goes for languages, "It's Projects, Not Languages!"
....sorry, back to serial bootloaders, this is very interesting and useful stuff!
If for some reason the new image doesn't work correctly, or gets corrupted, there is usually a way to flip the image flag back to the older image. On many products this is usually done through a network interface or by pushing a button during bootup to get into a configuration screen. However, sometimes this mechanism will fail and the system is bricked. In the case of the Prop, you would have to plug in a PropPlug and reprogram it that way to get it working again.
I concur. The problem of remote upgrades is language independent.
The deal is that when an upgrade goes bad, due to communications errors, power failures, because your new code is not as good as you thought it was, Murphy's law, etc, you need a way to stay on line and accept a repair.
I have seen this go wrong so many times on so many systems from flashing a BIOS on a PC, to Windows updates, to virus scanner updates, to re-flashing embedded avionics systems.
Some try to tackle it by having multiple images of the run time in FLASH or whatever. The idea being that if the new upgrade is bad you can revert to the new one.
But how do you know it's bad? How do you know when to switch back to the last good version if the latest, and now running version, is garbage it may not communicate at all.
My conclusion is that this needs a solid, immutable, piece of code that you can always talk to. With that in place you can tell it from far away to halt the application and replace it whenever you like. This also needs some hardware assistance in the form of a watchdog in case that solid piece of code crashes.
Now, perhaps the Forth argument is valid. If the Forth run time is that immutable system that you can always talk to then you can tell it to halt and replace application code at will.
I'm at risk of finding myself in boiling pig fat here. So the question is how to do that in a more socially acceptable manner in C or Spin for example.
Heaters. comment about overwriting the lower EEPROM was about the safest. I can't speak for the other Forths but as PropForth exists now, you can have an immutable kernel in the lower EEPROM that can the upload application words from a remote source to the upper EEPROM and then reload those application words into the active dictionary. Out of the can, PropForth gives you this capability with 6 free cogs for you application (1 cog is used for serial I/O on P30/31 and the other effectively runs a Forth interpreter you use as the management monitor. Forth comes up ready to talk to something over P30 & P31 - that's just what it does, it sees it as serial, but it doesn't really care what it is. Bluetooth has been done, probably most anything else could be. From running Forth interpreter, you can reload, debug, test, whatever you want to do. Yes, you can even much up the running kernel OR the lower EEPROM if you are really a klutz but you would really, really have to try hard!
Can all this be done in other languages, most certainly. You can write the immutable management/monitor functions and whatever minimal EEPROM support you need to load binary blobs to upper EEPROM and somehow get them into RAM to run. There will be lots of work involved as you do all the infrastructure and then finally figure out how to get these binary blobs into RAM to execute from your immutable kernel. If the processor resets, you load the kernel from lower and then load and append the blobs from upper? I'm not sure how to do that part yet...it seems like loading binary Spin and C blobs is a topic of discussion every so often. We need those loadable blobs.
I think you are right that in any solution, you need a watchdog to trigger a reset.
I'm not touting Forth as the solution, it's just an option. I'd love to see small immutable monitors like this developed in more socially acceptable languages. I think it would be a great feature for promoting the Prop. To have a cog that runs a bullet-proof (mostly) monitor that lets you load and control program execution in the other cogs? Way cool!!!
Now, if you run out of cogs on one REMOTE Propeller?? Well, then we're talking a PropForth solution that's really cool......but, I digress!
Ouch! I was typing while you were typing. Why is sending a text image of words to be stored on EEPROM much different than sending a binary blob to be stuffed into EEPROM? You really don't have to type everything. you can upload chunks of text (or *.f files) if you like. Is that less reliable than sending a *.bin or *.hex file remotely?
Provided that you can send a hardware reset to the Prop over the network connection and provided your connection can support the Propeller loader protocol.
If not then you need some hardware watchdog assistance to reset the thing when everything goes real bad. And you need that immutable other boot loader code that is always available (Thanks to the watch dog) to maintain communication.
Ultimately I would want two independent intelligent devices at the remote end. One is simple and immutable. It does nothing except maintain communication back to home base, it is made fail safe with a hardware watch dog. That immutable thing loads application code into the second device as and when required.
Perhaps two Propeller chips, perhaps a Prop and an AVR in front. Whatever.
You are right. If you only have one non-volatile store a failure in updating a binary or a Forth source code text is the same. Everything fails.
I have yet to be convinced that Forth is a silver bullet to the remote upgrade problem.
A Forth interpreter could boot up in two stages, where the first stage cannot be corrupted. However, that can be done with any language. Boot up a monitor that allows downloading, and the monitor then starts up the application code.
You do need two stages, whatever you do. The first stage of which is immutable and able to interrupt and halt whatever the application (second stage) is doing.
Checksums on the downloaded code will not save you. Murphy will be sure to arrange that the file you are downloading is buggy before it leaves the control centre. Downloaded correctly it will then crash your remote system.
I'm not sure why I'm debating this from my tablet - it just irritates me more?
Can't you checksum a text file? It's just a stream of bits, bytes, however you want to see it.
As for the monitor program, of course you can do it with any language but a lot of the infrastructure you need already exists in Forth. I'm far from saying Forth is the only way to solve this problem but much of what you need to do it already exists. This may be the case where it may actually be the tool for the job. If I was trying this, I wouldn't sit down with C or Spin and start writing a monitor and a command language and a module downloader and a module loader and anything else that was needed to get to the point where I could start writing the "application" but then I'm not as experienced a programmer as either of you (that's not sarcastic, that's a honest assessment and compliment).
@mindrobots, you could build a Forth kernel that uses CRC, hardware watchdog, and more. It would look very similar to a monitor that uses CRC, hardware watchdog, and more, and not very similar to any Forth kernel that I've ever read about.
I suppose that you are right... for expensive spacecraft. But for a budding home automation system that is going to evolve as the learner evolves, Forth might be the perfect tool. Think of Forth more as a development kit, not a final OS.
I have tried to point out all along that the reason I wanted to learn Forth was to actually use it as a stepping-stone into learning PASM. That has worked out extremely well with pfth v1.03 and studying eForth's explanation of their Forth kernel.
I abandoned PropForth and Tachyon just because the underlying code was too clever for me to follow without huge efforts.
IN sum, there is a role for Forth to play. Just bashing Forth because your skill set doesn't require learning it is not helping the educational aspect of the Propeller.
Parity is not really much good in trapping errors; checksums with defined packet lengths can be done and would be more reliable. And YES, loading a Propeller binary is NOT language dependent.
I suppose Heaters proposal to load the top 32K of a 64k EEPROM with a new binary and then have the Propeller take that image and overwrite the lower 32k EEPROM byte-by-byte with a final reset might work. But there certainly is not a safety net if one hits a glitch. And it still has to have some sort of interaction to target the right Propeller on a RS485 party line.
I strongly suspect you need another 32K EEPROM image to toggle to as a failsafe backup. Or a processor sandwiched between the Prop and the RS485 bus to present and manage the unique identity issue.
The Xbee serves the 'sandwiched processor' approach without a lot of development.
And please note that I DID NOT start out by suggesting Forth would do it all for this particular query. I am not a Forth fanatic (to that degree).
In my scheme the lower half of the EEPROM has a Prop program that is NEVER EVER overwritten.
That program on reset looks for and loads a program from high EEPROM or SD or wherever into the Propeller and runs it. It's a boot loader and it swaps itself out of RAM for the application code.
If there is no valid program to be loaded by that boot loader it listens on whatever network protocol you have until you send it one. That is an excellent idea and one that I would go for myself. A dinky little AVR or even a second Prop would be the network interface and application watchdog etc. It's software would never be changed. You will always be able to talk to it and get it to reset and reprogram the Propeller on that node.
Okay, I just presumed this 32K barrier for loading to PropRam could never be violated.
Have you actually seen any productive work in this direction? I guess that the lower 32K could in force a identity query where all the Propellers on the same RS485 listen and negotiate who should shut up. And then offer a green light for a upper EEPROM load.
I still wonder if a RS485 half-duplex set up is going to play well, but at least I begin to envision some of the components.
You may have noticed my 4x2 PropPlug compatible header on many of my boards, the second row of 4 pins has Vcc, <nc>, SDA, and SCL so I can also directly inject a program into the EEPROM without using the Prop's bootloader. I can send a "firmware stick" out to an installation and all they have to do is plug it in and it automatically loads the new firmware. The 8-pin header also doubles as an expansion port too
You were right, the 32K barrier cannot be violated. We only have 32K RAM and the Propellers built in loader will only load a max of 32K from the EEPROM.
That is no what I meant.
The idea is that the program the Propeller loads from EEPROM at reset is itself a boot loader. A secondary boot loader. Let's call it "booter".
Booter knows how to talk over the network using whatever hardware drivers and network protocols you define.
Booter can receive code from the network and store it in the upper half of EEPROM. Or SD card or whatever you like. Let's call that application code "app"
Booter can also read that code, app, back from upper EEPROM or SD card, replacing itself in memory with that code and starting it running.
We are assuming that app also knows how to speak the network protocols as part of it's function. It can be instructed to start booter, perhaps by simply resetting the Prop, when you want to load a new app version. There will also be some kind of hardware watchdog that resets us back to booter if app fails. No but in my Zog ZPU emulator there is exactly that kind of boot loader. to load ZPU code and get the emulator kernel running. That boot loader replaces itself by doing a cogstart on the cog it is running in.
I'm told the technique got used for the propgcc loader as well.
Network protocols and addressing are not something I have thought about much.