Update Flash from SD?
JonnyMac
Posts: 9,105
in Propeller 2
I have several projects on P1 that can update themselves by copying the update (binary) file into the EEPROM and then rebooting. Can this be done with the P2, that is, can I have PNut create a binary file that I can overwrite the flash with?
So far, my experiments have failed. I can confirm that the flash for the new code gets written from the binary file, but when I reboot the P2, it seems bricked.
Comments
I would write something to dump the eeprom. Load it in RAM and compare the eeprom image between being loaded by PropTool and being loaded with your updater. Play "one of these things is not like the other".
I think you can't just write the binary straight to flash, there needs to be some sort of second-stage loader in front. Don't remember the details exactly. Have you checked the actual flashloader program? I think PNut comes with it.
I have a RAM program that will compare what's on the SD card with the number of pages in Flash required to hold that program -- they match.
Loading the .bin file created by PNut into Flash didn't work, I wrote a RAM program that would write the Flash (which held the image I wanted) to a file, then reloaded it back into the flash with my loader code. That didn't work either. Ada is probably right, there's more to it that just copying the file into Flash, though Chip told me that would work. That said, he was buying car parts and I was out getting lunch. I'll see if I can catch up with him tomorrow.
I wonder if some of the old AES copy protection is still in the boot loader. I seem to recall something about all the fuses being blown to ensure it wouldn't cause problems. But that was the Rev A. chip.
No that doesn't make any sense. Probably a 2nd stage bootloader as @Wuerfel_21 said.
Taqoz can write an update of itself to Sd and also to Flash. I have not seen a second Stage bootloader. The word to look up is BACKUP.
You can just rename a binary file to xx.BIX to copy it on sd, which would not work, if there was a bootloader., I think.
Edit this is about the first 128k. Probably a 2nd bootloader is needed, if you need more.
I have it sorted (thanks to a couple of phone calls with Chip last night) and am working on a demo. The goal for me is to have my P2/Spin2 products be updated in the field using an SD the same way I do with the P1/Spin1 today. The Spin2 program has to do some of the work of the loader -- which is a bit of a PITA because the loader (e.g. PNut) can buffer the whole file into the PC's RAM. Obviously, this is not the case with an app running on the P2. I'll explain everything when I post my demo.
The attached demo shows how to a P2 application can overwrite the flash with a new application. It's a little tricky -- and not practical. I post this for educational purposes only. Thanks to a late night call and explanation from Chip, I was able to have Spin modify the contents of the binary image so that it will work in flash. To let people experiment I embedded a small file (flashes P56 and P57) into the app and wrote methods that simulate dealing with a file system.
The attached difference checker image shows the .binary file on the left, and what's in flash after F11 programming on the right. This will make sense as you look through the very simple and verbose demo code.
For work and clients, I'm going to make the modifications using a Python script -- unless I can get the tool makers to respond to my next suggestion.
@cgracey, @macca, @ersmith, @RossH
For those of you who create tools that export a binary file for the P2, could you add a feature that would export a file without flash programmer code (not needed by the P2 that's already running, and with the values needed to boot (app longs, app longs, negative app sum, negative loader sum) in the file. That would dramatically simplify a P2 program updating itself from an external files. These features may be available in Eric's and Ross' tools -- sorry, I don't have a lot of familiarity with them.
I suggest the modified binary (with loader and ready-to-boot app) have a .p2img extension.
Thanks for documenting this. Appears there is some preamble that needs to go before the actual code, right?
Yes; four longs, just before the app data. There are three values in the four longs: the number of longs in the application (two copies), the sum of the longs in the application, and a pad value that will make the sum of the 1st 256 longs be $706F7250 ("Prop"). The comments in the code explain how to get from A-to-B -- I just don't think we need to do this in a P2 app when the binary can be modified before we need it.
I'm working on a Python utility that will take a .binary file, snip off the flash programmer, then put those values into place before writing a new file. Self updating in P2 apps becomes easier:
Easy-peasy -- no on-the-fly calculations. Hopefully, the tool makers will consider this a worthy to addition to their tools.
A standard .binary shouldn't have a flashloader to begin with? May be a PNut oddity, but I don't think it even does that by default. I know flexspin only does plain executable binaries (well there's the LZ4 decompressor, but that's functionally transparent)
Calculating the checksum on-the-fly shouldn't be a big deal - just need to write the first sector last.
PS because it just came to me: If there's interest I can see if I can make a python/whatever standalone version of flexspin's built-in P2 executable compressor
I know enough about Python to write very simple things -- like this, which is simple. I've not had the patience to learn Python properly, so I'm sure it can be improved.
Goal: Take the .binary file created by PNut, strip off the flash programmer segment (first 0x160 bytes), then do the calculations and touch-ups on the loader segment so that the image can be read from a file, written to flash, and will run on the next reset.
This little Python program does that. It has only been tested on one file, so be careful.
It would be nice if the forums would allow Python files to be uploaded without ZIPping....
Follow-up. I have tested with an SD card. It worked.
Seems there is a missing source for the flasher.p2img itself.
I did a quick test and seems fairly easy to do since, in my case, the flash loader is prepended at the programming time, I need to split the flash_loader.spin2 to have only the bootloader source and prepend it (with the correct checksums) to the actual compiled binary.
Unfortunately seems that I can't get PNut and my compiler aligned to produce a byte-exact binary so it is a bit difficult to test.
It's included in the jm_p2img_loader_demo archive. It was created from flasher.bin that's in an archive from an earlier post.
Was this with an sd card that shares pins with flash? As in Eval board?
No. Separate SD with power control (my own P2 accessory based on a suggestion for adding a PFET from Terry Trap).
To use that SD socket on the Eval I think we'd need a combined SD/Flash driver object. My Flash object uses smart pins which would probably create issues for the SD drivers I'm experimenting with.
On the P2, Catalina programs are just pure PASM programs assembled with Dave Hein's fabulous P2 assembler. The program starts at ORG 0. No flash programmer.That must be a recent PNUT/Propeller Tool addition. Catalina doesn't use any Parallax tools. I do have an old flash loader written in assembly by @ozpropdev (see attached) which I still use and it seems to do much of what you need. But it only loads flash from Hub RAM and would need additional work to be able to load from the SD.
Ross.
Got some time to work in this:
$ $HOME/spin-tools/spinc -u -e -o flasher.p2img Hello.spin2 Spin Tools Compiler - Version 0.38.0 Copyright (c) 2021-24 Marco Maccaferri and others. All rights reserved. Compiling... Hello.spin2 +-- jm_fullduplexserial.spin2 +-- jm_nstr.spin2 Program size is 7132 bytes Done. $ $HOME/spin-tools/spinc -u -r -p /dev/ttyUSB0 -T 230400 jm_p2img_loader_demo.spin2 Spin Tools Compiler - Version 0.38.0 Copyright (c) 2021-24 Marco Maccaferri and others. All rights reserved. Compiling... jm_p2img_loader_demo.spin2 +-- jm_fullduplexserial.spin2 +-- jm_nstr.spin2 +-- jm_p2_flash.spin2 Program size is 15316 bytes Uploading... Propeller 2 on port /dev/ttyUSB0 15320 bytes sent Verifying checksum ... OK Done. Entering PST terminal mode. CTRL-C to exit. P2 Image Loader Demo Update file..... flasher.p2img File size....... 7132 Flash sectors... 2 Press any key to update flash... Erasing flash Done (100171us) Writing flash Done (18832us) Press any key to reboot into new code Hello, Flash World!Looks good to me.
Anything I need to check ?
Looks good to me, too. I appears you've added a new feature to spinc.exe (-e) that will create the .p2img file. Am I correct?
I also noticed that spinc.exe has -u for unused methods removal. Can this become a menu option?
Thank you, Marco.
Yes.
This command line option is a bit "special". Originally the compiler always removed the unused methods (and still does) and this options simply activated a filter to exclude all the unused method warnings. However I had the need to generate the full binary to compare with PNut binaries and was tired to change the source for that so I changed the command line compiler to not remove the unused methods by default and the -u to enable the removal.
I don't think I'l add a menu option for that since I don't think there is really the need to keep unused methods.
Okay. In the P1 laser tag we are (presently) using BST to compile without unused methods -- it's the only way to fit into 32K. It will be less a problem in the P2, but in the next gen (P2) laser tag I have to send updates across an XBee link, so I want to minimize file size.
I released the standalone executable compression script the other day: https://forums.parallax.com/discussion/175981/p2-executable-compression-as-standalone-script
The really big gains are when you have empty or repetitive data areas in your binary*, but it should eek out a decent improvement regardless. I think PNut's bytecodes may be more suitable to LZ-style compression than PASM. It should also reduce boot time.
(*: for reasons(tm) flexspin always includes pre-initialized VAR space into the binary, PNut clears out some memory after the loaded code. So flexspin always has really large file sizes without compression. The extreme of this happens in my emulators where most of the memory is laid out manually to ensure some alignments - so there's just multiple 64K blocks of just zeroes in the middle of the code. Current MisoYume goes from 445112 bytes to 68916!)
Is another way to just program flash with what you might want. Then, read flash into a file on uSD. Then, when needed, write that file back to flash?
This seems foolproof, right? Even for cases where code is in upper flash...
That's how I detected the differences between what PNut creates as a binary and what is actually in flash (post #8). That said, it's an extra, and in my opinion, unnecessary step (I already created an extra step that's simpler with my Python fixer). I'm grateful that Marco (Spin Tools) is so responsive to user requests. After I presented the Python fix he integrated it into his command-line compiler and tested with my demo program.
One day, I'm hoping @ersmith will help me get micropython working with code in upper flash (think between 512 kB and 2024 kB). If that happens and all you care about is loading various versions of micropython, then maybe what I suggested makes sense.
But, on reflection, I see it's not practical in the general case...
@JonnyMac Do you think the uSD should have a capacitor for VDD, close to socket?
Might borrow some ideas from your board design there, thanks for posting it.
It seems like the right thing to do, and I see other designers do it as well. I made a small update to my uSD board and have just ordered new PCBs and parts (see attached).
For myself, might add a QWIIC connector using the two extra pins.
Might also see if an SMT RGB LED could coexist with the "detect" signal...
With a QWIIC connector, would be easy to add an RTC, so could time stamp files...
Might also contemplate adding a blue led with say 1k resistor between cs and vdd so as to monitor activity…
I do that on a lot of my P2 accessory boards -- when I have pins and it's easy to route.
You might consider adding a separate header so that you can easily connect a logic analyzer. I have a P2 Accessory extension board that Steven Moraco gifted me that I use for that.