Binary Download for Propeller 1 and 2 Field Personnel
Ken Gracey
Posts: 7,401
in Propeller 2
Hey all,
We have a continual need to give our customer's customer (the end-end user) a way of easily updating their Propeller firmware. The current approach (command line with PropLoader, or installing development tools) is a bit cumbersome, and we're looking for a simple solution.
Therefore, I present a proposal to you, attached as a PDF. I've talked with @macca and he's certainly got some good ideas, though Javascript could make a big installation package. Are there other ideas, and if so, would anybody be interested in developing this system?
I know that @JonnyMac prefers an SD card system. We will get that done, too.
Our goal is to make it available by March 2025.
Thank you,
Ken Gracey
kgracey@parallax.com
Comments
I've been doing that on the P1 for ages. As of yesterday, Chip tells me that he has an update to PNut that will create a file that can be loaded from the SD directly into the Flash (I know how to do this) and will run after the P2 is rebooted.
My needs for an SD loader do not preclude the need for a serial downloader app as described. I think @macca may have the best handle on this as his Spin Tools IDE is a very nice x-platform GUI that knows how to find and download to a Propeller.
That's correct! He told me about it at 4:52 am this morning.
For the serial solution described above, I think Macca has the best solution so far and that Java may be the easiest way to get us to where we want to go. I'd just like to verify that there are no simpler, leaner filesize solutions that should be considered. Does Python save us from any big installers? And should @macca be willing to take this on, we will all be thankful!
Ken Gracey
I've never done a GUI app in Python, but according to the Internet -- where all information is reliable
-- there is something called PyInstaller that alleges to convert a Python program to a single file for distribution.
-- https://pyinstaller.org/en/stable/
Seems like you'd want the option to first copy the existing flash image to a backup file.
Then, verify the new flashed image after copying it over...
Ken,
Some companies are restrictive on what Windows users can do.
At my recent job someone from IT had to allow each of us to run the custom applications on each computer we used.
When I was at J&J they instituted restricting USB drives unless you really needed it.
Without knowing any other context, has anyone considered using Qt or WxWidgets in C++ rather than going with Java, Javascript, or Python?
I used to do a lot of Java and Python and a reasonable amount of Javascript. My intuition (which, again, is only that) would say for someone to take a look at QT first.
PyInstaller was a good program back when I used it, but it has a fanout problem that should be considered, see https://pyinstaller.org/en/stable/usage.html#supporting-multiple-operating-systems :
"If you need to distribute your application for more than one OS, for example both Windows and macOS, you must install PyInstaller on each platform and bundle your app separately on each."
They do helpfully point out that you can run the other target operating systems in virtual machines, of course. It's great if you're big enough to have automated builds and continuous integration.
(It also brings up questions about packaging for Apple on a non-Apple platform, but I suspect Apple is going to be the hardest target because of Gatekeeper anyway. The codesigning aspect of all this could be nontrivial given how locked down Windows and Apple platforms have become. You might do better to have a single cross-platform "programmer" app that's signed by Parallax, but where you can easily drag-and-drop a file containing the Propeller binary and additional customer information onto it just like a drag-and-drop upload in Chrome.)
@"Ken Gracey", @frederickthe3rd The use of a language instead of another may certainly solve the package size issue, but doesn't solve other issues:
Single .exe “Propeller Firmware Loader”
Python requires a runtime and also all libraries / DLLs for the frameworks used to design the UI and the needed features, the only way to achive this is with a C/C++ program statically linked to all needed libraries and this may not be possible simply because not all frameworks ships with statically linkable libraries, unless you want to include the sources in the project (which IMHO is overwhelming).
There are other options, like for example Appimage for Linux or, if I'm not wrong, the Portableapps packaging for Windows, but these things are all read-only packages (Linux Appimage is a readonly filesystem appended to a frontend that mounts the image to a temporary directory and starts the executable from there) and the tools to create them are not multiplatform. Paradoxically MacOS apps are the easiest to manage because are simple compressed archives.
Signed Windows, Linux, MacOS versions
Excluding Linux that doesn't have the code signing requirements of the other OSes, this is a real nightmare, especially with the customization options (the firmware itself, customer logo, etc.) because whenever you alter the executable you also invalidate the signature thus the PPCT tool need to sign it again and this is not possible unless the developer has its own sign certificates, and again, these sign tools are not multiplatform. I don't think a simple hack to append the firmware to a signed exe will work.
On that matter, I strongly suggest to skip the code sign for now. On Windows (10/11, not required for previous versions, AFAIK) it just asks the user to confirm it want to run the program (one time) this can be written in the tool documentation. The most problematic is MacOS, I'm looking at how to overcame the issue, but still it just requires a similar override by the user. Package validation can be made with a simple hash sum.
So my proposal is to build a simple "universal" programming tool for the end user, shipped in a traditional zip/tar.gz package with all required runtimes and libraries that doesn't need any installation procedure. All OSes can unpack a zip file from the UI. This is exactly how Spin Tools IDE is packaged.
This has a number of advantages, the executable can be signed by Parallax and the PPCT can include customized configuration files without altering the signature. The end user also can dowload the tool one time and then receive only the updated firmware files, doesn't need to download the tool every time. May also be possible to include a button to download the updates from the developer web site.
I have some ideas and also draft an UI for the programming tool, I'll post them later today.
Here are some ideas for the implementation:
Firmware packaged in a json file.
Instead of using the binary image I tought of encapsulating it in a json file with additional settings for the end user.
Being a text file is easy to edit without specific tools and can be generated with a simple python script (or any other language), may not be even necessary to have a dedicated creation tool.
Can be extended if and when needed.
With an universal firmware programming tool, the developer can send only the json file instead of the whole package each time, or the file downloaded directly from the developer's web site.
An example:
{ "firmwareList": [ { "description": "jm_i2c_devices.binary - P8X32A Firmware", "binaryVersion": 1, "binaryImage": "..." } ], "enableLocal": true, "enableNetwork": false }The firmwareList array may contain multiple firmware versions, for different devices (also P1 and P2 versions) or to have the old version to downgrade a device in case of problems.
The draft UI for the firmware programmer.
We have a file selection box that can either select a binary or json-packed file.
A drop-down list to select the firmware versions available in the package.
A section to select the device(s) to program, with choices to program all available devices or only the selected from the list. The list is also filtered by binary type (P1 or P2) so only valid targets are shown (except WiFi which, AFAIK, can't be detected with the discovery query). The list shows the device type, wifi name and IP and an indicator if the programming was successfull.
Button to discover devices from local (usb) serial ports and/or network (Wifi).
The user can do a non destructive upload to RAM, if the Write to EEProm / Flash box is unchecked, to test the firmware.
The above shows a possible customization option, with a company logo and an embedded firmware file (the user can't select another file).
Looks good. Very clear.
I suppose the Json settings could enforce certain settings too, like always write to eeprom instead of providing a tickbox choice.
Looks good. Very clear.
I suppose the Json settings could enforce certain settings too, like always write to eeprom instead of providing a tickbox choice.
I don't really like forcing settings, I prefer to have defaults that the user can change, but yes, we can have a parameter to force the eeprom to be written.
I must add that the settings should never be saved locally, at each program launch the defaults are read from the json file or from the internal defaults.
One reason to force certain settings is to simplify the interface and remove the option for a user to make a mistake- like only updating ram.
We have to consider that not every end user tasked with an update wants to be presented with options they know nothing about!
I suppose if you have the JSON for "default programming mode", either RAM or EEPROM, then adding an option to "allow user to change programming mode" would make sense ?
That would hide (or freeze) that selection box if the user is not supposed to access it. Whilst also allowing the automatic setting to be chosen.
I was thinking about a parameter like "writeFlash" with arguments "true", "false" or "force". true/false sets the default that the user can change, force hides the checkbox and always write to eeprom (or P2 flash).
That is very similar. I'm not sure if there might be occasion that someone would want to force RAM only write, but that was what I was shooting for by splitting the two settings as I did.
That way the requirement to have a "force" parameter is not required either- it's just true/false to allow or disallow user adjustment of the program mode.
And by also setting the programming default mode, you either ensure, or suggest, that the user chooses that mode
Of course- your call for what you think best as you are developing the thing!
@macca watching this from a distance and letting the questions unfold. It seems you are well on your way and we would be thrilled to have your implementation as our preferred solution. Even with signing, with JavaScript - this is not a problem. Can you make this tool work both standalone and as part of Spin Tools IDE?
I will also contact you by e-mail offline.
Thank you, Ken Gracey
Yes, I would like to integrate the firmware package creation tool into Spin Tools IDE.
I think I can also integrate the "bulk" uploader, if that's what you mean.