A Propeller-WebTool Framework for Compiled Languages?
jazzed
Posts: 11,803
I'm interested in exploring the idea of porting the Propeller-Tool (or other "compiled") Propeller development system entirely to the web browser context. The requirements of the system would be to function almost identically to the Propeller Tool. For lack of a better name, I would call this a Propeller-WebTool.
While it the concept may be oriented toward a web browser, it is theoretically possible to use this framework with any Propeller module having an SD Card and a serial port.
A Propeller-WebTool SystemWhat would be a Propeller-WebTool development system? Major components:
1. A compiler. A Javascript program that will take user's source and convert to binary code.
2. An editor. An Javascript interface for editing user's sources (need load/store ability).
3. A loader. A Javascript program that will send compiled binary code to the propeller through a special EEPROM loader.
4. A terminal. A Javascript program that will allow connecting to the propeller.
5. A Network Enabled Propeller Device. The target device for programming.
That list is essentially the same as what we have today except the last item. Addressing each item in turn as required ....
1. A compiler.The compiler is high on the list of complexities and costs. Please note that this thread is about "compiled" languages (like SPIN, C, PropBasic) moderators will receive requests for off topic posts.
For this particular concept, having the compiler run in the client's web-browser would be ideal especially because of the cheap and light-weight wireless (Bluetooth, Wifi, or Xbee) devices that can provide serial connectivity to the Propeller chip.
In another thread I asked Heater to translate openspin to Javascript since he mentioned having nothing to do. Within a matter of hours he produced some results using the Emscripten engine (I haven't used Emscripten, so I wasn't sure how to proceed).
I have some questions:
1. Can I run the javascript in a web browser?
Apparently yes, but how do I interact with it?
2. Will it run in all web browsers? Apparently not Internet Explorer:
"Assertion failed: Cannot fallback to non-typed array case: Code is too specialized."
Can that be fixed?
3. Can the resulting code be modified to use a buffer (DOM textarea) as input for compile?
TBD.
4. Assuming the compiler can be adopted, can a Javascript propeller loader be created using a Wifi serial port?
TBD.
5. Can other language compilers be ported?
TBD.
2. An EditorThere is a web-enabled HTML/Javascript editor for SPIN programs already provided by a forum member. I haven't used it, so I don't know what it's capabilities are. Does the author or anyone have a link?
There is also the Adafruit supported WebIDE. Can that be configured for Spin syntax?
Source code Load/Store support: The easiest implementation for the user of load/store is to have the source saved on the target device. That is the network device that is hosting the HTML/Javascript for the Propeller-WebTool.
For example, in the case of RPi it is easy because everything can go into the RPi SDCard file-system. In the case of other devices the on-board Propeller SDCard would be used for user's Propeller-WebTool storage.
3. A LoaderA loader would need to be written. It could be part of the compiler, but it would not be a standard Propeller loader. The current Propeller loader is not tolerant of delays that would be caused by network issues. While determinism is great, in this respect insisting on it is actually a problem.
The loader needs to be "2 sided." That is, on the web-browser "client side", a transfer mechanism should be worked out that is compatible with the Propeller target "host side" protocol. The client/host protocol must be tolerant of network delays. We have lots of these kind of details already worked out in the Propeller-GCC loader's XMM serial_load protocol.
4. A TerminalA terminal would be required for "debug output" or just general message input/output on the target Propeller Device. It is not clear what code might be available to act as a serial port web terminal. The protocol application layer used must be very light-weight. The terminal program should attempt to automatically choose the device to use.
5. A Network Enabled Propeller DeviceA target Propeller device should be network enabled. This could be achieved using Bluetooth, Wifi, Xbee, or another device like Rasberry PI. The Propeller lower 32KB EEPROM would contain a boot-loader program (upper 32KB should be available for the user). The boot program would talk to the serial port to handle programming packets. This program would reside entirely in a COG and allow overwriting all of HUB RAM. One difficulty involves resetting the device because not all network virtual serial port devices have a control signal that can be used for remote reset.
Final thoughts.Someone might say that this is all solved with the RPi or other super cheap Linux box. Well it is, but the only thing different really is that the compiler itself would run on the host system in the case of RPi. The other elements discussed are web-enabled supplements. And theoretically could work with any Propeller with SD Card attached.
Comments
I don't know if this is the one you're thinking about:
Here's the forum thread:
-Phil
http://forums.parallax.com/showthread.php/134496-Forums-SPINtax-Highlighter-(Chrome-AND-Firefox-supported!)?p=1035739&viewfull=1#post1035739
http://forums.parallax.com/showthread.php/148261-CyanIDE-Web-based-propeller-IDE?p=1187091&viewfull=1#post1187091
Reviewing the notes though, it seems like it does not meet my criteria for clent-side only support in all browsers.
I did notice some javascript syntax highlighting regex code here somewhere ... wish i had bookmarked it.
Have you ever heard of XMLHttpRequest? That can be used to send/receive information to a propeller bootloader that is attached to a network virtual serial port.
There is no need for any server side processing support.
Actually there is a need, but the commands can be handled by the propeller bootloader.
As long as the server has the propeller device attached, we can send/receive data to propeller.
-Phil
What questions do you have?
Yes, I'm very sure we can get the Open Souce Spin Compiler to run in the browser. As Jazzed says one can "compile" C and C++ to Javascript with emscipten, This sounds crazy what with JS being "a stupid slow interpreted" language. But it works very well and the resulting code runs at usable speed. You end up with a JS function that can be called to get running at main() it takes the normal arguments.
Emscipten provides the usual standard C libraries so no C code should need changing to run in the browser and have access to stdio and a file system.
It's probably up to us to privide functions that upload/download files from a server or enable access to local files, There are lots of emscripten demos that show how this can be done here: https://github.com/kripken/emscripten/wiki Cross browser compatibility should be good. It's only JS after all. I have no idea about the particular error. I suspect such bugs will get sorted with time. Try the demos in IE. They even have Qt GUI demos! See above. No changes required. It's best not to change the original source so as to avoid maintenance problems. Just hook the JS fread and fwrite to wherever your data is. Modern HTML5 browsers have a file system API and local storage to play with. Or have the files in the cloud. You can go directly to a serial port in Chrome OS! Not possible in browsers normally.
Otherwise that seems a bit tricky. You can upload the binaries using the normal HTTP requests or websockets so what ever is connected to the Prop needs to be able to handle that.
As a start I would make a simple server in node.js that accepts the binaries from the browser, programs the Prop and relays results back to the browser. That node.js can also relay serial terminal communications when the Prop is running. Sticking my neck out I will say yes! With some confidence:
Emscripten's front end is Clang. Clang produces LLVM "bit code" which is then translated to JS.
The Catalina C compiler is compiled with Clang when it runs on the Mac. Ergo it should be possible to "emscripten" it!
Not sure about propgcc, What is that compiled with on the Mac?
Just for fun here attached fft_bench emscriptenized for the browser. Just unpack it and point your browser at the fftbench.html
There is the Cloud9 IDE: https://github.com/ajaxorg/cloud9/ Which is hosted for free at http:// https://c9.ioif you want to try it out.
Or just use the ACE editor which has customizable syntax highlighting (It is the editor component of C9) http:// http://ace.c9.io
Source code Load/Store support:
Save files on a server using HTTP requests or via web socket messages, websockets will be needed for terminal communication with the Prop when it runs anyway.
Or store files local, using HTML5 local storage APIs,
How about Github integeration?
In the browser you need term.js https://github.com/chjj/term.js
term.js is a very nice VT100 terminal component for the browser It can communicate via websockets back to a server process which is connected
to the Propeller via serial.
A first run at this might be simple a node.js server handling the web socket connection on one side and the serial port communications on the other. It would also be responsible for accepting binaries and firing up a loader to program the Prop. The whole web sockets thing is very easy to deal with in
node.js.
Apart from the actual "doing it" part of course:)
This is a great thread and a super start to writing up the problem and a start on solutions.
I agree with Heater. on most of this (ouch! Sorry man if that hurts your forum cred!
#1-#4 are all doable now except for gluing all the pieces together.
#5 is going to always be the trickiest with the P1 (and probably the P2).
Mike G did some early spinneret work with being able to download new images via the ethernet connection so that is doable. The bluetooth is being done with certain non-compiled languages as you know. I'd like to see if the new Xbee WiFi module Ken is playing with has some capabilities we can exploit for remote loading. We need to get some of those to play with!
The trick is always going to be the low level wrapper that needs to be put around the code to be loaded and/or the small footprint network arware boot loader that needs to be developed/enhanced/maintained for the selected technologies. The P2 might make this a little easier with it's monitor program and larger memory capabilities.
Outside of Chrome OS, I don't think you will be able to directly talk to a Propeller. I don't know if the OTG features on many Android tablets can be exploited to provide the booting/loading functions but then you are needing a little glue in the form of an Android app and limit your solution to Android tablets. The RasPi route has its drawbacks (but probably the least expensive solution).
I'll certainly be following along and willing to play along whenever I get past my current distraction.
I haven't figured out what goes on with file I/O yet but if the worst comes to the worst we can always write our own fread/fwrite etc.
https://github.com/kripken/emscripten/wiki/Interacting-with-code
https://github.com/kripken/emscripten/wiki/Emscripten-browser-environment
https://github.com/kripken/emscripten/wiki/Filesystem-Guide
https://github.com/kripken/emscripten/wiki/Filesystem-API
Experiments are under way...
The Open Source Spin Compiler does indeed run in the browser when converted to JS, Turns out we have to adjust the total memory allowed for the C memory space.
I set TOTAL_MEMORY to 64 Mega Bytes (up from 16MB) in the emscripten settings and bingo it comes to life.
Here it is: http://a.linuxsecured.net/openspin.html
Point your browser at that and you should see the openspin sign on message. If you open your browsers JS console you should see openspin's usage instructions.
Now, how to get command line args and files in and out of there...
Heater, MikeG, Phil, and Rick: thanks for all the input so far. I hope we're building a team here ... at least core contributors are showing up.
Heater, Wow
MikeG, how hard would it be to write a bootloader on propeller that could also serve pages from the SD Card to the serial port?
Phil, you've done so much already with code-warrior. It seems there is still much to do with respect to wrapping that into a web-app.
Rick, if you're available, your wonderful word-smithy could be very helpful.
I think Parallax could help here too since what we are discussing ultimately serves them.
Two generic hardware connectivity cases seem most important (are there others?). 1) network connected devices, and 2) USB serial devices (most common case). Both cases have their own problems, but can be overcome. In 1) network connection, we need to carry the extra cost of external hardware and a bootloader. In 2) serial, we need to find a common way to connect - I've mentioned virtual serial port before, but the details need to be worked out.
I've looked around at various Wifi modules. The one that seems most likely to meet the need is the Digi International XBee Wifi (I ordered this one also available here). Why? It has many regulatory approvals as a stand-alone unit, it has modern ad-hoc connectivity, it has strong documentation of API and other features.
Parallax has an existing relationship with Digi, and hopefully some beneficial arrangements can be made ... assuming a qualifying solution.
Much more to do....
Check http://a.linuxsecured.net/openspin.html again.
Haha! I just hit the link with my iPhone and it compiled it! :0)
There do appear to be some browser issues - I tried before on my cheap Android tablet and got the same error Steve got a few posts back. We'll need more testing and see if we need to limit browsers or can fix it for the trouble makers!
We'll done, Heater.!!
Object {dev: 1, ino: 16, mode: 33279, nlink: 1, uid: 0 }
blksize: 4096
blocks: 8
ctime: Sat Dec 28 2013 22:55:26 GMT+0200 (EET)
dev: 1
gid: 0
ino: 16
mode: 33279
mtime: Sat Dec 28 2013 22:55:26 GMT+0200 (EET)
nlink: 1
rdev: 0
size: 32768
uid: 0
That's a stat the 32768 byte eeprom file floating around in the browsers memory some place.
By the way, it worked on my Samsung Android tablet running official Chrome!
I'm not sure. I have a pretty good idea how to do it though.
I verified it worked in both browsers on my Android phone, Chrome, and Firefox. Still fails in Exploder9 though.
Was a little concerned about performance at first, but it seems to be a download issue more than anything. The question is whether or not the Propeller is fast enough to deliver such a huge javascript page (1.6MB) from SD Card fast enough to keep a user's attention. That could be a make or break at 115200bps if SD Card read is that fast ....
Cheers.
Wait up a minute. Who said we were serving up all this from a Propeller?
The JS can be compressed with gzip so that 1.6MB will get much smaller.
I would have thought we could get the browser to cache the JS so that huge download only happens once. I haven't had much luck with that.
Now the big problem: openspin is currently run once when the page is loaded. What we want is for it to be run on demand when we have edited a file.
I tried to hook up a run button that calls the run() method again. run() is created by emscripten and ultimately calls main.
It didn't work. openspin did run but the output file was empty and then it grabbed a huge pile of memory each time and eventually locked up!
Googling around it seems that running things repeatedly is a bit of a problem and messy. Might need to put the JS in a webworker or make openspin into and endless loop that never exits and acts like a sort of compile server.
Or, I notice the compiler in openspin is a built as a separate library. Perhaps we can use that directly from JS and forget having a main.
I just minified the .js using this - it went from 1699kb down to 640kb.
I wanted to test the big source versus the minified source on my PC so I copied Heater.'s .html and .js down to my PC but when I run it locally, openspin.js gives me this error; from source line 46 - are there some helper .js files (or others) that are generated from emscripten that I might be missing?
I want to have the two different .js files generate some non-trivial Spin to see if they both work.
This is currently being done on Windows7 under Firefox - this combination works fine with your test site.
I'll go look at other aspects of this project.
Steve, I'm also a little confused now.
How does the SD card, the serial link and a/the propeller serving up the .js file all hook together?
Sorry yes. I have been hacking away on the code in that live server! Learned a few new emscripten tricks along the way.
But now I have totally broken it:) The emscripten building is all being done manually at the moment and the generated code overwrote my hand modified code. Despite the fact I was keeping back ups here and there I messed it up and lost the latest nice working version. I blame the flue I have been suffering from all over Christmas.
That error you got "Access to restricted URI denied" happens when a browser is not allowed to read files due to some security constraints. For example when your view HTML from a local file Chrome is not allowed to load js from a local file. This can be enabled somehow, I forget how.
Ideally you want those files served up from a server running on your local machine. I use nginx.
Anyway, I'm going to have to start again, this might take a while....
Oh yes, I'm not sure but I think emscripten has an option to automatically minify the js with the closure compiler.
I'm working on a Product Requirements Document that we can discuss. Once I have the requirements and usage configurations listed, I'll post it.
We don't need requirements! You gave us a great user story and Heater is running in full agile mode now!!
Propeller-WebTool
Name subject to change.
Document Author
Jazzed
Contributors and Stakeholders
Heater, Jazzed, MikeG, Mindrobots, PhilPilgrim, Others
Purpose
To provide a multi-platform, multi-OS, internationalized Propeller programming environment via a user's web browser.
Requirements
The minimum requirements of the product are listed here. At a high level, it is simply a feature set. The first post in this thread goes into some detail on each item. Requirements with greater detail can be developed with contributor feedback.
Feature Overview
- Editor
- Compiler
- Loader
- Terminal
- Network Serial port access
- PC/SmartPhone capable
- International capable
WebTool ConfigurationsMinimum WebTool configurations
- Platforms with a Web Browser accessing network connected Propeller
- PC device browsers: Chrome, Firefox, Internet Explorer, Safari
- Touch-screen device browsers: Android phone or tablet, iPAD, iPhone
- On-line or off-line Internet capable
- Resource use (compiler and other tools) determined by server
Propeller ConfigurationsConfigurations define the use cases where the product may be applied. The product should work with these target configurations.
- Propeller board with XBee Wifi
- USB Propeller on a Linux, Mac, Windows PC (includes RPi)
Specific examplesMe, agile?
In the old days we called it "hacking", now it's an a bona fide methodology
The value has dropped to zero just now. I have to regroup my remaining braincells and try again. Perhaps I it's best if write down what I learned/did so far some place.