Actually it sounds like you are handling files the same way I did in my demos. We have to start somewhere.
Yes, you can have Emscripten fetch files via XMLhttprequests. The are lots of compile time options and I did have it doing that at some point. Can't remember how now, But I wanted it to work from local files anyway. We now have local storage facilities in HTML5 so it would be good to use that so one can work off line.
Regarding the memory leak, you may just have found the closure I was looking for. I have not looked at you code yet. I even tried removing all the code that passes files in and out and it still leaked!
Yes, this same code can run under node.js. I tried it. It worked. Just now I set up a simple web worker experiment. So far I only have a "hello world" web worker running though.
Memory leak is still there in Editor5 I'm afraid. After a few compiles Chrome on Debian is hanging on to a gigabyte of RAM! Firefox does not seem to do that.
Attached is a screen shot of Chrome's Task Manager showing memory usage. (And my browsing habits )
I just hooked Editor5 ino my simple node.js webserver, those loads and saves work a treat.
I am digging into WebWorkers. I think I have a handle on how to do that. Since they definitely run in a different thread it should solve the problem with the memory.
So you have get / put working with your nodes server?
Nice.
So what we need for file access would be to look FIRST in MEMFS for a file dropped by US and IF NOT FOUND try XMLhttprequest on same file.
As far as I can see emscripten does not do that.
There is a way to log file access to printErr. So we might be able to catch the error (missing file) and just restart with the missing file added after loading it from the main thread.
Isn't the editor quite nice, yet?
I will add your css as console style... it is really nice
The opening page is me experimenting for the first time with webworkers and bootstrap so it does not do much except prove I can start and stop a web worker and communicate with it. Try the buttons. Don't know if any of the code behind it is useful for you.
I have also taken the liberty of putting your Editor5 on line there. Follow the link given there.
If anyone wants to see openspin working in a browser go there.
Now you can all fill up my hard drive with your Spin snippets !
Isn't the editor quite nice, yet?
The editor looks OK. I just dropped CodeMirror into the page above. Do you have the Spin mode file I could use with it?
OK. No worries about the Spin mode. I cut and pasted the relevant bits out of Editor5.html and now we have the spin theme on http://a.linuxsecured.net:3000
Now back to playing with Emscripten, openspin and web worker mode.
openspin_web_demo3 is attached. It should work if you unpack it and point your browser at openspin_web_demo3.html.
In that package is the little build script that makes openspin.js. There are no special options used to get that in memory file system in use. So you should be able to recreate it.
If you look in the html you can see how moduleConf.FS_createDataFile() is used to put the spin source into test.spin in that FS.
msrobots Editor5 also uses Emscripten's in memory FS. He does all the ajax stuff himself.
The idea is wrap openspin.js into a web worker. This means that we can kill any memory leaks forever just by starting a web worker process which does the compilation and the dies completely. It's also nice because if the compilation was taking a long time it does not hang up the user interface in the browser process.
I did think about github, in passing. They do have an API of course. It's used by the Cloud 9 web IDE for example https://c9.io/
Seems like rather a big job to make use of it though. My "cloud" storage is likely to simple PUT's and GET's to my own server for the foreseeable future.
The next big challenge is getting the serial I/O access working from Chrome and wrap the thing up as a Chrome app.
That's assuming we can compile the propeller loader from propgcc with Emscripten.
Minor update. The web worker openspin compiler can now send the compiled binary back to the browser window where it is dumped in hex. http://a.linuxsecured.net:3000/
I couldn't wait for the weekend and tried your Demo. Works perfect with Windows 8.1 Chrome/ IE.
So I opened your website with Editor5.htm - nicely provided by yourself. Works like a charm.
I opened file '/' in the Editor - and voila here we go. Very nice integration with the Webworker.
Next I opened file 'worker.js' in a new tab. Perfect. The messaging needs some additions for my purpose, but YEAH this is great.
Then I opened file 'openspin-msrobots-5.js' in a new tab, prepared to wait. But no. Your nodes server does ETAG or the browser does cache it. Instantly the Editor showed the content. Amazing. 1.8 KB Source.
Again very nice made with the Webworker integration. I got stuck at that Message-Thing. I couldn't wrap myself around in a nice way.
So you start openspin, but don't run it. And return the File System in the ModulConfig. This is brilliant.
Now we can fill the filesystem with multiple msgType 'file' calls. Then call compiler. Then pull out what we need with msgType 'get_file'
And then just kill the webworker and done with any memory problems. Perfect.
So I typed in a new filename for each opened tab and made (for example) http://192.168.1.1/worker.js out of worker.js and saved all files I opened from your webserver on my spinneret here in my network. Works like a charm.
Just a thought, but what about offering a server offload capability, to extend to additional compilers?
What @Heater and me are doing here now is offloading the load to the browser. There is no server side here besides PUT and GET requests. But other compilers are possible. If written in C you may be able to compile them with emscripten to javascript. That is what @Heater did with OpenSpin. Now with the webworker running a new compiler is 2 external .js files away.
The problem is the File System. By now we need to provide every file the compiler will need UPFRONT into the Browser File System of emscripten. Then we can run it and pull out what we need.
Huge task for things with libraries like Spin and C.
Having everything work in the browser is great because it solves many, many problems. Using google-drive for data is a perfect storage solution. I'm sure iCloud has something similar.
Still, not everything will work in a browser. Any option is open based on customer's needs.
Well done Heater! Working fine in Opera. And this is on a work machine where I can't install anything, not even the Proptool, so now I can tinker with Spin code at work
what about offering a server offload capability, to extend to additional compilers?
That is where we started with this idea of a web IDE for the Propeller. It all started with a discussion about providing tools for iPad users on an another thread. In my case I saw the tools living on a Raspberry Pi which could easily program a Propeller. Sort of like a super intelligent Prop Plug (They are about the same price around here). I suspect this is the way to go for the C compilers. Getting them to run in a browser is not going to be so simple.
Server side tools will be the way to go if your Propeller is remote. For example both the Prop and the server are at home and you are far away.
What we have here though, if extended far enough, is a browser based IDE that can be used as an app for Chrome books. Chrome books can access serial devices and program the Props. I will be looking into that next I think.
The main point is that server side compiling using existing tools is just not crazy enough for me. Compiling C++ into JavaScript to run in a browser is:)
My suggestion about having a "server side" plugin was after reading the first few pages of this thread.
I personally would have approached the problem that way, because anyone can setup their own server and have a Web-UI (schools for classes, etc).
I'm not scared about the resources required to compile on the server, SPIN, C, etc are all very fast at the scale we are talking about. The advantage of having a "server side" plugin is that you can offload languages that aren't supported in the browser.
As for serial access, I agree that the Chrome native API is a good solution. It would probably be nice to have a local socket service that can interface with JS to do serial on the 3 big platforms: Windoze, OSX, Linux. (I would scrape the code from SimpleIDE and make a cross-platform QT app that can launch from the browser as an application helper).
I would target Android with more native code, I've already explored this and got serial access working with USB OTG.
Good to hear this works on other browsers. Even if they are Windows and IE.
I don't know about "Very nice", "Perfect" or "Amazing". What we have here is a bunch of little experiments in how HTML5, webworkers, CSS, bootstrap, CodeMirror and all the rest works. Lashed together to be almost useful. Hence having everything attached to buttons you have to operate manually.
The messaging needs some additions for my purpose,
Yes it does. I want to tidy it up and generalize it some how. I'm thinking it could be used in many other situations. When ever you have a program that processes files, in whatever way, we should be able to reuse that worker/file transfer/messaging interface.
Working fine in Opera. And this is on a work machine where I can't install anything, not even the Proptool, so now I can tinker with Spin code at work.
Oh God, what have I done? This could be putting peoples lives at risk. I can imagine you hacking on your next Prop creation whilst pretending to be listening to their problems and reading their medical notes.
HTML5 local storage is on my ever growing list of things to look at next. I would certainly want it for use as a Chrome book IDE for example.
The issue of server side or not is not a big deal. One could imagine having a server running on the local machine and taking care of C compilation and such.
Or as I said the server runs locally to the Propeller as an intelligent Prop plug. Such a server can be as small as one of those WIFI enabled, Linux running, SD cards now a days.
A server like that is only a few lines of JavaScript to create in node.js. Yet another thing on my TODO list.
It can get as complex as you like if you start thinking about thousands of users and managing them and their projects. I'm not going that way.
I thought about an Android app ages ago. Nice but not a job for me. I cringe at the thought of spending time working on things that are not cross-platform and Java is just awful.
I was not so serious about the SD card thing but can imagine that soon we will expect a device like a Prop plug to be smart enough to contain the entire development environment and tools. I'm sure its already possible in some way or another.
I think openSpin needs to run/init direct after webworker start.
Then we can call msgType 'file' multiple times with every file we need to put into the FS. - So msgType 'file' needs to put the file direct into the FS - no need for a variable file in the worker. Hence msgType 'file' needs a property 'name'.
Then msgType 'openSpin' can run compiler and msgType 'get_file' pulls the result out.
Now terminate openspin and webworker and we are done.
Finally this can lead to a single click compile button starting the process.
OpenSpin has a command line option to list the files needed by the top object. Using that I will be able to GET the needed files from sd card or external libraries on a webserver supporting CORS. So Libraries - here we come!
By now I am able to run editor and compiler from the spinneret load the needed files from the sd card of the spinneret or any other webserver supporting CORS and save source and/or binary to the sd card of the spinneret or any other webserver supporting CORS and PUT requests.
If I could load another propeller from the spinneret the IDE on a prop-plug is doable. But by now my spinneret code is quiet big (just 200 longs left for stack) and I need to remove stuff to gain space for a loader.
If I remember correctly Mike G. has/had some code to load another propeller from the spinneret...
Comments
Actually it sounds like you are handling files the same way I did in my demos. We have to start somewhere.
Yes, you can have Emscripten fetch files via XMLhttprequests. The are lots of compile time options and I did have it doing that at some point. Can't remember how now, But I wanted it to work from local files anyway. We now have local storage facilities in HTML5 so it would be good to use that so one can work off line.
Regarding the memory leak, you may just have found the closure I was looking for. I have not looked at you code yet. I even tried removing all the code that passes files in and out and it still leaked!
Yes, this same code can run under node.js. I tried it. It worked. Just now I set up a simple web worker experiment. So far I only have a "hello world" web worker running though.
Memory leak is still there in Editor5 I'm afraid. After a few compiles Chrome on Debian is hanging on to a gigabyte of RAM! Firefox does not seem to do that.
Attached is a screen shot of Chrome's Task Manager showing memory usage. (And my browsing habits
I just hooked Editor5 ino my simple node.js webserver, those loads and saves work a treat.
I am digging into WebWorkers. I think I have a handle on how to do that. Since they definitely run in a different thread it should solve the problem with the memory.
So you have get / put working with your nodes server?
Nice.
So what we need for file access would be to look FIRST in MEMFS for a file dropped by US and IF NOT FOUND try XMLhttprequest on same file.
As far as I can see emscripten does not do that.
There is a way to log file access to printErr. So we might be able to catch the error (missing file) and just restart with the missing file added after loading it from the main thread.
Isn't the editor quite nice, yet?
I will add your css as console style... it is really nice
Enjoy!
MIke.
I have made openspin.js available on the net here: http://a.linuxsecured.net:3000
It is served up by a simple node.js web server.
The opening page is me experimenting for the first time with webworkers and bootstrap so it does not do much except prove I can start and stop a web worker and communicate with it. Try the buttons. Don't know if any of the code behind it is useful for you.
I have also taken the liberty of putting your Editor5 on line there. Follow the link given there.
If anyone wants to see openspin working in a browser go there.
Now you can all fill up my hard drive with your Spin snippets !
The editor looks OK. I just dropped CodeMirror into the page above. Do you have the Spin mode file I could use with it?
Now back to playing with Emscripten, openspin and web worker mode.
I don't see an emscripten openspin.js local file system in the browser context. It seems to only be using AJAX. Any help?
In that package is the little build script that makes openspin.js. There are no special options used to get that in memory file system in use. So you should be able to recreate it.
If you look in the html you can see how moduleConf.FS_createDataFile() is used to put the spin source into test.spin in that FS.
msrobots Editor5 also uses Emscripten's in memory FS. He does all the ajax stuff himself.
https://github.com/kripken/emscripten/wiki/Filesystem-Guide
https://github.com/kripken/emscripten/wiki/Filesystem-API
The idea is wrap openspin.js into a web worker. This means that we can kill any memory leaks forever just by starting a web worker process which does the compilation and the dies completely. It's also nice because if the compilation was taking a long time it does not hang up the user interface in the browser process.
I just managed to get this working. See here: http://a.linuxsecured.net:3000
It's very crude as I'm exploring, experimenting and debugging but basically you can:
a) Type Spin code into the edit window.
b) Hit the buttons: "Spawn Worker", "Send File", "Compile", "Terminate Worker"
openspin will compile your source and display the results in the window below.
Now to automate that process into a single "Compile" button. And get the binary file out.
What needs to run on the server to make it work?
I don't have any "load", "save" files to server or such features in there.
I'm actually using a noddy little server written in node.js to do that but any other server would do.
I'd package the whole thing up, server and all, but it's very crude and messy just now.
I tried the same but not successful yet. Will look into this over the weekend...
Enjoy!
Mike
Seems like rather a big job to make use of it though. My "cloud" storage is likely to simple PUT's and GET's to my own server for the foreseeable future.
The next big challenge is getting the serial I/O access working from Chrome and wrap the thing up as a Chrome app.
That's assuming we can compile the propeller loader from propgcc with Emscripten.
http://a.linuxsecured.net:3000/
I couldn't wait for the weekend and tried your Demo. Works perfect with Windows 8.1 Chrome/ IE.
So I opened your website with Editor5.htm - nicely provided by yourself. Works like a charm.
I opened file '/' in the Editor - and voila here we go. Very nice integration with the Webworker.
Next I opened file 'worker.js' in a new tab. Perfect. The messaging needs some additions for my purpose, but YEAH this is great.
Then I opened file 'openspin-msrobots-5.js' in a new tab, prepared to wait. But no. Your nodes server does ETAG or the browser does cache it. Instantly the Editor showed the content. Amazing. 1.8 KB Source.
Again very nice made with the Webworker integration. I got stuck at that Message-Thing. I couldn't wrap myself around in a nice way.
So you start openspin, but don't run it. And return the File System in the ModulConfig. This is brilliant.
Now we can fill the filesystem with multiple msgType 'file' calls. Then call compiler. Then pull out what we need with msgType 'get_file'
And then just kill the webworker and done with any memory problems. Perfect.
So I typed in a new filename for each opened tab and made (for example) http://192.168.1.1/worker.js out of worker.js and saved all files I opened from your webserver on my spinneret here in my network. Works like a charm.
I will build this into the Editor and will report
Enjoy!
Mike
What @Heater and me are doing here now is offloading the load to the browser. There is no server side here besides PUT and GET requests. But other compilers are possible. If written in C you may be able to compile them with emscripten to javascript. That is what @Heater did with OpenSpin. Now with the webworker running a new compiler is 2 external .js files away.
The problem is the File System. By now we need to provide every file the compiler will need UPFRONT into the Browser File System of emscripten. Then we can run it and pull out what we need.
Huge task for things with libraries like Spin and C.
Enjoy!
Mike
Full public server-side access on one machine would be a huge, sweaty, and disappointing work-out though
every user has its own browser, right?. And there it runs. what setup needed?
it is not server side compile it is browser side compile.
All of this is just html and js.
no setup needed at all. Just runs in a browser.
Enjoy!
Mike
Still, not everything will work in a browser. Any option is open based on customer's needs.
Server side tools will be the way to go if your Propeller is remote. For example both the Prop and the server are at home and you are far away.
What we have here though, if extended far enough, is a browser based IDE that can be used as an app for Chrome books. Chrome books can access serial devices and program the Props. I will be looking into that next I think.
The main point is that server side compiling using existing tools is just not crazy enough for me. Compiling C++ into JavaScript to run in a browser is:)
It was a challenge from Jazzed anyway.
http://www.w3schools.com/html/html5_webstorage.asp
My suggestion about having a "server side" plugin was after reading the first few pages of this thread.
I personally would have approached the problem that way, because anyone can setup their own server and have a Web-UI (schools for classes, etc).
I'm not scared about the resources required to compile on the server, SPIN, C, etc are all very fast at the scale we are talking about. The advantage of having a "server side" plugin is that you can offload languages that aren't supported in the browser.
As for serial access, I agree that the Chrome native API is a good solution. It would probably be nice to have a local socket service that can interface with JS to do serial on the 3 big platforms: Windoze, OSX, Linux. (I would scrape the code from SimpleIDE and make a cross-platform QT app that can launch from the browser as an application helper).
I would target Android with more native code, I've already explored this and got serial access working with USB OTG.
--Perry
Good to hear this works on other browsers. Even if they are Windows and IE.
I don't know about "Very nice", "Perfect" or "Amazing". What we have here is a bunch of little experiments in how HTML5, webworkers, CSS, bootstrap, CodeMirror and all the rest works. Lashed together to be almost useful. Hence having everything attached to buttons you have to operate manually. Yes it does. I want to tidy it up and generalize it some how. I'm thinking it could be used in many other situations. When ever you have a program that processes files, in whatever way, we should be able to reuse that worker/file transfer/messaging interface.
What additions do you have in mind?
HTML5 local storage is on my ever growing list of things to look at next. I would certainly want it for use as a Chrome book IDE for example.
The issue of server side or not is not a big deal. One could imagine having a server running on the local machine and taking care of C compilation and such.
Or as I said the server runs locally to the Propeller as an intelligent Prop plug. Such a server can be as small as one of those WIFI enabled, Linux running, SD cards now a days.
A server like that is only a few lines of JavaScript to create in node.js. Yet another thing on my TODO list.
It can get as complex as you like if you start thinking about thousands of users and managing them and their projects. I'm not going that way.
I thought about an Android app ages ago. Nice but not a job for me. I cringe at the thought of spending time working on things that are not cross-platform and Java is just awful.
I think openSpin needs to run/init direct after webworker start.
Then we can call msgType 'file' multiple times with every file we need to put into the FS. - So msgType 'file' needs to put the file direct into the FS - no need for a variable file in the worker. Hence msgType 'file' needs a property 'name'.
Then msgType 'openSpin' can run compiler and msgType 'get_file' pulls the result out.
Now terminate openspin and webworker and we are done.
Finally this can lead to a single click compile button starting the process.
OpenSpin has a command line option to list the files needed by the top object. Using that I will be able to GET the needed files from sd card or external libraries on a webserver supporting CORS. So Libraries - here we come!
By now I am able to run editor and compiler from the spinneret load the needed files from the sd card of the spinneret or any other webserver supporting CORS and save source and/or binary to the sd card of the spinneret or any other webserver supporting CORS and PUT requests.
If I could load another propeller from the spinneret the IDE on a prop-plug is doable. But by now my spinneret code is quiet big (just 200 longs left for stack) and I need to remove stuff to gain space for a loader.
If I remember correctly Mike G. has/had some code to load another propeller from the spinneret...
Enjoy!
Mike