Basically rename main() and introduce a new main that calls it in a forever loop. Thus endlessly compiling the same Spin file in a similar way to what happens in my latest Emscripten set up Like so:
int main_(int argc, char* argv[]);
int main(int argc, char* argv[])
{
while (1)
{
main_(argc, argv);
}
}
// This is the original main.
int main_(int argc, char* argv[])
{
.....
.....
What happens is that is does endlessly recompile. But of course it also rapidly eats all the RAM on the machine. Looks like openspin leaks memory something rotten. Normally this is not an issue as it exits after compiling but it's an issue in this environment.
Yes, JS is garbage collected.
No, there is no way to force Chrome to garbage collect on demand.
Imagine, if you will, some kind of emulator written in JS. All of the memory of the machine being emulated is represented in JS as a big typed array. Now a program for that machine runs under the emulator and starts requesting RAM. What it actually gets is locations in that typed array. When the array is all used up the JS emulator can give up or it can extend the array and continue.
So, if the program running on the emulated machine leaks memory the JS does not know it. It just, correctly, runs the code.
What we have is a memory leak in the emulated code not the emulator.
I'm not 100% sure yet that this is what is happening with openspin.
It's very odd that the demo3 version, which tears down the entire "emulation" for each spin compile has no leak under FireFox but kills Chrome.
There must be a trick I'm missing here as much bigger programs run just fine under Emscripten.
Hey there Heater -- maybe of no help but this looks interesting:
I found a solution. Apparently Chrome leaks DOM nodes, at least in the current version (26.0.1410.65 right now)
I recorded dev tools timeline in my app and it showed the Event Listeners count going up and down rhythmically along with my app screen's contents, but the DOM Node count was steadily increasing over time, until the tab crashed.
I tried the latest Chrome Canary (28.0.1500.3) and they seem to have fixed the problem. DOM Node count graph now follows the same rhythmic pattern as the Event Listeners.
and
as it turns out that gc() can be called from webpage code when --js-flags="--expose-gc" is used. At least on my 23.0.1271.64
Ha ha yes. I found that page yesterday. That business about calling gc() just instantly crashed my Chrome!
Not a nice solution anyway if it relies on running with special secret flags set.
I don't know a bout the leaking DM nodes. That Emscripten generated JS does not deal in any DOM nodes.
What I did no was build a simple C program that reads text from a file, created from a text areas, and then writes the data out to a different file, again displayed in text area. Basically it does what openspin does without the compiler in the middle. That program has been doing that in Chrome and Firefox for many hours now with little memory used and no signs of crashing out. This at least shows the Emscripten framework and connection between Emscripten and the web page is OK.
I am sorry you guys are spending so much time on this. As mainly a Windoze user, I am fine if you can get any browser to work.
I think this is a case where the benefits of being able to work via the web and a browser gives us so many options, that what ever browser works in (Linux, iOS, Android, Mac, Windoze) makes it worthwhile to load it.
This is going to bug me for a while longer. My "model" program works fine, it accepts the same command line arguments, outputs to standard out and standard error and reads and writes files. Just like the real compiler. It just runs and runs. Everything is wired up correctly except for that pesky memory leaking in Chrome.
Has anyone tried the demo3 above in Safari and or Opera? It would be nice to know if Chrome is an odd case or not.
There is yet another way around this. Web workers. Now a days you can fire of a bunch of Javascript to run in a totally separate OS process. That way when it is done the process is destroyed and all the memory returned to the OS. Just as if you were running it from the command line.
Webworkers also have the benefit that if the job you are running takes a long time it does not hang up the user interface whilst it's running.
This might be the thing to look at next. It's a bit harder to do as processes do not share memory at all so you need a way to exchange messages and files between them.
I'm not ready to post a package yet since there are so many things left to do for cross-platform use. Instead, I thought I would offer a video of my desktop running the web-tool prototype in use (no audio). Everything is on one web-page at the moment. I had to make some layout choices that may not be optimal to make things work.
There is much room for improvement - including the name Enjoy the video (no audio).
[video=youtube_share;ko3HLUlxo-Q]
This video demonstrates a prototype web-tool development environment. At this point, only Chrome and Firefox are supported on Windows. There are challenges for multi-platform. The server is written in Python(2.7.x) so multi-platform is possible. It is also possible to port the web server to a Wifi or other Network connected Propeller for a bootloader - that is a work in progress.
For non-optimal, I think usability is high on your first pass. Well done. I like what I see. Personally, I wouldn't worry about all browser support. People can load the ones they need. If it's possible, great, but if it's a PITA, maybe no worries.
This looks really great!! I'm ready to try out the hosted version on a RasPi.
As for the name, you know I'm really fond of "Spinny Weby Devy"...no, really! My concern is that it may not resonate with some of the demographic groups the product needs to target so I have a proposal. Since the tool *IS* web based and is msrobots keeps at it will be hostable on a Spinneret, maybe SPIDeR would be more in line with the marketplace?
SPIDeR is of course, SPin Internet Development Resource (feel free to use it, I'll release the naming right for a free copy of the product! )
I'm using Python because it has everything I needed to quickly get some things working. Also with Python, I can make a self-contained server .exe program for windows users so they don't have to install anything. Python is usually installed by default on MAC and Linux.
I can't complain if t works. I keep forgetting about those legacy Windows users.
I was a bit curious about your statement "Everything is on one web-page at the moment."
The current trend is that we should have "web apps". Only one page, no page reloading.
In terms of and IDE for Spin projects that means being able to switch from one source file to another, with some tab interface or whatever, without page reloading.
Yes, a tab interface would be good to keep from loading different files often. I'm just trying to get the basics worked out at the moment.
What I meant about "on one web-page" is this: At some point we may have a separate editor/compiler and loader/terminal. This would allow very CPU intense operations of non-javascript compiler like gcc to run on a separate machine than the loader/terminal. I haven't figured out exactly how that would work yet.
I was a bit curious about your statement "Everything is on one web-page at the moment."
The current trend is that we should have "web apps". Only one page, no page reloading.
In terms of and IDE for Spin projects that means being able to switch from one source file to another, with some tab interface or whatever, without page reloading.
a) You never need to reload the page.
b) The user interface will never become unresponsive.
The latter implies that a long compilation or download should happen in the background. Perhaps a "web worker" processes is fired off in the browser or perhaps the server does that CPU intensive stuff.
Turns out all of this is not so straight forward as a Qt app.
a) You never need to reload the page. YES.
b) The user interface will never become unresponsive. YES ....
There are some issues where things need to be done one step at a time, and this implies synchronization. For example, you can't load a program from a changed file before it gets compiled. What I have at the moment relies on synchronous transactions, but that obviously doesn't get it because it makes the browser is unresponsive for a few moments. Internet Explorer requires asynchronous communications, and that means b) will be enforced. The trick will be in making the steps be synchronized while also using asynchronous communications.
It's all very straight forward LOL (Number one suspicious management phrase.)
a) You never need to reload the page.
b) The user interface will never become unresponsive.
The latter implies that a long compilation or download should happen in the background. Perhaps a "web worker" processes is fired off in the browser or perhaps the server does that CPU intensive stuff.
Turns out all of this is not so straight forward as a Qt app.
Something is a bit wonky about the use of bold for keywords. Seems bold characters are a different width than normal which leads to indentation not lining up.
The python BaseHTTPServer is a single threaded server. Internet Explorer 10&11 are relying on multiple thread use. This means that if Explorer sends 2 XmlHttpRequests, it will only receive 1 back from the Python server. This happened many times today after I converted everything to asynchronous requests.
With a "real" web server this should not be a problem. However one of the use cases that needs to work is sending requests to a simple server like the Digi S6B Wifi module. As far as I know, the S6B can only process one request at a time like the python server.
Conclusion? Use Chrome/Firefox which gladly support Synchronous XmlHttpRequests and bandon Internet Explorer. Move on to testing other platforms, and see if there are other issues.
The python BaseHTTPServer is a single threaded server. Internet Explorer 10&11 are relying on multiple thread use. This means that if Explorer sends 2 XmlHttpRequests, it will only receive 1 back from the Python server. This happened many times today after I converted everything to asynchronous requests.
With a "real" web server this should not be a problem. However one of the use cases that needs to work is sending requests to a simple server like the Digi S6B Wifi module. As far as I know, the S6B can only process one request at a time like the python server.
Conclusion? Use Chrome/Firefox which gladly support Synchronous XmlHttpRequests and bandon Internet Explorer. Move on to testing other platforms, and see if there are other issues.
Or learn yet another framework twisted framework for writing IP based servers?? Once you get used to the framework, it is actually worth it.
Conclusion. Don't use Python, use JavaScript running under node.js.
I was thinking of this synchronous vs asynchronous issue when I asked "Why Pyhon? I didn't realize Python was so bad at it so I let it go.
Anyway, here is a skeleton webserver in node.js that can handle tens of thousands of simultaneous http requests and web socket connections:
// Import required modules and wire them up
var express = require('express');
var app = express();
var server = require('http').createServer(app)
var io = require('socket.io').listen(server);
// Log http requests
app.use(express.logger());
// Respond to requests for "message"
app.get('/message', function(req, res){
res.send('A message from your server.');
});
// Serve static files from "public" directory
app.use(express.static(__dirname + '/public'));
// Any other request gets this
app.use(function(req, res){
res.send('Nothing here.');
});
// Serve up an index page
app.get('/', function (req, res) {
res.sendfile(__dirname + '/index.html');
});
// Handle web socket connections
io.sockets.on('connection', function (socket) {
// Send hello message to all new socket connections
socket.emit('hello', { hello: 'world' });
// Just log socket client messages
socket.on('message', function (data) {
console.log(data);
});
});
// Start listening
server.listen(3000);
console.log('Listening on port 3000');
I've committed my sources here: https://code.google.com/p/spinaweb/
The project requires Python2.7 and pyserial2.7 for operation at the momen.
There are some improvements to be made. Specifically:
Use a push methodology for the terminal receive. Long polling is used now and works, but is a bit clunky.
Change the user interface to be less prototype.
Also, I want to find a way to break the application into two pieces: 1) Editor/Compiler and 2) Loader/terminal.
@Heater, regarding the server.
As I said one of the use cases for this project will not offer a multithreaded server because of the limits of the given server, so python is fine. Testing with Python ensures that things stay single threaded for that case.
Usually real webserver can reply to thousands of requests without worrying.
But Microcontroller can't. I do not know anything about the WIFI Modules talked about, but I did some homework with the spinneret.
The current spinneret (WizNet W5100) can support 4 sockets (connections) at a time. But the Propeller has to keep up with it. The new proposed spinneret2 will use the W5whatever chip from WizNet and will support 8 sockets/connections at a time. But still the Propeller has to keep up with it.
When a Browser requests a Page it will get the Page from the server and will then send requests for every external css and js file you included. And for every Image you have in there. Usually a single request from the browser to load something like www.parallax.com will produce TONS of requests for the rest of the mosaic - to puzzle it together on the screen. ALL browser will send out these requests in parallel. Usually 10-12 at a time, if not more. If a request fails the browsers usually try it silently three times and then fail. (showing those X things instead of a Image...)
Stuff like that will never work on an Spinneret. You need to code your pages accordingly and inline css, js and if possible images. If all fail the spinneret now supports ETAGS for static files from the SD card. But it is still a request...
My current spinneret firmware does support PUT requests so you can load and save files without problem. My primary goal was to implement WEB-DAV or SMB to get the spinneret SD card usable as file system via Ethernet form a host PC. I basically came short on both attempts. So WEB-DAV under windows still does not like my PROPFIND response from the spinneret. Even if I think them look right. And SMB need way more ports/sockets open to have the spinneret useful after that. But what I have now can be plugged into a network, gets its IP, sets its Name and workgroup and uses one port/socket permanent for NetBIOS to provide its Name Service. So you can ping and browse by NAME not IP. Handy.
But goals evolve and while I was running out of space after adding support for DHCP, NetBIOS and DNS client I decided to get something like dynamic web pages running on the Spinneret. Since I do not have the ability to write a usable scripting language, I decided to go for the simple way and use PASM. Works nice but just 206 longs free for stack or spin now....
Anyways - If you would like to run all of this from a spinneret you need to be very careful with asynchrony operations, so I like @jazzed approach to have it running single threaded.
But this is all about the file/request stuff to save and load. Once you have the stuff in the browser (or HTML5 file system?) you can run javascript as multithreaded as you want. The server (Spinneret) will not care about this.
Now to the Editor
@heater - found it - it was not the 'font-weight: bold' messing it up but some 'letter-spacing: -1px;' Phil tweaked in the stylesheet for whatever reason he had at that time.
So the indentation is shown correctly now for SPIN files. And they are shown in the Parallax font if available.
I have auto completion working for htm, javascript, css and xml pretty nice for all other just a simple 'anyword' list. I still need to integrate Phils Spin autocomplete. Not as easy that task.
Also included is now some CRUDE search and replace thing. This is very odd still.
hmm. you are cheating, a bit. Could not help me and had to look at your .py files. You other way to do this allows us to run propgcc, catalina, propbasic,,el al also - with some setup - or do I have a basic misunderstanding there?
You other way to do this allows us to run propgcc, catalina, propbasic,,el al also - with some setup - or do I have a basic misunderstanding there?
Yes you got it. That's the idea with the other way - that is, split the editor/compiler and loader/terminal into two pages. Just gotta figure that out. That would allow fast editor/compiler load and use since those items can be fully asynchronous. Only the loader/terminal need to be synchronous because of the target (as you clearly understand).
Are you able to use the spinaweb code with the updated editor?
The emscipten spin compiler is important for the case where we don't want to install a compiler. We still need spin libraries and a loader/terminal for that though.
If I recall correctly, dgately has created an iOS version of openspin. I'm not sure if the same can be done for Android - anyone know?
You can build native apps for Android with their native dev kit. No doubt that includes using native libs/programs from a java application.
You can even build Andriod apps in Qt !
I have not played with that for a long time but it did work and Qt has adopted it and is fully supporting it now.
Comments
Good to know. I'm updating to IE 10 today. If you don't hear from me again, it means my computer Explodered
I haven't tried the Javascript compiler in the Webby IDE just yet.
Basically rename main() and introduce a new main that calls it in a forever loop. Thus endlessly compiling the same Spin file in a similar way to what happens in my latest Emscripten set up Like so:
What happens is that is does endlessly recompile. But of course it also rapidly eats all the RAM on the machine. Looks like openspin leaks memory something rotten. Normally this is not an issue as it exits after compiling but it's an issue in this environment.
Yes, JS is garbage collected.
No, there is no way to force Chrome to garbage collect on demand.
Imagine, if you will, some kind of emulator written in JS. All of the memory of the machine being emulated is represented in JS as a big typed array. Now a program for that machine runs under the emulator and starts requesting RAM. What it actually gets is locations in that typed array. When the array is all used up the JS emulator can give up or it can extend the array and continue.
So, if the program running on the emulated machine leaks memory the JS does not know it. It just, correctly, runs the code.
What we have is a memory leak in the emulated code not the emulator.
I'm not 100% sure yet that this is what is happening with openspin.
It's very odd that the demo3 version, which tears down the entire "emulation" for each spin compile has no leak under FireFox but kills Chrome.
There must be a trick I'm missing here as much bigger programs run just fine under Emscripten.
and
both from this clickety:
http://stackoverflow.com/questions/13950394/forcing-garbage-collection-in-google-chrome
Ha ha yes. I found that page yesterday. That business about calling gc() just instantly crashed my Chrome!
Not a nice solution anyway if it relies on running with special secret flags set.
I don't know a bout the leaking DM nodes. That Emscripten generated JS does not deal in any DOM nodes.
What I did no was build a simple C program that reads text from a file, created from a text areas, and then writes the data out to a different file, again displayed in text area. Basically it does what openspin does without the compiler in the middle. That program has been doing that in Chrome and Firefox for many hours now with little memory used and no signs of crashing out. This at least shows the Emscripten framework and connection between Emscripten and the web page is OK.
That is what I want openpin to do.
I think this is a case where the benefits of being able to work via the web and a browser gives us so many options, that what ever browser works in (Linux, iOS, Android, Mac, Windoze) makes it worthwhile to load it.
This is going to bug me for a while longer. My "model" program works fine, it accepts the same command line arguments, outputs to standard out and standard error and reads and writes files. Just like the real compiler. It just runs and runs. Everything is wired up correctly except for that pesky memory leaking in Chrome.
Has anyone tried the demo3 above in Safari and or Opera? It would be nice to know if Chrome is an odd case or not.
There is yet another way around this. Web workers. Now a days you can fire of a bunch of Javascript to run in a totally separate OS process. That way when it is done the process is destroyed and all the memory returned to the OS. Just as if you were running it from the command line.
Webworkers also have the benefit that if the job you are running takes a long time it does not hang up the user interface whilst it's running.
This might be the thing to look at next. It's a bit harder to do as processes do not share memory at all so you need a way to exchange messages and files between them.
I'm not ready to post a package yet since there are so many things left to do for cross-platform use. Instead, I thought I would offer a video of my desktop running the web-tool prototype in use (no audio). Everything is on one web-page at the moment. I had to make some layout choices that may not be optimal to make things work.
There is much room for improvement - including the name Enjoy the video (no audio).
[video=youtube_share;ko3HLUlxo-Q]
This video demonstrates a prototype web-tool development environment. At this point, only Chrome and Firefox are supported on Windows. There are challenges for multi-platform. The server is written in Python(2.7.x) so multi-platform is possible. It is also possible to port the web server to a Wifi or other Network connected Propeller for a bootloader - that is a work in progress.
For non-optimal, I think usability is high on your first pass. Well done. I like what I see. Personally, I wouldn't worry about all browser support. People can load the ones they need. If it's possible, great, but if it's a PITA, maybe no worries.
Why Python?
This looks really great!! I'm ready to try out the hosted version on a RasPi.
As for the name, you know I'm really fond of "Spinny Weby Devy"...no, really! My concern is that it may not resonate with some of the demographic groups the product needs to target so I have a proposal. Since the tool *IS* web based and is msrobots keeps at it will be hostable on a Spinneret, maybe SPIDeR would be more in line with the marketplace?
SPIDeR is of course, SPin Internet Development Resource (feel free to use it, I'll release the naming right for a free copy of the product! )
I'm using Python because it has everything I needed to quickly get some things working. Also with Python, I can make a self-contained server .exe program for windows users so they don't have to install anything. Python is usually installed by default on MAC and Linux.
I was a bit curious about your statement "Everything is on one web-page at the moment."
The current trend is that we should have "web apps". Only one page, no page reloading.
In terms of and IDE for Spin projects that means being able to switch from one source file to another, with some tab interface or whatever, without page reloading.
What I meant about "on one web-page" is this: At some point we may have a separate editor/compiler and loader/terminal. This would allow very CPU intense operations of non-javascript compiler like gcc to run on a separate machine than the loader/terminal. I haven't figured out exactly how that would work yet.
The general idea in web apps is that:
a) You never need to reload the page.
b) The user interface will never become unresponsive.
The latter implies that a long compilation or download should happen in the background. Perhaps a "web worker" processes is fired off in the browser or perhaps the server does that CPU intensive stuff.
Turns out all of this is not so straight forward as a Qt app.
b) The user interface will never become unresponsive. YES ....
There are some issues where things need to be done one step at a time, and this implies synchronization. For example, you can't load a program from a changed file before it gets compiled. What I have at the moment relies on synchronous transactions, but that obviously doesn't get it because it makes the browser is unresponsive for a few moments. Internet Explorer requires asynchronous communications, and that means b) will be enforced. The trick will be in making the steps be synchronized while also using asynchronous communications.
It's all very straight forward LOL (Number one suspicious management phrase.)
No autocompletion yet but a bag of goodies. Syntax-highlighting for c,cbl,cpp,cs,css,h,htm,js,spin and xml.
Full Screen editing and Code Folding....
Still all included in one file.
Rename Editor2.htm.txt to Editor2.htm to use...
Enjoy!
Mike
Cool. works here.
Something is a bit wonky about the use of bold for keywords. Seems bold characters are a different width than normal which leads to indentation not lining up.
The python BaseHTTPServer is a single threaded server. Internet Explorer 10&11 are relying on multiple thread use. This means that if Explorer sends 2 XmlHttpRequests, it will only receive 1 back from the Python server. This happened many times today after I converted everything to asynchronous requests.
With a "real" web server this should not be a problem. However one of the use cases that needs to work is sending requests to a simple server like the Digi S6B Wifi module. As far as I know, the S6B can only process one request at a time like the python server.
Conclusion? Use Chrome/Firefox which gladly support Synchronous XmlHttpRequests and bandon Internet Explorer. Move on to testing other platforms, and see if there are other issues.
Conclusion. Don't use Python, use JavaScript running under node.js.
I was thinking of this synchronous vs asynchronous issue when I asked "Why Pyhon? I didn't realize Python was so bad at it so I let it go.
Anyway, here is a skeleton webserver in node.js that can handle tens of thousands of simultaneous http requests and web socket connections:
I've committed my sources here: https://code.google.com/p/spinaweb/
The project requires Python2.7 and pyserial2.7 for operation at the momen.
There are some improvements to be made. Specifically:
- Use a push methodology for the terminal receive. Long polling is used now and works, but is a bit clunky.
- Change the user interface to be less prototype.
Also, I want to find a way to break the application into two pieces: 1) Editor/Compiler and 2) Loader/terminal.@Heater, regarding the server.
As I said one of the use cases for this project will not offer a multithreaded server because of the limits of the given server, so python is fine. Testing with Python ensures that things stay single threaded for that case.
But Microcontroller can't. I do not know anything about the WIFI Modules talked about, but I did some homework with the spinneret.
The current spinneret (WizNet W5100) can support 4 sockets (connections) at a time. But the Propeller has to keep up with it. The new proposed spinneret2 will use the W5whatever chip from WizNet and will support 8 sockets/connections at a time. But still the Propeller has to keep up with it.
When a Browser requests a Page it will get the Page from the server and will then send requests for every external css and js file you included. And for every Image you have in there. Usually a single request from the browser to load something like www.parallax.com will produce TONS of requests for the rest of the mosaic - to puzzle it together on the screen. ALL browser will send out these requests in parallel. Usually 10-12 at a time, if not more. If a request fails the browsers usually try it silently three times and then fail. (showing those X things instead of a Image...)
Stuff like that will never work on an Spinneret. You need to code your pages accordingly and inline css, js and if possible images. If all fail the spinneret now supports ETAGS for static files from the SD card. But it is still a request...
My current spinneret firmware does support PUT requests so you can load and save files without problem. My primary goal was to implement WEB-DAV or SMB to get the spinneret SD card usable as file system via Ethernet form a host PC. I basically came short on both attempts. So WEB-DAV under windows still does not like my PROPFIND response from the spinneret. Even if I think them look right. And SMB need way more ports/sockets open to have the spinneret useful after that. But what I have now can be plugged into a network, gets its IP, sets its Name and workgroup and uses one port/socket permanent for NetBIOS to provide its Name Service. So you can ping and browse by NAME not IP. Handy.
But goals evolve and while I was running out of space after adding support for DHCP, NetBIOS and DNS client I decided to get something like dynamic web pages running on the Spinneret. Since I do not have the ability to write a usable scripting language, I decided to go for the simple way and use PASM. Works nice but just 206 longs free for stack or spin now....
Anyways - If you would like to run all of this from a spinneret you need to be very careful with asynchrony operations, so I like @jazzed approach to have it running single threaded.
But this is all about the file/request stuff to save and load. Once you have the stuff in the browser (or HTML5 file system?) you can run javascript as multithreaded as you want. The server (Spinneret) will not care about this.
Now to the Editor
@heater - found it - it was not the 'font-weight: bold' messing it up but some 'letter-spacing: -1px;' Phil tweaked in the stylesheet for whatever reason he had at that time.
So the indentation is shown correctly now for SPIN files. And they are shown in the Parallax font if available.
I have auto completion working for htm, javascript, css and xml pretty nice for all other just a simple 'anyword' list. I still need to integrate Phils Spin autocomplete. Not as easy that task.
Also included is now some CRUDE search and replace thing. This is very odd still.
work in progress...
rename Editor3.htm.txt to Editor3.htm to use it.
Enjoy!
Mike
hmm. you are cheating, a bit. Could not help me and had to look at your .py files. You other way to do this allows us to run propgcc, catalina, propbasic,,el al also - with some setup - or do I have a basic misunderstanding there?
Enjoy!
Mike
Are you able to use the spinaweb code with the updated editor?
The emscipten spin compiler is important for the case where we don't want to install a compiler. We still need spin libraries and a loader/terminal for that though.
If I recall correctly, dgately has created an iOS version of openspin. I'm not sure if the same can be done for Android - anyone know?
You can build native apps for Android with their native dev kit. No doubt that includes using native libs/programs from a java application.
You can even build Andriod apps in Qt !
I have not played with that for a long time but it did work and Qt has adopted it and is fully supporting it now.