I just managed to build a program with emscripten that can be run repeatedly from the browser without leaking memory.
It's only a "hello world" program but I think the technique will work for openspin.js as well.
It's kind of disturbing the way Chrome is very lazy about releasing memory. It will let garbage accumulate up to hundreds of megabytes and then thinks about collecting it. So memory usage can ramp up and down quite a lot.
Firefox seems to be more prompt about garbage collection.
The one I'm using keeps getting hung trying to parse fds.start(31,3 .....
I nailed it down to the problem but do not understand why it does that. RegEx is not my friend..
Phils parser gets hung on commas not followed by a space in DAT sections. Same in CON and VAR
funny thing. In PASM it works. but not on data?.
Examples:
mov a, b ' works
mov a,b 'works
BUT
byte $0,$0 ' hangs
byte $0, $0 'works (space after comma)
has to do with the comma since
byte $0+ $0 ' works
byte $0+$0; works also
enter <comma><space>$0
will work
remove <space>
will hung
I did run phils code thru a debugger and it is not advancing the stream somehow and hangs.
adding a space behind the comma in the source to be shown will fix that. (if you find the offending comma without space)
I am looking at Phils code since 2 days since I am moving the editor to CodeMirror3. Not being a JavaScript programmer does not help me much, so I am fighting myself thru the CodeMirror Docs and work on try by example mode.
I do have a new Editor.htm working with CodeMirror3 but by now just for C,C++,C# and JavaScript. Adding Phils SPIN/PASM code is my current challenge.
So hopefully Phil can chime in. I am way over my head understanding the way he is using regEx. (yet)
'Really sorry, but give me a couple days, okay? I'm beset right now with prepping a lesson plan for my robotics class tomorrow and a suddenly tight deadline getting a package design for a friend's fishing lure company off to China. Plus, it will take some time to grok what I wrote so many months ago. IIRC, I had to modify the underlying CodeMirror2 driver to get it to do what I wanted. So that may be the basis for the cited anomalies, if the stock code is assumed.
I had no inclination when I originally did this that it might eventually have legs.
I need more information here. The Editor can not save anything by itself. The webserver has to do it.
Do you start Editor.htm from your local file system or from a website?
If started from website - does server allow PUT ?
To use POST you need to write some POST handler on your server.
The PUT request replaces the content of a URL and is usually disabled on webservers.
I am using this with a spinneret - so I do have full control over the webserver. I might be able to check how to enable this on a windows-based webserver. No clue about Linux.
I am using this with a spinneret - so I do have full control over the webserver. I might be able to check how to enable this on a windows-based webserver. No clue about Linux.
Do you start Editor.htm from your local file system or from a website?
Actually both... After getting the page up from the .html file itself, I placed the file into my web server documents directory, along with a test.spin and and debugging in Safari's Web Inspector. So, the URL is a real one and is being accessed through my Mac's apache web server.
If started from website - does server allow PUT ?
Good question... as far as I know, I've enabled PUT requests, but that does but seem to help, so far.
To use POST you need to write some POST handler on your server.
Ah, will look into that.
The PUT request replaces the content of a URL and is usually disabled on web servers.
Yes, from what I've gathered, so I did as mentioned above tried to enable PUT requests.
I'll keep investigating, though it's not an issue from the Spinneret, so I'll probably go that route for my tests.
Editor.htm is a HTML Page with embedded JavaScript. All of it included in one file, nothing hidden, no external libraries. All of it in there.
clicking Load creates a local request in your browser to load (GET) url <whatever> into the editor
clicking Savecreates a local request in your browser to save (PUT) url <whatever> from the editor
where <whatever> is what you enter in the TextBox.
PUT requests are usually allowed with WRITING allowed on your webserver for distinct directories.
To use a POST request for saving you will need to write some handler (CGI?) on your server and send a different XMLhttprequest.
We are talking about security levels here. All of it makes sense, just read forward...
First of all it depends FROM where you started Editor.htm. Two cases possible. local file system or Webserver.
If run from local file system, you can just use intra/internet based files systems. Those servers need to allow Cross-Origin requests. You do not have access to your local file system.
If run from a Webserver you will ALSO be able to open files on that server. XMLhttp allows local requests, so you are fine. for reading at least.
Now we come to the point of saving.
Editor.htm just send a request to the server to save the document.
The server needs to do the work and save the file.
If I follow you correctly, you are doing this on YOUR local network and not outside in the wild.
Enable writing or WEBDAV on your webserver. This MAY work.
I do not own a Mac, so I am helpless there.
Do you have a Spinneret?
If not, get one. One of the best Products of Parallax...
Please give me a hint to some nice read about RegEx. I need to grook this. I almost get what you are doing in function tokenLexer(stream, state) ...
But then I don't.
This may not be optimal, but it gets me past the major pain:
In function tokenLexer(stream, state) { ....
if (stream.match(/^[+-\.]*[$%\d]/, false)) {
stream.match(/^[+-]?\d[\d_]*(\.\d[\d_]*(e[\+\-]?\d[\d_]*)?)?/i)
|| stream.match(/^[+-]?\$[\da-f][\da-f_]*/i)
|| stream.match(/^[+-]?%[01][01_]*/)
|| stream.match(/^[+-]?%%[0-3][0-3_]*/)
if (stream.match(/^,+/)) {
// this ensures that , is not returned as a number or error.
}
else if (stream.match(/^[\da-f$%\._]+/)) {
return 'error';
} else {
return 'number';
}
}
This version does not leak memory in Firefox although it does in Chrome.
I have added a "test" button that causes the compiler to run every two seconds. With that running you can edit code in the input window and see the results automatically come to the output. Also watch your processes and see how well your browser handles this.
There is a little build script included if you want to recreate this.
In all of this thread and the other about using an iPad,I did not see anything about testing and/or debugging applications. Am I the only one that has to repair coding errors? I have a hard time thinking of these as a complete solution for any significant project without having some way to inspect, analyze and display aspects of a running program.
@Max72 - my intend was to understand Phipi's code not the whole world of regEx. But thanks anyway.
@jazzed - thanks for the links, I will study them. Your fix worked for me too. But I still need to adapt Phipi's code to CodeMirror3. Small differences I guess but I would like to use CM3 instead of CM2.
@heater. Can you maybe post this as zip? Just for me, pretty please? I even installed Chrome, just because of you...
@TinkersALot - one step after the other. By now we are talking about an alternative web-ide for spin/pasm. So we need/have EDIT, COMPILE, SEND to prop and a serial terminal (still needed).for user interaction / debugging. I would love to integrate ViewPorts debugging features, but - honestly - the PropTool does not really offer any ways "to inspect, analyze and display aspects of a running program" either. Or does it?
OK, just for you. Let's not make a habit of this. I thought Windows could handle tar files now a days.
TinkersALot,
...I did not see anything about testing and/or debugging applications....
This is on the WEB now. No body does that kind of thing on the WEB:)
It has been mentioned indirectly. The intention is to stream data back from the Propeller to the browser where it will be displayed in a terminal Window. There you can interact with your running program as you would using the terminal in BST for example. That's about the only debugging aid apart from flashing LEDs I have ever used on the Prop.
P.S. Firefox has been recompiling the same code over and over for many hours now. There really is no sign of a memory leak.
I'm kind of curious why the thing needs 64MB of RAM to run anyway. The compiled JS is only 1.5MB. I took a peek at the openspin source code and immediately found this global declaration:
I try to run all of this from my spinneret - my version supports ETAG for static files since a while. I MAY not need to put everything in one file to save sockets on the spinneret.
But basically - for the spinneret - one big request was better than several small ones. I still need to check the impact of the ETAG handling...
BTW - I am slowly getting annoyed by this Windows ate my computer stuff. I am sort of willing to convert.
A little status update. In addition to items in post #173, I have a web terminal working, but I want to try another methodology before posting anything because of some issues. Guessing it will be a few days from now before something better is ready.
@Heater, looks like you made some good progress with Firefox.
@msrobots, I look forward to your codemirror 3 demo. At a glance, t seems a bit more complicated than the codemirror 2 solution.
In the default way of running programs compiled with Emscripten in the browser the whole C run time is instantiated, the program's main() function is called and then the environment is disposed of. That's nice if you are working with a Qt or SDL program that runs forever with an event loop. For openspin we don't have an event loop we just want to run from beginning to end.
To make it possible to run the program many times, like every time you want to compile some Spin with openspin.js, we can wrap all the Emscripten generated code in a function and call that every time we want a compilation. That function does as above: Instantiate the C run time, call main(), dispose of everything.
This works great in Firefox but for some unknown reason causes Chrome to eat piles of memory each time you run the thing, Eventually we run out of RAM and everything dies.
It turns out there is another way to do this by setting some freaky Emscripten options:
a) Instantiate the C run time when the page loads but do not have it call main immediately. This is only done once.
b) When you have some work to do call main(). This is actually done with an Emscripten function callMain().
This works very well except openspin has a pile of global variables. Those globals are not reinitialized if you don't re-instantiate the C run time, we only set up the run time once, so anything other than the first run goes nuts.
As I said above, openspin needs some hacking to make it work in the browser.
@Heater and @msrobots -- Thank you for replying -- I think this is very interesting work that you are exploring, so I hope my comments do not appear to be too contrarian with a call for some testing / debugging tools to be somewhere on the radar scope.
I sometimes think I must be surrounded by geniuses around here; because I think it must be only me that spends nearly as much time trying to fine tune (and honestly even just make work correctly) projects as I did when I was writing it to begin with.
There is no genius on this end of the line. It's two weeks now and this compiler is still not working as I want. It's only two hundred lines of HTML/JavaScript required. You are missing out on all the failed experiments, hair pulling, frustration, cursing at missing documentation, cursing at things that don't do what they are documented to do, sleepless nights, and general chaos. A typical project life cycle round here:)
There is no genius on this end of the line. It's two weeks now and this compiler is still not working as I want. It's only two hundred lines of HTML/JavaScript required. You are missing out on all the failed experiments, hair pulling, frustration, cursing at missing documentation, cursing at things that don't do what they are documented to do, sleepless nights, and general chaos. A typical project life cycle round here:)
OpenSpin compiles to only 200 lines of Javascript?
I re-jigged the openspin.js demo to do what I described in post #203 above. That is, instantiate the C environment once and then call main() every time a compilation is required. I also added some code to openspin to intialize global variables from main().
The result is much worse!
Firefox seemed to be doing OK but memory usage did go up to 30% of what my machine has. It was stable for a while, automatically compiling every 10 seconds, then CPU load ran up to 96 percent and it eventually died.
Chrome was eating even more RAM and did about the same thing!
I was about to post a package last night that supports Chrome and Firefox. Internet Exploder 9 fails completely though and I know there would be "complaints". I'm going to IE10/11 to see how that works. This is all pretty exhausting to me
Comments
Do you have an updated Codemirror file?
The one I'm using keeps getting hung trying to parse fds.start(31,3 .....
I just managed to build a program with emscripten that can be run repeatedly from the browser without leaking memory.
It's only a "hello world" program but I think the technique will work for openspin.js as well.
It's kind of disturbing the way Chrome is very lazy about releasing memory. It will let garbage accumulate up to hundreds of megabytes and then thinks about collecting it. So memory usage can ramp up and down quite a lot.
Firefox seems to be more prompt about garbage collection.
I've noticed the Chrome laziness before. I usually have many Chrome processes running and they all eat memory like birthday cake.
I am with you there.
I nailed it down to the problem but do not understand why it does that. RegEx is not my friend..
Phils parser gets hung on commas not followed by a space in DAT sections. Same in CON and VAR
funny thing. In PASM it works. but not on data?.
Examples:
You even can try it out on PhilPhi's Spinedit here http://www.phipi.com/editspin/
for example:
scroll down to VAR section
locate long cog
enter <comma><space>$0
will work
remove <space>
will hung
I did run phils code thru a debugger and it is not advancing the stream somehow and hangs.
adding a space behind the comma in the source to be shown will fix that. (if you find the offending comma without space)
I am looking at Phils code since 2 days since I am moving the editor to CodeMirror3. Not being a JavaScript programmer does not help me much, so I am fighting myself thru the CodeMirror Docs and work on try by example mode.
I do have a new Editor.htm working with CodeMirror3 but by now just for C,C++,C# and JavaScript. Adding Phils SPIN/PASM code is my current challenge.
So hopefully Phil can chime in. I am way over my head understanding the way he is using regEx. (yet)
Enjoy!
Mike
'Really sorry, but give me a couple days, okay? I'm beset right now with prepping a lesson plan for my robotics class tomorrow and a suddenly tight deadline getting a package design for a friend's fishing lure company off to China. Plus, it will take some time to grok what I wrote so many months ago. IIRC, I had to modify the underlying CodeMirror2 driver to get it to do what I wanted. So that may be the basis for the cited anomalies, if the stock code is assumed.
I had no inclination when I originally did this that it might eventually have legs.
-Phil
Nice work!
Still looking for Save to work. I get a DOMException when using PUT or POST.
dgately
I need more information here. The Editor can not save anything by itself. The webserver has to do it.
Do you start Editor.htm from your local file system or from a website?
If started from website - does server allow PUT ?
To use POST you need to write some POST handler on your server.
The PUT request replaces the content of a URL and is usually disabled on webservers.
I am using this with a spinneret - so I do have full control over the webserver. I might be able to check how to enable this on a windows-based webserver. No clue about Linux.
Enjoy!
Mike
Do you start Editor.htm from your local file system or from a website?
If started from website - does server allow PUT ?
To use POST you need to write some POST handler on your server.
The PUT request replaces the content of a URL and is usually disabled on web servers.
I'll keep investigating, though it's not an issue from the Spinneret, so I'll probably go that route for my tests.
Thanks,
dgately
Editor.htm is a HTML Page with embedded JavaScript. All of it included in one file, nothing hidden, no external libraries. All of it in there.
clicking Load creates a local request in your browser to load (GET) url <whatever> into the editor
clicking Savecreates a local request in your browser to save (PUT) url <whatever> from the editor
where <whatever> is what you enter in the TextBox.
PUT requests are usually allowed with WRITING allowed on your webserver for distinct directories.
To use a POST request for saving you will need to write some handler (CGI?) on your server and send a different XMLhttprequest.
We are talking about security levels here. All of it makes sense, just read forward...
First of all it depends FROM where you started Editor.htm. Two cases possible. local file system or Webserver.
If run from local file system, you can just use intra/internet based files systems. Those servers need to allow Cross-Origin requests. You do not have access to your local file system.
If run from a Webserver you will ALSO be able to open files on that server. XMLhttp allows local requests, so you are fine. for reading at least.
Now we come to the point of saving.
Editor.htm just send a request to the server to save the document.
The server needs to do the work and save the file.
If I follow you correctly, you are doing this on YOUR local network and not outside in the wild.
Enable writing or WEBDAV on your webserver. This MAY work.
I do not own a Mac, so I am helpless there.
Do you have a Spinneret?
If not, get one. One of the best Products of Parallax...
Enjoy!
Mike
Phil was here.
Great.
Please give me a hint to some nice read about RegEx. I need to grook this. I almost get what you are doing in function tokenLexer(stream, state) ...
But then I don't.
Help. I am stuck.
Enjoy!
Mike
http://www.amazon.com/Mastering-Regular-Expressions-Jeffrey-Friedl/dp/0596528124
Not for the fainted of earth, but the regex is a whole world
Here is a Javascript RegExp reference: http://www.w3schools.com/jsref/jsref_obj_regexp.asp
Thanks Mike, for the quick reference on this...
Yes, I'm doing all of this within my local network.
I don't have a Spinneret, but will probably get one, soon...
dgately
This may not be optimal, but it gets me past the major pain:
In function tokenLexer(stream, state) { ....
if (stream.match(/^[+-\.]*[$%\d]/, false)) { stream.match(/^[+-]?\d[\d_]*(\.\d[\d_]*(e[\+\-]?\d[\d_]*)?)?/i) || stream.match(/^[+-]?\$[\da-f][\da-f_]*/i) || stream.match(/^[+-]?%[01][01_]*/) || stream.match(/^[+-]?%%[0-3][0-3_]*/) if (stream.match(/^,+/)) { // this ensures that , is not returned as a number or error. } else if (stream.match(/^[\da-f$%\._]+/)) { return 'error'; } else { return 'number'; } }This version does not leak memory in Firefox although it does in Chrome.
I have added a "test" button that causes the compiler to run every two seconds. With that running you can edit code in the input window and see the results automatically come to the output. Also watch your processes and see how well your browser handles this.
There is a little build script included if you want to recreate this.
@jazzed - thanks for the links, I will study them. Your fix worked for me too. But I still need to adapt Phipi's code to CodeMirror3. Small differences I guess but I would like to use CM3 instead of CM2.
@heater. Can you maybe post this as zip? Just for me, pretty please? I even installed Chrome, just because of you...
@TinkersALot - one step after the other. By now we are talking about an alternative web-ide for spin/pasm. So we need/have EDIT, COMPILE, SEND to prop and a serial terminal (still needed).for user interaction / debugging. I would love to integrate ViewPorts debugging features, but - honestly - the PropTool does not really offer any ways "to inspect, analyze and display aspects of a running program" either. Or does it?
Enjoy!
Mike
OK, just for you. Let's not make a habit of this. I thought Windows could handle tar files now a days.
TinkersALot,
This is on the WEB now. No body does that kind of thing on the WEB:)
It has been mentioned indirectly. The intention is to stream data back from the Propeller to the browser where it will be displayed in a terminal Window. There you can interact with your running program as you would using the terminal in BST for example. That's about the only debugging aid apart from flashing LEDs I have ever used on the Prop.
P.S. Firefox has been recompiling the same code over and over for many hours now. There really is no sign of a memory leak.
I'm kind of curious why the thing needs 64MB of RAM to run anyway. The compiled JS is only 1.5MB. I took a peek at the openspin source code and immediately found this global declaration:
Which accounts for 2MB straight away that is normally mostly unused.
Then we have: Which immediately eats another 5MB
Similarly there is: Another 5MB.
So there is 12MB reserved in heap space which I'm sure is mostly unused. It's time to have a look at unbloating openspin.
Stock Windows maybe not
but there is 7zip to the rescue.
http://www.7-zip.org/
Massimo
I try to run all of this from my spinneret - my version supports ETAG for static files since a while. I MAY not need to put everything in one file to save sockets on the spinneret.
But basically - for the spinneret - one big request was better than several small ones. I still need to check the impact of the ETAG handling...
BTW - I am slowly getting annoyed by this Windows ate my computer stuff. I am sort of willing to convert.
Enjoy!
Mike
A little status update. In addition to items in post #173, I have a web terminal working, but I want to try another methodology before posting anything because of some issues. Guessing it will be a few days from now before something better is ready.
@Heater, looks like you made some good progress with Firefox.
@msrobots, I look forward to your codemirror 3 demo. At a glance, t seems a bit more complicated than the codemirror 2 solution.
Thanks.
In the default way of running programs compiled with Emscripten in the browser the whole C run time is instantiated, the program's main() function is called and then the environment is disposed of. That's nice if you are working with a Qt or SDL program that runs forever with an event loop. For openspin we don't have an event loop we just want to run from beginning to end.
To make it possible to run the program many times, like every time you want to compile some Spin with openspin.js, we can wrap all the Emscripten generated code in a function and call that every time we want a compilation. That function does as above: Instantiate the C run time, call main(), dispose of everything.
This works great in Firefox but for some unknown reason causes Chrome to eat piles of memory each time you run the thing, Eventually we run out of RAM and everything dies.
It turns out there is another way to do this by setting some freaky Emscripten options:
a) Instantiate the C run time when the page loads but do not have it call main immediately. This is only done once.
b) When you have some work to do call main(). This is actually done with an Emscripten function callMain().
This works very well except openspin has a pile of global variables. Those globals are not reinitialized if you don't re-instantiate the C run time, we only set up the run time once, so anything other than the first run goes nuts.
As I said above, openspin needs some hacking to make it work in the browser.
I hope Roy is out there:)
I sometimes think I must be surrounded by geniuses around here; because I think it must be only me that spends nearly as much time trying to fine tune (and honestly even just make work correctly) projects as I did when I was writing it to begin with.
cheers!
There is no genius on this end of the line. It's two weeks now and this compiler is still not working as I want. It's only two hundred lines of HTML/JavaScript required. You are missing out on all the failed experiments, hair pulling, frustration, cursing at missing documentation, cursing at things that don't do what they are documented to do, sleepless nights, and general chaos. A typical project life cycle round here:)
Openspin compiles to a tad over 38000 lines of JS. 1.8 mega bytes.
That 200 hundred lines is only the wrapper I put around it to get it running in the browser.
I re-jigged the openspin.js demo to do what I described in post #203 above. That is, instantiate the C environment once and then call main() every time a compilation is required. I also added some code to openspin to intialize global variables from main().
The result is much worse!
Firefox seemed to be doing OK but memory usage did go up to 30% of what my machine has. It was stable for a while, automatically compiling every 10 seconds, then CPU load ran up to 96 percent and it eventually died.
Chrome was eating even more RAM and did about the same thing!
I despair.
I was about to post a package last night that supports Chrome and Firefox. Internet Exploder 9 fails completely though and I know there would be "complaints". I'm going to IE10/11 to see how that works. This is all pretty exhausting to me
we should be OK on IE10.
Have you got your own thing going there Jazzed or are you using something derived from demo3 ?
My demo4 is a total disaster.