Ah ha! Yes. Perhaps. I had been transforming it from UTF16 to UTF8 to ASCII. It still looked OK but now I found even native openspin could not compile it without the undefined symbol error.
So I deleted all the old Spin files and downloaded a new FullDuplex4Port object. Now it compiles in my editor, yipee!
But not in Editor9 on my server or Editor10 on your site.
For Editor9 running here I can have it fetch the file from my server or cut and paste the code into the editor window and it will fail.
For Editor10 I can only paste the code into the window and it fails in the same way.
I'm using the files from fullDuplexSerial4port_1v01.zip as in Tracy Allen's fulluDuplexserial4Port OBEX entry. http://obex.parallax.com/object/248
Specifically fullDuplexSerial4port.spin which oddly enough says "FullDuplexSerial4portPlus version 1.01" at the top.
By the way, the syntax highlighting looks great now. Well done.
By the way, the syntax highlighting looks great now. Well done...
Thank you. I tried to match background and foreground as close as possible. And after hours of 'snipping' screenshots of various sources in the PropTool, just to get the colors, basically as I was done with it, I found the Color Dialog in the PropTool.
I know. Life is hard. But its harder when you are stupid.
btw. The Editor does highlighting for html,js,css,C, C++ and Cobol also...
We have two stacks spaces and two heaps/garbage collectors to think about.
There is a stack and a GC in JavaScript itself of course.
Then there is the stack and heap in the "virtual machine" that is Emscripten running C code.
As far as I can tell that stack space error comes from JS itself. As I said I have never seen that unless I did something really stupid like a run away recursive call.
Also the error comes on the very first file that is written to the worker. The only file in my case. Why should that be so stack consuming?
I had a little look at your code. I notice you are still doing the string to binary to string conversion as you send files to the worker.
That is different to my code now. After you told me not to worry about encoding and let JS take care of it I removed all that binary conversion and just send the file as a string.
Perhaps it's that "ab2str(data.buffer)" you have in your put_file handler that is causing the problem somehow.
Yeaaah, I was really missing that COBOL syntax highlighting!
I finally found it. You know, that thing lurking around and messing everything up. It was hiding in the shadow, as usual.
I will explain:
For the need of multiple calls to openspin you added
[SIZE=2][COLOR=#008000][SIZE=2][COLOR=#008000]// TBD: What to do with this?
[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2]ModuleConf[[/SIZE][SIZE=2][COLOR=#a31515][SIZE=2][COLOR=#a31515]'noExitRuntime'[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2]] = [/SIZE][SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0]true[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2];[/SIZE]
to our webworker calling openspin.
so I followed it up and found out why this is affecting us is in -> openspin.js
[SIZE=2]Module[[/SIZE][SIZE=2][COLOR=#a31515][SIZE=2][COLOR=#a31515]'callMain'[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2]] = Module.callMain = [/SIZE][SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0]function[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] callMain(args) {[/SIZE]
...
[SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0]try[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] {[/SIZE]
[SIZE=2] ret = Module[[/SIZE][SIZE=2][COLOR=#a31515][SIZE=2][COLOR=#a31515]'_main'[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2]](argc, argv, 0);[/SIZE]
[SIZE=2][COLOR=#008000][SIZE=2][COLOR=#008000]// if we're not running an evented main loop, it's time to exit[/COLOR][/SIZE][/COLOR][/SIZE]
[SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0]if[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] (!Module[[/SIZE][SIZE=2][COLOR=#a31515][SIZE=2][COLOR=#a31515]'noExitRuntime'[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2]]) {[/SIZE]
[SIZE=2] exit(ret);
}
}
[/SIZE]...
so we inhibit exit(returncode) to run and do not do it by our self later.
SOLUTION:
just like running
[SIZE=2]ModuleConf[[/SIZE][SIZE=2][COLOR=#a31515][SIZE=2][COLOR=#a31515]'callMain'[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2]](ModuleConf[[/SIZE][SIZE=2][COLOR=#a31515][SIZE=2][COLOR=#a31515]'arguments'[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2]]); [/SIZE][SIZE=2][COLOR=#008000][SIZE=2][COLOR=#008000]// Call openspin's main method. [/COLOR][/SIZE][/COLOR][/SIZE]
I just woke up and I'm not sure if I understand what you are getting at.
What memory leak are we trying to fix?
Firstly. There are two reasons why I put the web worker in:
1) Kills all memory leaks. If you load, compile, terminate. Then if anything leaks it is gone away with the worker going away.
2) User friendly. The browser UI does not hang up while compilation goes on.
So, the way I use it there are no memory leaks. We load all files to the worker, do the compilation, get the binary, kill the worker. Done.
OK. That aside I don't get the next part:
What is the difference between us calling "ModuleConf" and just changing the setup to "ModuleConf = true;" thus allowing to Emscripten call it for us?
Further, how does calling "ModuleConf(exitstatus);" immediatly prior to having the worker terminate help anything? The worker is going to die and whatever memory leaks it had internally will die with it.
...we just need to call ModuleConf before terminating the webworker. that's all.
Yes but why? What does it do for us?
As far as I can tell when the web worker terminates, it's gone. Everything, gone. Whatever calling exit() does that's gone to. I don't see how it can change anything. What is the problem we are trying to fix with this anyway?
The webworker is a whole new process, like any other in your operating system. In our case that process runs a JS engine which in turn runs our JS which in turn runs Emscripten's JS which in turn is running openspin.
Terminating the web worker destroys that process as surely as "kill -9" under Linux or whatever in Windows. That in turns kills the JS engine and our JS and Emscripten. Everything. No matter what state it is in. All process memory goes back to the OS just as it does for any other process.
So it should not matter if we exit the emscripten run time or not. It's all going away any way.
1) We start the webworker. Which starts up the Emscripten run time but does not run main().
2) We throw files at it which end up in the Emscripten FS.
3) We tell the worker to compile. It calls main() to do that.
4) We fetch the binary from the worker.
4) WITHOUT stopping the worker we can continue at 2) probably with different files.
This would be clean and simple and quicker than what we do now.
The problem comes with running that main() a second time. Every time we call main() we leak memory. I believe that this is because openspin itself allocates a lot of heap at start up which it never frees. They are global data available to all openspin modules. It may also have other memory leaks as it runs.
Normally those leaks don't matter as the OS cleans it up on exit. When the process that runs it dies.
That's what we do by killing the worker.
Potentially we could fix all the leaks in opespin but I don't know if that is ever going to happen.
Once again, a new build of openspin.js. This time with all misaligned memory accesses fixed. It's now compiled without the Emscripten UNALIGNED_MEMORY option which should in theory make it a bit faster and enable further optimizations.
All at the links given above.
Now that openspin.js is stable, thanks Roy, I switched on optimizations when compiling with Emscripten. It's about 10 times faster!
openspin.js now compiles WebServer_W5100_RTC in 2.53s in my Firefox browser and 2.81s in Chrome. That compares well with native openspin which takes 0.61s on the same machine.
That's is, JS is only a factor of 4 slower than native code !
I think if it had been required to write it in Javascript initially, it never would have happened. :P
Also, the C++ version is going to get faster, so your converted version may improve also.
Comments
So I deleted all the old Spin files and downloaded a new FullDuplex4Port object. Now it compiles in my editor, yipee!
But not in Editor9 on my server or Editor10 on your site.
For Editor9 running here I can have it fetch the file from my server or cut and paste the code into the editor window and it will fail.
For Editor10 I can only paste the code into the window and it fails in the same way.
All very odd.
both work. hm my testfile is called FullDuplexSerial4portPlus.spin different file?
what are you using?
can you try to 'upload local file' in the first menu instead of cut and paste?
Enjoy!
Mike
Specifically fullDuplexSerial4port.spin which oddly enough says "FullDuplexSerial4portPlus version 1.01" at the top.
By the way, the syntax highlighting looks great now. Well done.
I just tried to compile the whole spinneret msrobots. This one is complicated and includes a lot of objects, some multiple times.
And there it is. Out of stack space while saving files to MemFS.
So still memory issues. bummer. I restart the webworker every round.
Is it maybe wrong to return the FS object to the main (caller)? Does the memory issue is there?
It is handy to have there, but we may disable the GC with it?
Enjoy!
Mike
Thank you. I tried to match background and foreground as close as possible. And after hours of 'snipping' screenshots of various sources in the PropTool, just to get the colors, basically as I was done with it, I found the Color Dialog in the PropTool.
I know. Life is hard. But its harder when you are stupid.
btw. The Editor does highlighting for html,js,css,C, C++ and Cobol also...
Enjoy!
Mike
We have two stacks spaces and two heaps/garbage collectors to think about.
There is a stack and a GC in JavaScript itself of course.
Then there is the stack and heap in the "virtual machine" that is Emscripten running C code.
As far as I can tell that stack space error comes from JS itself. As I said I have never seen that unless I did something really stupid like a run away recursive call.
Also the error comes on the very first file that is written to the worker. The only file in my case. Why should that be so stack consuming?
I had a little look at your code. I notice you are still doing the string to binary to string conversion as you send files to the worker.
That is different to my code now. After you told me not to worry about encoding and let JS take care of it I removed all that binary conversion and just send the file as a string.
Perhaps it's that "ab2str(data.buffer)" you have in your put_file handler that is causing the problem somehow.
Yeaaah, I was really missing that COBOL syntax highlighting!
Sure enough I removed str2ab() and ab2str() from your Editor9 when talking to the web worker and now it compiles fullDuplex4Port just fine !
It's up on my server now.
I wonder why ab2str() eats stack like that?
Even if it does not make any sense. Will try out tonight.
I was able to install emscripten on win8. At least I hope I did. Not tested yet.
Enjoy!
Mike
attached is Editor11. for testing available at http://parallax.msrobots.net/Editor11.htm
Enjoy!
Mike
Chrome was able to compile the complete spinneret-msrobots. Ha. 12 included sub-objects. all well.
IE10 got stuck at including Dhcp.spin. Maybe due to earlier errors while testing? I need to add more error checking. to figure that out.
Anyways - besides some problems it does quite nice.
Enjoy!
Mike
Turns out Firefox does not like the "console.log()" call that you have in the web worker, take that away and it works on Firefox as well
I will get rid of console output completely.
I finally found it. You know, that thing lurking around and messing everything up. It was hiding in the shadow, as usual.
I will explain:
For the need of multiple calls to openspin you added
to our webworker calling openspin.
so I followed it up and found out why this is affecting us is in -> openspin.js
[SIZE=2]Module[[/SIZE][SIZE=2][COLOR=#a31515][SIZE=2][COLOR=#a31515]'callMain'[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2]] = Module.callMain = [/SIZE][SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0]function[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] callMain(args) {[/SIZE] ... [SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0]try[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] {[/SIZE] [SIZE=2] ret = Module[[/SIZE][SIZE=2][COLOR=#a31515][SIZE=2][COLOR=#a31515]'_main'[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2]](argc, argv, 0);[/SIZE] [SIZE=2][COLOR=#008000][SIZE=2][COLOR=#008000]// if we're not running an evented main loop, it's time to exit[/COLOR][/SIZE][/COLOR][/SIZE] [SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0]if[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] (!Module[[/SIZE][SIZE=2][COLOR=#a31515][SIZE=2][COLOR=#a31515]'noExitRuntime'[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2]]) {[/SIZE] [SIZE=2] exit(ret); } } [/SIZE]...so we inhibit exit(returncode) to run and do not do it by our self later.
SOLUTION:
just like running
we need to run
when done with openspin. But this will throw a exception as implemented by emscripten FOR A REASON. So handle like this:
... [SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0]case[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2][COLOR=#a31515][SIZE=2][COLOR=#a31515]'terminate'[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2]:[/SIZE] ... [SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0]try[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] {[/SIZE] [SIZE=2] ModuleConf[[/SIZE][SIZE=2][COLOR=#a31515][SIZE=2][COLOR=#a31515]'exit'[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2]](exitstatus); [/SIZE][SIZE=2][COLOR=#008000][SIZE=2][COLOR=#008000]// Call openspin's exit method. This will throw an exception with value exitstatus if successful.[/COLOR][/SIZE][/COLOR][/SIZE] [SIZE=2] } [/SIZE][SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0]catch[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] (e1) {[/SIZE] [SIZE=2] log ([/SIZE][SIZE=2][COLOR=#a31515][SIZE=2][COLOR=#a31515]'w OpenSpin run time env. destroyed. '[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] + e1);[/SIZE] [SIZE=2] log ([/SIZE][SIZE=2][COLOR=#a31515][SIZE=2][COLOR=#a31515]'w OpenSpin run time env. destroyed. Stack: '[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] + e1.stack);[/SIZE] [SIZE=2] } [/SIZE]in your webworker and you are fine.
NO memory leaks in CHROME and IE anymore.
will post Editor12 in about 10 min or so.
Enjoy!
Mike
I just woke up and I'm not sure if I understand what you are getting at.
What memory leak are we trying to fix?
Firstly. There are two reasons why I put the web worker in:
1) Kills all memory leaks. If you load, compile, terminate. Then if anything leaks it is gone away with the worker going away.
2) User friendly. The browser UI does not hang up while compilation goes on.
So, the way I use it there are no memory leaks. We load all files to the worker, do the compilation, get the binary, kill the worker. Done.
OK. That aside I don't get the next part:
What is the difference between us calling "ModuleConf" and just changing the setup to "ModuleConf = true;" thus allowing to Emscripten call it for us?
Further, how does calling "ModuleConf(exitstatus);" immediatly prior to having the worker terminate help anything? The worker is going to die and whatever memory leaks it had internally will die with it.
@Heater. - You did everything right. No worry.
ModuleConf = true has to stay
we just need to call
ModuleConf before terminating the webworker. that's all.
sorry to confuse you
update on the way
Enjoy!
Mike
Yes but why? What does it do for us?
As far as I can tell when the web worker terminates, it's gone. Everything, gone. Whatever calling exit() does that's gone to. I don't see how it can change anything. What is the problem we are trying to fix with this anyway?
here we go attached the latest version:
Editor12.htm.seems to run stable with all I throw at it. multiple times, multiple files, no problem.
@Heater. I think killing the webworker did not kill the emscripten runtime.
By setting ModuleConf = true we say to emscripten NOT to kill the runtime. Because we still need it. And that is all OK.
Alas when we are done we NEED to tell emscripten that it is like that OR wait for the GC to figure it out.
Anyways. attached is Editor12.
view live here: http://parallax.msrobots.net/Editor12.htm
Enjoy!
Mike
I have no clue.
But Chrome AND IE do now compile anything I throw at them without memory errors. Even without opening a new editor/browser-window.
without call to exit they don't.
Enjoy!
Mike
I still don't get it.
The webworker is a whole new process, like any other in your operating system. In our case that process runs a JS engine which in turn runs our JS which in turn runs Emscripten's JS which in turn is running openspin.
Terminating the web worker destroys that process as surely as "kill -9" under Linux or whatever in Windows. That in turns kills the JS engine and our JS and Emscripten. Everything. No matter what state it is in. All process memory goes back to the OS just as it does for any other process.
So it should not matter if we exit the emscripten run time or not. It's all going away any way.
Where were you seeing memory errors?
1) We start the webworker. Which starts up the Emscripten run time but does not run main().
2) We throw files at it which end up in the Emscripten FS.
3) We tell the worker to compile. It calls main() to do that.
4) We fetch the binary from the worker.
4) WITHOUT stopping the worker we can continue at 2) probably with different files.
This would be clean and simple and quicker than what we do now.
The problem comes with running that main() a second time. Every time we call main() we leak memory. I believe that this is because openspin itself allocates a lot of heap at start up which it never frees. They are global data available to all openspin modules. It may also have other memory leaks as it runs.
Normally those leaks don't matter as the OS cleans it up on exit. When the process that runs it dies.
That's what we do by killing the worker.
Potentially we could fix all the leaks in opespin but I don't know if that is ever going to happen.
try this link: http://parallax.msrobots.net/Editor12.htm?a=http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/WebServer_W5100_RTC.spin
this will open the editor with the top object of spinneret-msrobots
hit compile binary
wait and watch
it will take some time. You can look at the source while compiling.
now tell me WHY it sometimes work and sometimes not?
I am clueless. Need to sleep. Your turn.
Enjoy!
Mike
Impressive.
I can't make it fail or see anything going bad in Chrome on my Debian boxes, 32 and 64 bit. I'll try again later,
I'm cluless too. Have to think about this some more...
Sorry I'm too busy with other stuff to help at the moment. The progress so far without me is tremendous:) I have faith it can continue.
See it in action here : http://the.linux.d.org:3000
Grab it from here : http://the.linuxd.org:3000/openspin.js
As far as I can tell it works fine.
All at the links given above.
Roy has done a great job of removing memory leaks from openspin. Now the JS version can be run repeatedly without having to take down the web worker.
Same links as above.
Now that openspin.js is stable, thanks Roy, I switched on optimizations when compiling with Emscripten. It's about 10 times faster!
openspin.js now compiles WebServer_W5100_RTC in 2.53s in my Firefox browser and 2.81s in Chrome. That compares well with native openspin which takes 0.61s on the same machine.
That's is, JS is only a factor of 4 slower than native code !
This is a very usable speed.
This is exciting having this project and PropellerIDE coming along so well. It's like the Perfect Storm for Spin programmers and Propeller users!
I think if it had been required to write it in Javascript initially, it never would have happened. :P
Also, the C++ version is going to get faster, so your converted version may improve also.