I guess we can live without the change history and if you don't need it that's that. To me change history is only important when I break stuff
You might want to tag the initial commit as v1.0.0 and just remember to add tags for sensible releases in the future.
I've been trying to figure out what to do a bout the misaligned accesses in openspin. They don't matter much now a days as pretty much every modern CPU architecture can handle them. But they do slow down running openspin under Emscripten in the browser an awful lot.
I tried to get Emscripten to tell me where the problems are but have yet to get anything meaningful out of it.
Do you know of anyway to detect misaligned memory accesses, when running on a PC for example?
Cancel that last question. I have detected all the places where alignment errors happen at run time. There are only 6. At least detected so far compiling some large projects.
If I can figure out what is going on I'll try and fix them and submit patches or something.
Heater,
Yeah, I'll likely use tags when I do a binary build for Windows and update the google drive download stuff. I can tag the repo with whatever version that I put in the pdf description and zip download name.
Regarding the alignment errors, if you can send me the locations where it does it, I might be able to fix them more easily than you, since I am more familiar with what's going on. At the very least I can give you some insight into what they are doing, to help you fix them.
I have quoted all the files and lines where I found problems. Sorry the line numbers will be a bit off as things moved around as I instrumented the code.
If you want to see how I did that checkout my fork of OpenSpin https://github.com/ZiCog/OpenSpin and look in the "alignment-issues" branch. Grep for enableAlignCheck and disableAlignCheck. It's quite a challenge to get an x86 machine to make bus errors on misaligned access. Sadly what I did there only works on 32 bit Intel.
Edit: There are a bunch of other places where that instrumentation can generate bus errors. They are all in things like memcpy and strcat so I hope they are all false alarms for some odd reason.
Heater,
So, those look trivial to fix. I'll try to get them addressed in the next few days.
However, there is one that I am not sure how it's an alignment error:
PropellerCompiler.cpp:1412: unsigned char* pObj = &(g_pCompilerData->obj_data[g_pCompilerData->obj_offsets]); // FIXME Alignment error.
obj_data is byte sized, and pObj is byte sized. How is that triggering a misaligned access?
Some looked trivial to fix others looked like I'd need a coffee or two, that one I did not understand at all. As you say it looks totally legit.
I just woke up from a nap with an idea though:
What if it's not the final source and destination that are misaligned but perhaps one of the pointers picked up along the way is. Perhaps g_pCompilerData or obj_data or obj_offsets ?
This could only happen if the compiler is packing data into structs and classes without worrying about alignment. Does GCC do that on x86? I'm not sure.
My plan to tackle that was to split the thing up into a lot of little fetches into temporary variables, if it still trips up we can then see where.
Now, it might be that when compiling for an architecture that is fussy about alignment the compiler lays out structs and classes accordingly and this is not actually a problem.
I suggest you not worry about hat one and I'll check it out. That will have to wait until I get back to the office tomorrow as I don't have 32 bit machine at home.
Like I said my instrumentation picks up misalignment in things like memcpy when called deep inside printf. I think we can say we know for a fact these are false alarms. After all openspin did accurately compile simple programs even before the alignment problem hit.
I believe I can fix all of the other ones in a matter of minutes when I get home tonight (which will be just before you head into the office tomorrow). As you say, they can all be turned into sequence of byte reads with shifts and or's.
Then you can sync up my changes and give it a shot.
I'm pretty sure the g_pCompilerData address will be aligned (most likely to 8 or even 16 bytes, depending on the runtime library memory allocator). Also, I am reasonably sure structures/classes are not being packed tightly by default. maybe it doesn't like the array index being non-integer sized?
OK. I can experiment with your changes and that last legit looking case tomorrow.
I love how I end up working in shifts with people from US or Oz here and there
Speaking of g_pCompilerData there is a lot of stuff allocated to global pointers that is not freed up prior to exit, or at least I thought there was last time I looked. This causes memory leaks when we want to re-run main() repeatedly from a web worker process in a browser. In fact things fail because global data is initialized by the run time before main is run and never gets reinitialized on subsequent runs.
Heater,
Yeah, main was not written to be called multiple times. I think I can clean up some of the more obvious issues pretty easily though, I'll look at it when I am in there tonight. I don't remember having any globally instantiated objects (before main). I do have several that are allocated in main, but never cleaned up (except by exiting the app).
Since I plan to integrate the compiler into PropellerIDE for the "SpinSense" features, I need to address this kind of stuff.
I guess to wrap the thing as a lib it needs an init function or constructor and should clean itself up nicely when done. That would fix the Emscripten issues as well as far as I can tell. Would be nice if the thing can always be built as a statically linked executable though (apart from stdlib stuff).
Looking again I don't see any globally instantiated objects but there are globals initialized to zero, false etc that I thought I saw not being zero or false when running main a second time.
Ah yeah, global base types need initializing inside main (or whatever function is the "main" in the lib).
Yeah, I talked with Steve, and we are axing the DLL idea completely. It'll just be static linked into PropellerIDE. Since it's open source, people can rebuild when they need to update one or the other. No need for DLLs at all. So openspin.exe will stay self contained, but the innards will be rearranged a fair bit.
Heater,
I looked at how you instrumented the code to find misaligned accesses, and I know why you are hitting it for memcpy/strcpy, and other runtime library stuff. Those functions internally do integer sized (or greater depending on the CPU) granularity when doing most of the copy, and then just finish off the copy in the smaller size as needed for the end bits.
Still doesn't explain the hit on the line mentioned above (PropellerCompiler.cpp:1412).
I've fixed up all the other ones (except the one that was a memmove since it's just what I explained above), and also fixed it so that main() now initializes all the statics/globals that were previously only initializing at exe startup. The code is committed, but I didn't do an EXE build yet, since you don't need that, and I want you do try it out first and see what other changes you need. Your usage case is an important one to get working.
Yeah, I figured memcpy and friends were doing something sneaky like that. That's why I excluded them from the issue report. Sorry did one get through? I'm sure it's OK.
I'll have a go with your changes today and see if I can make anything of that odd man out.
Your usage case is an important one to get working.
I'm glad it turns out I'm not wasting your time with all this.
I tested the alignment fixes against my instrumentation on x86 and in the browser from an Emscripten compilation. Everything works so far.
For the Emscripten compile I was able to remove the compilation options for UNALIGNED_MEMORY which means faster code. Running with CHECK_HEAP_ALIGN no longer produces exceptions as it did before.
Not sure if I'm imagining this but your changes seem to have made it so that Chrome does not madly eat all my memory if I repeatedly compile, that is run main(), without restarting my web worker process. Certainly before the Emscripten run time would complain that it had used all the heap I said it was allowed to, it does not do that any more !
Just now I have things set up to recompile endlessly, lets see what happens.
Ooops, spoke too soon, the alignment things seem to be OK but after a bunch of compiles in the same worker process we run out of heap:
Propeller Spin/PASM Compiler 'OpenSpin' (c)2012-2013 Parallax Inc. DBA Parallax Semiconductor.
Compiled on Mar 5 2014 15:02:46
Compiling...
WebServer_W5100_RTC.spin
Cannot enlarge memory arrays. Either (1) compile with -s TOTAL_MEMORY=X with X higher than the current value 134217728, (2) compile with ALLOW_MEMORY_GROWTH which adjusts the size at runtime but prevents some optimizations, or (3) set Module.TOTAL_MEMORY before the program runs.
Not to worry. We can get by with respawning the worker.
Heater,
I just committed another change with a bunch of memory cleanup added. I believe it's now properly cleaning up everything it allocates. Give it a try with your recompile endlessly test.
@Roy: Just got the latest git revision (8). Compiles OK but when run on its own (to show help screen) it dies (after showing it). Is CleanupMemory() perhaps getting confused being called that early?
kuroneko,
I have more changes locally that fix that among other things. What is there, was mostly for Heater to try. I'll get stuff committed soon (tonight or tomorrow night) and do a new binary build as well.
(I have not read the whole thread, sorry if the following has already been reported/solved)
I've built a new main Linux box running Ubuntu 13.10 64 bit. It would not run BST (no 32 bit libs for 13.10) nor Propeller tool under wine, so I wanted to build OpenSpin.
I quickly found out svn was not installed, but that just needed 'sudo apt-get install subversion'
The problem is that the makefile exits with:
make -C PropellerCompiler all
make[1]: Entering directory `/home/bhenning/open-source-spin-compiler-read-only/PropellerCompiler'
g++ -Wall -g -static -o SymbolEngine.o -c SymbolEngine.cpp
SymbolEngine.cpp: In member function ‘SymbolTableEntry* SymbolEngine::FindSymbol(const char*)’:
SymbolEngine.cpp:351:55: error: ‘stricmp’ was not declared in this scope
if (_stricmp(pSymbol->m_data.name, pSymbolName) == 0)
^
SymbolEngine.cpp:364:55: error: ‘stricmp’ was not declared in this scope
if (_stricmp(pSymbol->m_data.name, pSymbolName) == 0)
^
SymbolEngine.cpp:377:54: error: ‘stricmp’ was not declared in this scope
if (stricmp(pSymbol->m_data.name, pSymbolName) == 0)
^
make[1]: *** [SymbolEngine.o] Error 1
make[1]: Leaving directory `/home/bhenning/open-source-spin-compiler-read-only/PropellerCompiler'
make: *** [PropellerCompiler/libopenspin.a] Error 2
Which is strange, because <string.h> is clearly included in SymbolEngine.c
Tracked it down to
#if __GNUC__
// if GCC version is 4.8 or greater use stricmp, else use strcasecmp
//#if __GNUC__ > 4 || (__GNUC__ == 4 && (__GNUC_MINOR__ >= 8 ))
//#define _stricmp stricmp
//#else
#define _stricmp strcasecmp
//#endif
#endif
You can see the changes I hacked in to make it work with g++ 4.8.1
After that, it compiled cleanly - have not tried to use it yet.
Bill,
I couldn't find anything documenting when gcc changed things such that strcasecmp because stricmp. It seemed like it was with the change to 4.8, but now it appears that it must be in one of the minor versions between 4.8.1 and 4.8.4 (I think). I can tweak that version check and get it updated.
However, you'll need to switch over to the github version (see message #331 in this thread) since I won't be updating the google code one anymore. I say this because you mentioned needing subversion, so I assume you grabbed the google code version which uses svn.
jazzed,
GCC changed with 4.8.something to not have strcasecmp anymore and instead have stricmp. I used to have it use strcasecmp instead of stricmp for all GCC versions, but had to make the change because it failed with 4.8.something.
Apparently, some versions of gcc 4.8.1 don't have strcasecmp and some versions do. koruneko had reported that his install of 4.8.1 was failing on the strcasecmp stuff. I ended up installing a newer MinGW with a 4.8.x (can't remember for sure which one, and I'm not at home at the moment) version in it, and that failed also. So I made the change so that it would use strcasecmp for GCC versions prior to 4.8, and use stricmp for 4.8 and newer. Seemed to work for me with old MinGW (gcc 4.6) and new MinGW (gcc 4.8).
If someone can find out what the Smile is going on with gcc and what the correct way to determine if it should be strcasecmp or stricmp, then let me know please. I tried google and failed to find anything useful. It's stupid that stricmp (in some form) is NOT standard.
Heater,
JavaScript can still go suck it. :P It's great for your project, but not for me working on it or for the other projects using it. Thanks though.
Comments
Thanks for the heads up in a PM.
I guess we can live without the change history and if you don't need it that's that. To me change history is only important when I break stuff
You might want to tag the initial commit as v1.0.0 and just remember to add tags for sensible releases in the future.
I've been trying to figure out what to do a bout the misaligned accesses in openspin. They don't matter much now a days as pretty much every modern CPU architecture can handle them. But they do slow down running openspin under Emscripten in the browser an awful lot.
I tried to get Emscripten to tell me where the problems are but have yet to get anything meaningful out of it.
Do you know of anyway to detect misaligned memory accesses, when running on a PC for example?
Cancel that last question. I have detected all the places where alignment errors happen at run time. There are only 6. At least detected so far compiling some large projects.
If I can figure out what is going on I'll try and fix them and submit patches or something.
Yeah, I'll likely use tags when I do a binary build for Windows and update the google drive download stuff. I can tag the repo with whatever version that I put in the pdf description and zip download name.
Regarding the alignment errors, if you can send me the locations where it does it, I might be able to fix them more easily than you, since I am more familiar with what's going on. At the very least I can give you some insight into what they are doing, to help you fix them.
Openspin on github has it's first issue
I have quoted all the files and lines where I found problems. Sorry the line numbers will be a bit off as things moved around as I instrumented the code.
If you want to see how I did that checkout my fork of OpenSpin https://github.com/ZiCog/OpenSpin and look in the "alignment-issues" branch. Grep for enableAlignCheck and disableAlignCheck. It's quite a challenge to get an x86 machine to make bus errors on misaligned access. Sadly what I did there only works on 32 bit Intel.
Edit: There are a bunch of other places where that instrumentation can generate bus errors. They are all in things like memcpy and strcat so I hope they are all false alarms for some odd reason.
So, those look trivial to fix. I'll try to get them addressed in the next few days.
However, there is one that I am not sure how it's an alignment error:
PropellerCompiler.cpp:1412: unsigned char* pObj = &(g_pCompilerData->obj_data[g_pCompilerData->obj_offsets]); // FIXME Alignment error.
obj_data is byte sized, and pObj is byte sized. How is that triggering a misaligned access?
Some looked trivial to fix others looked like I'd need a coffee or two, that one I did not understand at all. As you say it looks totally legit.
I just woke up from a nap with an idea though:
What if it's not the final source and destination that are misaligned but perhaps one of the pointers picked up along the way is. Perhaps g_pCompilerData or obj_data or obj_offsets ?
This could only happen if the compiler is packing data into structs and classes without worrying about alignment. Does GCC do that on x86? I'm not sure.
My plan to tackle that was to split the thing up into a lot of little fetches into temporary variables, if it still trips up we can then see where.
Now, it might be that when compiling for an architecture that is fussy about alignment the compiler lays out structs and classes accordingly and this is not actually a problem.
I suggest you not worry about hat one and I'll check it out. That will have to wait until I get back to the office tomorrow as I don't have 32 bit machine at home.
Like I said my instrumentation picks up misalignment in things like memcpy when called deep inside printf. I think we can say we know for a fact these are false alarms. After all openspin did accurately compile simple programs even before the alignment problem hit.
Then you can sync up my changes and give it a shot.
I'm pretty sure the g_pCompilerData address will be aligned (most likely to 8 or even 16 bytes, depending on the runtime library memory allocator). Also, I am reasonably sure structures/classes are not being packed tightly by default. maybe it doesn't like the array index being non-integer sized?
OK. I can experiment with your changes and that last legit looking case tomorrow.
I love how I end up working in shifts with people from US or Oz here and there
Speaking of g_pCompilerData there is a lot of stuff allocated to global pointers that is not freed up prior to exit, or at least I thought there was last time I looked. This causes memory leaks when we want to re-run main() repeatedly from a web worker process in a browser. In fact things fail because global data is initialized by the run time before main is run and never gets reinitialized on subsequent runs.
But I think those are fish for another day.
Yeah, main was not written to be called multiple times. I think I can clean up some of the more obvious issues pretty easily though, I'll look at it when I am in there tonight. I don't remember having any globally instantiated objects (before main). I do have several that are allocated in main, but never cleaned up (except by exiting the app).
Since I plan to integrate the compiler into PropellerIDE for the "SpinSense" features, I need to address this kind of stuff.
I guess to wrap the thing as a lib it needs an init function or constructor and should clean itself up nicely when done. That would fix the Emscripten issues as well as far as I can tell. Would be nice if the thing can always be built as a statically linked executable though (apart from stdlib stuff).
Looking again I don't see any globally instantiated objects but there are globals initialized to zero, false etc that I thought I saw not being zero or false when running main a second time.
Anyway, no hurry for any of that.
Yeah, I talked with Steve, and we are axing the DLL idea completely. It'll just be static linked into PropellerIDE. Since it's open source, people can rebuild when they need to update one or the other. No need for DLLs at all. So openspin.exe will stay self contained, but the innards will be rearranged a fair bit.
I looked at how you instrumented the code to find misaligned accesses, and I know why you are hitting it for memcpy/strcpy, and other runtime library stuff. Those functions internally do integer sized (or greater depending on the CPU) granularity when doing most of the copy, and then just finish off the copy in the smaller size as needed for the end bits.
Still doesn't explain the hit on the line mentioned above (PropellerCompiler.cpp:1412).
I've fixed up all the other ones (except the one that was a memmove since it's just what I explained above), and also fixed it so that main() now initializes all the statics/globals that were previously only initializing at exe startup. The code is committed, but I didn't do an EXE build yet, since you don't need that, and I want you do try it out first and see what other changes you need. Your usage case is an important one to get working.
Yeah, I figured memcpy and friends were doing something sneaky like that. That's why I excluded them from the issue report. Sorry did one get through? I'm sure it's OK.
I'll have a go with your changes today and see if I can make anything of that odd man out.
I'm glad it turns out I'm not wasting your time with all this.
I tested the alignment fixes against my instrumentation on x86 and in the browser from an Emscripten compilation. Everything works so far.
For the Emscripten compile I was able to remove the compilation options for UNALIGNED_MEMORY which means faster code. Running with CHECK_HEAP_ALIGN no longer produces exceptions as it did before.
Not sure if I'm imagining this but your changes seem to have made it so that Chrome does not madly eat all my memory if I repeatedly compile, that is run main(), without restarting my web worker process. Certainly before the Emscripten run time would complain that it had used all the heap I said it was allowed to, it does not do that any more !
Just now I have things set up to recompile endlessly, lets see what happens.
All in all it's looking good.
Not to worry. We can get by with respawning the worker.
I just committed another change with a bunch of memory cleanup added. I believe it's now properly cleaning up everything it allocates. Give it a try with your recompile endlessly test.
I have more changes locally that fix that among other things. What is there, was mostly for Heater to try. I'll get stuff committed soon (tonight or tomorrow night) and do a new binary build as well.
Thanks for reporting though!
(I have not read the whole thread, sorry if the following has already been reported/solved)
I've built a new main Linux box running Ubuntu 13.10 64 bit. It would not run BST (no 32 bit libs for 13.10) nor Propeller tool under wine, so I wanted to build OpenSpin.
I quickly found out svn was not installed, but that just needed 'sudo apt-get install subversion'
The problem is that the makefile exits with:
make -C PropellerCompiler all make[1]: Entering directory `/home/bhenning/open-source-spin-compiler-read-only/PropellerCompiler' g++ -Wall -g -static -o SymbolEngine.o -c SymbolEngine.cpp SymbolEngine.cpp: In member function ‘SymbolTableEntry* SymbolEngine::FindSymbol(const char*)’: SymbolEngine.cpp:351:55: error: ‘stricmp’ was not declared in this scope if (_stricmp(pSymbol->m_data.name, pSymbolName) == 0) ^ SymbolEngine.cpp:364:55: error: ‘stricmp’ was not declared in this scope if (_stricmp(pSymbol->m_data.name, pSymbolName) == 0) ^ SymbolEngine.cpp:377:54: error: ‘stricmp’ was not declared in this scope if (stricmp(pSymbol->m_data.name, pSymbolName) == 0) ^ make[1]: *** [SymbolEngine.o] Error 1 make[1]: Leaving directory `/home/bhenning/open-source-spin-compiler-read-only/PropellerCompiler' make: *** [PropellerCompiler/libopenspin.a] Error 2Which is strange, because <string.h> is clearly included in SymbolEngine.c
Tracked it down to
You can see the changes I hacked in to make it work with g++ 4.8.1
After that, it compiled cleanly - have not tried to use it yet.
I couldn't find anything documenting when gcc changed things such that strcasecmp because stricmp. It seemed like it was with the change to 4.8, but now it appears that it must be in one of the minor versions between 4.8.1 and 4.8.4 (I think). I can tweak that version check and get it updated.
However, you'll need to switch over to the github version (see message #331 in this thread) since I won't be updating the google code one anymore. I say this because you mentioned needing subversion, so I assume you grabbed the google code version which uses svn.
Roy
I recommend adding this to Makefiles CFLAGS += -Dstricmp=strcasecmp
GCC changed with 4.8.something to not have strcasecmp anymore and instead have stricmp. I used to have it use strcasecmp instead of stricmp for all GCC versions, but had to make the change because it failed with 4.8.something.
Apparently, some versions of gcc 4.8.1 don't have strcasecmp and some versions do. koruneko had reported that his install of 4.8.1 was failing on the strcasecmp stuff. I ended up installing a newer MinGW with a 4.8.x (can't remember for sure which one, and I'm not at home at the moment) version in it, and that failed also. So I made the change so that it would use strcasecmp for GCC versions prior to 4.8, and use stricmp for 4.8 and newer. Seemed to work for me with old MinGW (gcc 4.6) and new MinGW (gcc 4.8).
If someone can find out what the Smile is going on with gcc and what the correct way to determine if it should be strcasecmp or stricmp, then let me know please. I tried google and failed to find anything useful. It's stupid that stricmp (in some form) is NOT standard.
Edit: I think we can resolve this in the Makefile. My mingw make works either way.
What Mingw version failed?
svn diff Index: Makefile =================================================================== --- Makefile (revision 69) +++ Makefile (working copy) @@ -6,6 +6,9 @@ else CFLAGS+=-Wall -g -static endif +ifeq ($(OS),Linux) + CFLAGS += -Dstricmp=strcasecmp +endif CXXFLAGS += $(CFLAGS) TARGET=openspin Index: PropellerCompiler/Makefile =================================================================== --- PropellerCompiler/Makefile (revision 69) +++ PropellerCompiler/Makefile (working copy) @@ -6,6 +6,9 @@ else CFLAGS+=-Wall -g -static endif +ifeq ($(OS),Linux) + CFLAGS += -Dstricmp=strcasecmp +endif CXXFLAGS += $(CFLAGS)It needs to work across windows for both not GCC and with GCC, then linux and mac. Why can't GCC be consistent across platforms? WTF?
I'm getting a core dump on linux from the GIT repository. Windows also complains. No problem with the google-code repo.
The stricmp diffs I provided are the best solution to date. Complaining about lack of strcasecmp on windows never got me anywhere
The issue isn't windows VC++, it's gcc being different depending on platform AND versions of gcc.
Openspin.js has been recompiling all weekend. It's solid and leak free. Great stuff.
JavaScript can still go suck it. :P It's great for your project, but not for me working on it or for the other projects using it. Thanks though.