Out of Memory! Help!
MoZak18
Posts: 26
I have seen related topics discussed several times before, but I have not yet seen a topic that discusses this particularly, at least, not simply enough for my level. I am building a project with multiple sensors (temperature, accelerometer, pressure sensor, GPS, SD card writing) and as a result, I have a main calling A LOT of objects, and those objects calling A LOT of objects..........as a result, I'm out of memory to the point where some sensors are not initializing or working at all. I have pressed F8 to see the status of my memory, it's down to about 800 longs. I believe when it gets down to about 1400 longs it will consider itself "out of memory." I have read on another thread that this is actually a glitch on the Propeller Tool. Regardless, I am using the 24LC256 EEPROM as recommended in the Propeller Manual. Here are my questions:
1. When I press F8, I notice that my total memory size is only about 8000 longs. I read in the Propeller Manual that I should have about 1600 longs (due to having 64 kb of total memory, 32 RAM on the chip, and 32 ROM from the EEPROM). This is true even if I load the code onto the EEPROM (pressing F11 instead of F10). Why is my memory only 8000 longs? Is there any way to get it up to 1600 longs and will this make a difference?
2. Is there a larger EEPROM I can get as compared to the 24LC256? I do not quite understand the lingo of EEPROMs, sometimes I read that the 24LC256 has 32kb of memory, other times they say 256kb. Some people say that 128kb EEPROMs are bigger. Is there anything bigger than the 24LC256 that I can connect to the Propeller and will this help me run my code?
3. All in all, what can i do to run my code?? It seems that it's taking up just about 8000 longs at the moment. Based on what I plan to add, this should go up to around 10000-11000 longs. What can I do overall to run code of this size? (kind of a repeat and sum-up of the previous 2 questions).
Again, thank you for any and all responses. I really do appreciate it.
1. When I press F8, I notice that my total memory size is only about 8000 longs. I read in the Propeller Manual that I should have about 1600 longs (due to having 64 kb of total memory, 32 RAM on the chip, and 32 ROM from the EEPROM). This is true even if I load the code onto the EEPROM (pressing F11 instead of F10). Why is my memory only 8000 longs? Is there any way to get it up to 1600 longs and will this make a difference?
2. Is there a larger EEPROM I can get as compared to the 24LC256? I do not quite understand the lingo of EEPROMs, sometimes I read that the 24LC256 has 32kb of memory, other times they say 256kb. Some people say that 128kb EEPROMs are bigger. Is there anything bigger than the 24LC256 that I can connect to the Propeller and will this help me run my code?
3. All in all, what can i do to run my code?? It seems that it's taking up just about 8000 longs at the moment. Based on what I plan to add, this should go up to around 10000-11000 longs. What can I do overall to run code of this size? (kind of a repeat and sum-up of the previous 2 questions).
Again, thank you for any and all responses. I really do appreciate it.
Comments
The number behind the EEPROM type is given in kb. Also available are 512kb and 1024kb.
But this does not change the fact that the propeller has 32kB of HUB-RAM to store your program/data/stack - which is 8000 longs. Regardless which EEPROM you have it will always read the 32kB and no bit more than that! Nevertheless there are tricks how to make better use of the HUB-RAM in combination with bigger EEPROMs and/or SD cards.
Most time you start the COG which deals with sensors once on boot-time. So, the code of these COGs is transferred to COG-RAM and is running while the code in HUB-RAM is no longer needed.
One way is to store the COG-code in EEPROM or on SD-card. You main loads the code for the first COG into a buffer and starts the COG, then it loads the code for the second COG and starts it .....
Another way is that the program booted only loads all COGs doing all needed initializations and then reloads another 32kB image which runs the main-program.
Maybe you can give a list of drivers/objects you use.
Now, you have some options as suggested above. The simplest for now is to download bst and use it. Search the forum for a link (IIRC the link to Tools in my signature should take you to a list where you will find bst. It has an IDE too. Check the option to remove unused code and that should free up some hub memory.
After you have exhausted this space, then you will need to look at reusing the space occupied by the pasm code loaded into the cogs. Currently there isnt a really simple way to load the cogs... there is work being done in this area by a number of us, but nothing simple yet.
With all the I/O you probably don't have any spare pins to run external memory.
Probably it would be best to explain what you are doing and what objects you are using so we can make suggestions to trim your code.
Look at the arrays in the VAR sections. Each time an obect is referenced by another object there will be a dedicated copy of the VAR section for that object. Some arrays may not have to be in a VAR section, and could be moved to a DAT section to reduce the amount of memory needed.
Using BST is the easiest way to save memory. With the optimizations enabled I have saved around 3000 on many of my large programs.
If you post your code in a ZIP file we could give you more specific suggestions.
You might be able to get some ideas from http://obex.parallax.com/objects/411/
Best of Luck.
I will delete my suggestion about the stack.
Measuring stack usage is commonly done by seeing how much of the stack space gets overwritten. That doesn't tell you the worst case figure, but propstacklen does.
If you want, you can supply your code by posting it here or via a PM to me and I'll give propstacklen an outing on it. I'd need the full code, ie including the code for all referenced objects.
Work on propstacklen is currently suspended due to lack of responses, but if interest revives I'll restart it. There's more work needed on the reporting of the effects of recursion, and more test cases will almost undoubtedly reveal stuff that needs fixing. However, to the best of my belief the reports it generates are accurate.
@Dave Hein: Thanks to your spinsim program, propstacklen (while it is not a simulator) was made easier to develop. I've acknowledged this in the code but I wanted to say thanks again here.
And I would not bother with recursive calls too much, because if you do recursive calls you should rather switch to a PC ;o) You have to know that each recursion can be resolved with a loop as well and that's the prefered solution for our beloved prop, which easily runs out of stack (HUB-RAM) when using excessive recursions. A loop usually needs less HUB-RAM.
I agree with you: most projects have plenty of memory available for stack space - though the OP of this thread clearly has a counter-example, which is why I chipped in.
As for recursion, I have in the past done a fair bit of work with embedded systems such as telephone exchange switches where recursion is absolutely forbidden because of its unpredictable stack requirements. And as you say, it's well known that recursion can always be replaced with iteration. Still, it's a fact that people *are* using recursion with the prop and it's also a fact that a program such as propstacklen can be made to deliver a lot more information about the implications than propstacklen currently does. So if there's enough general interest in the program, recursion is an avenue I'd still like to pursue.
And after all, even if your program avoids recursion and has minimal stack requirements generally, what other easy way do you have of finding out whether any library objects you may have used employ recursion internally, or otherwise demand large amounts of stack?
@Dave Hein:
Please do try it out. And if it breaks, show me the code that broke it and I'll respond.
Just for pointing that out clearly: I don't argue against Propstacklen. It should be added to the propeller tool and should be automatically run during/after compile! I just wanted to give a possible explaination why there is only little feedback so far.
I think recursions are a pain for such a program. If you want to do a real analysis, you have to simulate the propeller. Each recursion needs an end-condition, but this can be anything - like
byte[ end_flag_ptr ] := 1
Or maybe a PASM COG sets the end condition ...
And recursions often work with variable input-data. So, you have to simulate the recursion with a set of different inputs...
So, "I'd not bother with recursions" means that the tool reports if a recusion has been detected - at least. Maybe it also reports how much memory is needed on the stack for a single recursion - and at best it tells you how much recursions are possible with the given stack-size. The rest has to be done by the programmer.
johnfos: "OP of this thread clearly has a counter-example, which is why I chipped in"
MoZak18: " about 8000 longs at the moment. Based on what I plan to add, this should go up to around 10000-11000"
I don't think that the OP has >25% of the HUB-RAM being used as stack-space. So, propstacklen might help to squeeze out some longs, but it relatively sure won't solve the whole problem.
@MoZak18:
Any news? Did you solve the problem? Tried some of the tips?
Thanks for the clarification - we're obviously of the same mind here.
Magi02: "So, "I'd not bother with recursions" means that the tool reports if a recusion has been detected - at least. Maybe it also reports how much memory is needed on the stack for a single recursion - and at best it tells you how much recursions are possible with the given stack-size. The rest has to be done by the programmer."
Yes, that's exactly what I have in mind. At the moment propstacklen detects and reports the existence of recursion. It should (and maybe one day will) then tell you what a single recursion will cost for each possible recursion sequence. And as you so rightly say, after that it's up to the programmer.
I guess I should take some time to clean propstacklen up and document it properly, and then submit it to the Obex. I may be some time...
@MagIO2: I went through a lot of the "libraries" I have and was able to cut out some code I wasn't using to save quite a bit of memory. I have had the Basic Stamp editor for a while, but I looked through it and couldn't figure out how to use it to locate unused code. Although I was able to trim away some of the fat, it wasn't enough to run all my code.....so for the project we have simply decided to use 2 Props each with their own EEPROMs, which solved the problem. At my level I wouldn't be able to get too creative!
Thanks again
It has a flag in the compiler preferences to not add all unused methods to the propeller binary. So, no manual code changes needed for that.
I think it's available on the propeller download page .. if not someone will certainly provide a link here.
MagIO2
I thought Ackermann's function could only be calculated recursively. But I guess we don't have much use for that.
Yep, I have seen a lot of that to. However...
This may be true, apart from some extreme functions, but often you might find yourself replacing the processors built in stack, as managed by your programming language, with a hand made stack of data within your code.
For example if you have to parse some XML or something with nested structures you could build a recursive decent parser where the regular stack is used or you could do it in a loop and save things on a hand coded stack as you go.
Either way you have a stack whose potential for excessive growth needs to be taken care of and handled safely.
So I conclude that such bans on recursion are somewhat misguided. Managing such a hand made stack may be more probe to error than just letting the compiler take care of it and monitoring your recursion depth.
On the other hand, I have worked on projects in aerospace where loops were absolutely forbidden in you source code because of their
unpredictable execution time requirements. In fact the language used, Lucol, did not support any way to make a loop. Your code was all event driven and had to complete in a timely manner.
Read about Lucol here: http://www.springerlink.com/content/m7p138ln8k178524/
Download it, install it and initially, just try compiling your code with BST instead of the prop tool.
Then go looking for that checkbox option MagIO2 says. It is in the tools/options or similar.
I use BST about half the time and the Proptool the other half. Main disadvantage of BST is there are not those little arrows which tell you the nesting of loops etc.
If you can optimise out all the unused code that will usually buy you some space. Though code does have a habit of growing to fit the available space, plus just a bit more for good measure.
You may well be able to get it back inside one propeller chip. There is always a cunning way to save some space.
Next step if you run out of memory again might be to look at C running in "XMM" mode where the program is run from external memory. Only problem with C is that not all the Spin objects have been ported over to C yet.
"There are some types of problems whose solutions are inherently recursive, because of prior state they need to track. One example is tree traversal; others include the Ackermann function, depth-first search, and divide-and-conquer algorithms such as Quicksort. All of these algorithms can be implemented iteratively with the help of an explicit stack, but the programmer effort involved in managing the stack, and the complexity of the resulting program, arguably outweigh any advantages of the iterative solution." found @ http://en.wikipedia.org/wiki/Recursion_(computer_science)
With a quick lookup of Ackermann I guess all you need to store on a stack (or array) is m, which is really easy. What you'd save then in an iterative implementation is having function-calls that additionally have to store return-address + both parameters (m, n) + return values on the stack. I think here I have found a task for this evenings programming session ;o)
"So I conclude that such bans on recursion are somewhat misguided. Managing such a hand made stack may be more probe to error than just letting the compiler take care of it and monitoring your recursion depth."
Which compiler do you use? My propeller tool did not give any warning about the recursion depth so far!
So, I conclude that for the expected problem domains we find on the propeller a ban of recursions is still valid - most times. But I'm not into any religion, so in the end it's only something you should think about.
"Read about Lucol here:" - nothing worth to read there without payment.