Optimizing EEPROM and RAM space

Mooneyguy · 2011-12-31 06:56

As the program that I am working on is evolving, it is also growing in size. Yesterday I started noticing strange behaviors in the program and it was not running like it used to. For example, one line of code that worked fine all along that grabs a reading from a subroutine, started to require a pause time of 250 to 1000 ms or else it would return a zero, and later, it would not work with any pause time. Then I noticed a value written to the EEPROM location 20 started returning a strange large number after returning the correct number for many runs previously. When I rewrote the value to store in the EEPROM location 20, it would be there for awhile but then when I booted the program (power down and then power up) a large, strange number would return instead of the number I put there.

I then checked my EEPROM and RAM Map space and found the map visually it looked pretty full (not sure how to quantify it other than there was less than about a row and a half of empty boxes). I created some space in the map by deleting some trouble shooting lines of code and then the program would behave normal again. The maps now look like the attachment and the program runs normal.

My problem is that I would like to actually do a few more things with this program and interface another sensor but don't know if I have the space needed.

My program is currently very heavily commented so I could remember what I did and my variable names are also quite long and descriptive. Does this take as much space as program command lines do? Is program space treated the same if I type a command, put a space between a word. For example, is every keyboard peck take up equal space in the program? I also have about 15 to 20 DEBUG lines of code.

What takes up most of the space? Is it long variable names? Is it white space (empty lines of code)? Comments? Or are they all treated the same when it comes to the map space

When I write a Word value to the EEPROM location 20, where is it located in the map attached? Can I move it to a different location so it is not as likely to get corrupted?

Is there a Stamp available with more memory than my BS2 Homework Board? I am just trying to understand my programming limitations.

Thank you in advance for the help.

Mike Green · 2011-12-31 12:23

Comments and spaces are removed from the program when it's compiled and downloaded to the EEPROM. Variable names and constant names are removed as well although DEBUG statements with "?" cause the variable name to be placed in the program as a string constant so it can be displayed. Statements are simplified and everything is encoded so it uses as little space as possible. If you are using the EEPROM for data storage (using DATA, READ, and WRITE), that uses the same EEPROM space as the program with the program stored from the "top" of the EEPROM downwards and data stored by convention from the "bottom" upwards. There is no checking whether you're writing data into space used by the program, so, if you have a large program, you will have to check the memory map carefully. The data areas don't show up in the map unless you define them using a DATA statement.

There are Stamps with more EEPROM, but they're organized into 2K "slots". A program in one slot can jump to more program in another slot, but there's no easy way to return to a place in a "calling" slot. It is fairly easy to use different slots, one or more for program and one or more for data. The BS2p/pe/px are particularly good at this and the BS2pe is specifically designed for datalogging and has 8 x 2K slots for just data. The BASIC Stamp Syntax and Reference Manual has more information on this and there's a Nuts and Volts Column specifically on the subject of multi-slot programming.

Paul Romsky · 2011-12-31 19:01

I agree with the moderator, there must be a piece of "rogue code" in your program that is "stepping" on a memory location other than the one that it was intended to write. With PBasic, like most low level compilers (C for an example) the compiler does not check for code that "oversteps" its boundries when the code is compiled (Compile Time). It is only when the code is running (Run Time) when the problem manifests itself. There is a reason for this, but it is rather complicated to explain. It is up to the programmer to carefully "walk" through the code and find that piece of rogue code that is causing the problem. A tool called a Debugger (not to be confused with the PBasic DEBUG command) could help, but there is no Debugger for PBasic (that I know of). Debuggers are great because you can step through the code and watch all of your variables to see what goes wrong. If you want, you can send me your code and I will take a look. In my 30+ years in programming, I have found that sometimes it requires a "fresh" set of eyes to find the problem, our brains tend to filter out obvious errors in our own code.

I can be reached at romsk22@gmail.com if you want me to take a look at your code.

Paul Romsky · 2011-12-31 19:17

Mike,

In my BS1 and BS2's programs, I find that all the text srrings for the DEBUG commands tie up a lot of the EEPROM. This makes sense because the compiler does not optimize strings. I use a lot of techniques to reduce the amount of redundant strings, but the best way I can think of does not seem to be possible.

Do you think Parallax could add a feature like "Label Addressing of Strings"?

Example:
BS2 (code fragment):

String1:
DATA @0, "This is string one that is used many, many times, in many different points in a program", 0
String2:
DATA "This is another string that is used", 0
String3:
DATA "And... so on", 0

'
' It would be nice if DEBUG could ALSO refer to the strings by their addresses (labels)
' Right now PBasic does not allow this, it only allows RAM Byte Arrays for the DEBUG STR command (and as you know, there is not much RAM in a BS2).
' Expanding this to subsitute an address from the EEPROM could save a lot a space in DEBUG rich programs that use a lot of the same sub strings over and over again.
'
DEBUG STR String1 ' Not allowed in PBasic 2.5
DEBUG STR String2 ' Not allowed in PBasic 2.5
DEBUG STR String3 ' Not allowed in PBasic 2.5

End Example:

Your thoughts?

Paul

Paul Romsky · 2011-12-31 19:36

Mooneyguy,

Whitespace (spaces, new lines, tabs) between commands, lines, and pretty much any non-quoted text ("This is quoted text" This is non-quoted text) does not take up any run-time memory.

The PBasic Complier (like almost all compliers) knows that these white spaces are for we "Humans" to help us read our code, the compiler simply ignores white space as it "boils down" your code. So remember this, Whitespace is your Friend! Use it all you want in your source code to make your code readable.

Comments are similar, the compiler knows that these are for the humans only and just ignores them. The only compiler that I can think of that takes up memory space from comments is my vintage 1982 Sinclair ZX81. It had a whopping 1K of RAM space of Line Addressed BASIC, and comments took up RAM because it did not use a compiler - the code (and comments) in RAM were interpreted on the fly (not tokenized like the PBasic code).

Paul

Mooneyguy · 2012-01-01 00:11

Thanks everyone for the great replies. Is there a way to find where my data is stored in the map when I write to location 20? If the program is stored from the "top" of the EEPROM downwards and data stored by convention from the "bottom" upwards, how do I read the map to know where location 20 is and if it is stored in the middle of the program?

Paul Romsky · 2012-01-01 06:51

Mooneyguy,

If you are storing a value at address 20 (which is 14 in hex, that is $14 in PBasic), then the Memory Map (which shows addresses in hex) should highlight locaton 014 in blue.

Post your code if you can and I will take a look. I am not sure what is going on and it is hard to tell without having some code in front of me.

I am a hardware engineer by profession so the terms Bottom and Top of memory always seems to cause confusion among hardware/software people.

So to make sure we are on the same page (no pun intended), here is how I "visualize" PStamp memory:

All values are in Hex

Bottom-Up:
Adresss 7FF Top of Memory
.
.
.
Address 000 Bottom of Memory

In PBasic:
Address 000 Bottom of Memory
.
.
.
Address 7FF Top of Memory

The PBasic method to "visualize" memory is quite common, but Top is at the bottom and the Bottom is on the top. So, Bottom ALWAYS means the lowest address which is address 0 (or 000 in this case).

So, address 20 ($14) is low in memory (near the bottom) which is near the upper left of the PBasic memory map.

You may already agree with this, but I have to make sure we "talk" memory in the same fashion.

I will be on and off the web all day today... my wife has many chores for me [smile].
Ask any questions you want and I will try to get back to you as promptly as possible.

Paul

Paul Romsky · 2012-01-01 07:11

Mooneyguy,

Try this and see if it helps you understand the memory arrangement.

' {$STAMP BS2}
' {$PBASIC 2.5}
' { PORT COM8}
'
' Memory.bs2
' Paul Romsky
' 01 JAN 2012
'
' Forum Discussion File for Memory Related Questions.
'
' This module provides the control code for the Parallax BS2 Pstamp
' Microcontroller. It serves no purpose other then to provide source
' code to demonstrate techniques and to explore possibilities.
'
'
' Revision History
'
' Ver Date Author Description
' ---

' 1.0 01 JAN 2012 P. Romsky Initial coding
'
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

'
' EEPROM Data
'
DATA @20, 5

'
' Variables
'
Arg1 VAR Byte

'
' Main Entry Point:
'
Main:
'
' No need to run this, just complie the code and look at the memory map.
' Scroll the window UP towrds address 000 (BOTTOM of Memory). Near
' the Bottom (memory bottom) you should see that addrss 014 (which is 20 in decimal)
' is blue. The code is at the TOP of memory (scroll the window DOWN towards
' address 7FF).
'
DO
READ 20, Arg1
DEBUG "Arg1=", DEC Arg1, CR
LOOP

'
' End
'

Mooneyguy · 2012-01-01 21:55

Thanks for the great explanation Paul! I have not yet used the data command. I just Write data into an EEPROM location and then Read it back. Here is that part of the code:

FOR Adjust_cycles = 0 TO 40 ' No. of Adjust_cycles
PAUSE 500 ' and pause time determines max allowed run time (pause 1000 = 1 sec). Seems like Pause must >100 or 200.
HIGH 7 ' Turn on device.
IF (IN10 = 1) THEN ' If P10 is high (1) due to push button, then we intend to reset EEPROMmvSet to new desired mV output.
DEBUG ? IN10
GOSUB GetADC ' Get mv output from analog to digital converter
GOSUB DisplayADC ' Display outputs from ADC on debug terminal
READ 20, Word EEPROMmVSet ' Retrieves the value stored in EEPROM location 20 and copy it into VAR EEPROMmVSet.
' Then checks new proposed reading to see it is not equal to new set so we don't rewrite
' same values for longer EEPROM life.
IF EEPROMmVSet > mVoltsOutput THEN WRITE 20, Word mVoltsOutput ' Write new set point in EEPROM location 20 (a Word takes 2 locations).
IF EEPROMmVSet < mVoltsOutput THEN WRITE 20, Word mVoltsOutput ' Write the VAR Word mVoltsOutput to EEPROM location 20 if needed.

Using this scheme I don't see the location 20 as in your demo. When I input the code: DATA @20, (2) I do see a reserved spot for my Word data. However, I don't think what I am currently doing puts the data in that same location. The data that I Write to 20 stays there after power is cycled but the data point can be written over by code which I would like to avoid.

Paul Romsky · 2012-01-02 15:25

Mooneyguy,

It is all my pleasure to help fellow engineers! I will look at your code and give you some pointers (no pun intended). I'll write back soon.

Don't be afraid to ask questions, that is what we (and the the forum) are here for... to get you going so you can take things to a new level and to help others along that journey.

Paul

Paul Romsky · 2012-01-02 16:29

Mooneyguy,

Yes, the DATA command was just an experiment to show you (at compile time) where your WRITE 20 data is being stored.

Your code fragment looks sound, and the WRITE and READ instructions are indeed using address 20 in the EEPROM (near the Bottom of Memory - Address 010 in the Upper Left of the map). But read on....

If your code is large enough to be filling in the EEPROM down to address 20 (014hex) and 21 (015hex) - which are on the 2nd line of the Map , then your code is stepping on this value in EEPROM (rogue self-modofing code), and that would explain the odd behavior. If your code ends short of these addresses, could it be that you are elsewhere in your code doing a Word WRITE to address 19 (like WRITE 19, Word Variable)?

The "Word" Option allows READ/WRITE to save values from 0 to 65535 in EEPROM but remember that will require TWO bytes of space.

Example

WRITE 20, Word 1024 ' 00(hex) is written into Address 20, and 04(hex) is written into Address 21 (Little Endian)
.
.
.
WRITE 19, Word 65535 ' FF(hex) is written into Address 19 and FF(hex) is written into Address 20 (Partially stepping on your Word stored at Address 20 and 21)

Sidebar question for you: Do you understand what "Endianess" is?

Could you WRITE/READ mVoltsOutput to a lower EEPROM address (like 0 for instance)?

Let me know how you are doing. If we can fix this problem then we can go into techniques to reduce your code, but right now it looks like WRITEs are stepping on each other (what we call Not Properly Aligned data).

Paul

erco · 2012-01-03 14:16

Great info in this very helpful thread. It sounds like Mooneyguy is packing the BS2 quite full for maximum performance. I'm all for the "push it to the limit" with the Stamps. Most everyone says "get a Prop", but Paul is the rare bird that digs in and offers this level of help with the BS2. Great job all around, guys.

Paul Romsky · 2012-01-03 16:46

erco,

Thank you for the complement.

I first found out about PStamps about 5 years ago when a UMass senior needed some help with his project. I was amazed how easy Parallax made it to get something working in minutes. I ordered my own BOE USB that night. Now every time I need to control or monitor something in the lab I turn to my trusty BOE.

I was thinking about a getting a Propeller system because I could use some concurrent processes now and then. I am used to coding FPGAs in VHDL (where everything is concurrent), so it makes sense to start using a propeller.

Anyway, no word yet from mooneyguy on how its going..... KPA441 10 10 on the side.

Romsk

Mooneyguy · 2012-01-10 14:18

Thanks again Paul. I found that using a lot of Debug statements, like Debug "Display millivolts", DEC millivoltsoutput, Cr, Cr, uses a lot of space. When my Debugs are "commented out" 25% or 30% of the space came back so that problem has been temporarily resolved. My problem now is that I have no RAM memory left for defining variables. In fact, I had to copy and paste over and over what I wanted my last Do Loop to perform, which looks kind of goofy, but it does work. I was thinking about saving one of my variable values in the EEPROM before entering the loop, using the variable in a loop, and then resetting the variable back to what it once was before it went into the loop. Seems like that might work. I have just enough RAM variables now to make it run but would like to double my RAM space if possible.

I had to look it up to learn that the term endian or endianness refers to the ordering of individually addressable sub-components within the representation of a larger data item as stored in external memory :-)

Also not sure what KPA441 1010 on the side is.

Mike Green · 2012-01-10 15:18

Yes, DEBUG statements take a lot of space. It helps to abbreviate text.

Look at the section of the Basic Stamp Manual on variable aliases. That's where you declare a new variable that occupies the same space as another variable when the variables are never used in the same sections of your program.

Paul Romsky · 2012-01-10 19:49

Mooneyguy,

Mike Green has very sage advice. But you have to manage your variables (you seem to be doing this) as to when variables can be reused even if aliases are used - the compiler (tokenizer) does not do this for you
.
As you know, using the EEPROM for variable storage is taboo because of the limted number of writes that can be made to it before the memory cells wear out and become "flakey".

I think Parallax has a serial RAM you could add for more variable space, I am sure Digikey has some. It might be worth looking at this as an option.
The more advanced BS2's have a scratchpad RAM that can be used - like the BS2px (my favorite PStamp).
Go to "Help" in the BASIC Stamp Editor and select "BASIC Stamp Hardware" for a handy chart on each PStamp's features.

Also look through your code and see if you are using:
A byte that will never have a value above 15 - this will fit in a nibble (and free up 4 bits for another variable).
A word that will never go above 255 - this will fit in a byte (and free up 8 bits)
Or any other permutations of Bits, Nibbles, Bytes, and Words that can be re-organized to save space.

BTW...

That was Citizens' Band (CB) radio talk:

KPA441 is my call sign
1010 is code for standing by
On the side means just listening

Paul

Mooneyguy · 2012-01-14 09:25

Thank you both Mike and Paul. Both with great ideas and advice for a novice like me. I will definitely look into alias' and additional RAM. You both have been a big help.