DEMO: Self Modify Spin Code Example
Beau Schwabe
Posts: 6,568
I was inspired by this thread...
http://forums.parallax.com/showthread.php?t=126887
... Phil's post got me thinking about something that I had stumbled across but didn't quite comprehend.
What this Example does is basically allow you define "HEX code snips" (<-- interpreter code) in the DAT section and call them as if they were originally defined as a PUB/PRI routine.
It's kind of funny, because the heart of this Example only uses two lines of code, and that's to show two separate program snips that are residing in the DAT section...
Main PUB pointer magic
DAT section
There is enough information in the Example, that you could directly read portions of memory as 'PUB snips' and store that data on a larger external memory for future recollection and execution as a LMM (Large Memory Module) environment since it's relatively easy to transfer data externally to a defined section in the DAT block.
For those that want to go further, it is possible to decode the HEX and break it down into an "OP code for the interpreter" format and a pseudo-human readable instruction by instruction level but anyone who has gone that route knows that it a tedious adventure. Also very rewarding when you begin to unravel and make those ah ha connections.
code History - below
http://forums.parallax.com/showthread.php?t=126887
... Phil's post got me thinking about something that I had stumbled across but didn't quite comprehend.
What this Example does is basically allow you define "HEX code snips" (<-- interpreter code) in the DAT section and call them as if they were originally defined as a PUB/PRI routine.
It's kind of funny, because the heart of this Example only uses two lines of code, and that's to show two separate program snips that are residing in the DAT section...
Main PUB pointer magic
PUB Loader '' <- Change PUB pointer to point directly to DAT memory as if DAT was a PUB
byte[$18] := @Code - $10
byte[$1C] := @KitCar - $10
{{ 
Note: First PUB/PRI is always at $14 ... Second at $18 ... Third at $1C ... etc.. + $04
In this example:
'PUB Main' points to the BYTE address of $14
'PUB SelfModify1' points to the BYTE address of $18
'PUB SelfModify2' points to the BYTE address of $1C
'PUB Loader' points to the BYTE address of $20
Note: DAT blocks are stored in memory 'before' PUB/PRI blocks.
The value at this BYTE address represents the BYTE offset from where your program starts in memory
which is at $10, where all Spin programs start. To get the actual BYTE address, add $10.
Likewise if you are looking at an address already within the memory offset ... i.e. the address of
'Code' in the DAT section. You need to subtract $10 in order to properly point to the data.
}}
DAT section
DAT
Code byte $37,$03,$38,$17 ''<- HEX values determined using F8 and studying the HEX code results
byte $3E,$D6,$1C,$37 '' under bare minimum Spin code snips.
byte $03,$38,$17,$3E
byte $D4,$47,$3F,$91
byte $35,$C0,$37,$01
byte $F6,$EC,$23,$04
byte $6E,$32
KitCar byte $37,$03,$38,$17 ''<- HEX values determined using F8 and studying the HEX code results
byte $3E,$D6,$1C,$36 '' under bare minimum Spin code snips.
byte $37,$03,$38,$17
byte $3E,$B4,$37,$22
byte $08,$13,$36,$37
byte $03,$38,$17,$3E
byte $D4,$43,$3F,$91
byte $35,$C0,$37,$03
byte $F6,$EC,$23,$09
byte $6D,$37,$22,$08
byte $13,$36,$37,$03
byte $38,$17,$3E,$D4
byte $42,$3F,$91,$35
byte $C0,$37,$03,$F6
byte $EC,$23,$09,$6D
byte $04,$50,$32
There is enough information in the Example, that you could directly read portions of memory as 'PUB snips' and store that data on a larger external memory for future recollection and execution as a LMM (Large Memory Module) environment since it's relatively easy to transfer data externally to a defined section in the DAT block.
For those that want to go further, it is possible to decode the HEX and break it down into an "OP code for the interpreter" format and a pseudo-human readable instruction by instruction level but anyone who has gone that route knows that it a tedious adventure. Also very rewarding when you begin to unravel and make those ah ha connections.
code History - below
History:
11-11-2010 v1.0 - Initial BETA release
11-12-2010 v1.1 - Added example to show that passing variables defined in the VAR
section is possible.
Note: This method should work if the variables within the VAR
section are fixed to a specific/consistent (used defined)
location.
- Modified Loader field to allow for code snips larger than 256 Bytes
- Modified Loader filed to allow it to move the pointer in other objects
before was confined only to the top object. instead of '-$10' it was
changed to '- @@0'
Note: This change is not fully tested, but suggested by a forum
member
11-14-2010 v1.2 - Added more examples to show variable passing
- Added a PUB routine called 'HEXExtract' which will extract and format
the HEX data to a serial terminal that can be directly cut-n-pasted
into the DAT section below.
Comments
This is amazing.
I once asked about pointers to methods in Spin and was told there wasn't any easy way to do it. If I understand you correctly, this technique is a big step towards method pointers.
This is just what I've been looking for.
I've been working on a menu for Raymans 3.5" Propeller Touchscreen Platform. I've been able to store most of the information for each menu in the DAT section (such as button color, size and actions to take with each corresponding button). I figured I could store this information in external memory if I needed. The limiting factor was the speciallized methods many of the menus required. I didn't know how I could store these externally. I think you just gave me the way to work this out.
This is awsome. Now there should be no limit to the number of menus I can use.
Thanks,
Duane Degn
Like Duane says, could be very useful switching between apps...
This is an interesting twist on modifying the call table to call relocated code. The idea of function pointers has been discussed a few times before. One thread is located at http://forums.parallax.com/showthread.php?t=120467 . You methods will work well for methods in the top object that don't access DAT or VAR variables (i.e., it only accesses stack variables.)
Dave
You sure about that, with other micros RAM can also be used for code execution self modifying or otherwise:)
Your code only allows for small programs that fit within the first 256 bytes of RAM. You can handle larger programs by modifying words instead of bytes as follows:
PUB Loader
word[$18] := @Code - @@0
word[$1C] := @KitCar - @@0
I also changed $10 to @@0 so that it will work in any object, and not just the top object. Also, your code requires that the code snippets in DAT have the same number of local stack variables as the dummy methods. You could fix this my modifying word[$1A] and word[$1E]. They should contain the number of bytes used for local stack variables.
Another restriction is that your code snippets cannot use DAT or VAR variables unless it was compiled with the same variable offsets as your unmodified code.
Dave
Edit: I forgot to mention that the code snippets must have the same number of calling parameters as the dummy methods. These parameters are loaded onto the stack just before the method is called.
This was just an idea, I'm sure there are lots of things to be discovered to this 'new door'.
Dave,
Your right about the VAR, but the DAT's can be a little more tricky. You just need to know where to look.
The VAR's are allocated at the end of your program space, while the DAT's are allocated immediately after the call tables that define where the PUB/PRI objects start.
Since the VAR's are relative to your 'top object' the interpreter knows where they are. The DAT's however are subject to change because of the self modifying nature of this example, and could be more tedious to locate.
BTW) thanks about the BYTE vs. the WORD reference to the program location ... something I overlooked since I was testing with small code snips.
Can't pointers to these just be passed as parameters to the code being called?
Thanks
In my opinion that opens to RUN some SPIN code of fly from SPI/I2C SRAMS/Flash and EEProms.
Way I are thinking on it --- Made some buffer for dynamically loading some code snippets. RUN it and if needed reload other snippet to it and run.
Ps. That open possibility to RUN SPIN code that are some 100KBytes
It makes relocatable code and loading it on the fly easier to understand and I am sure there are some real tricks that can be done here along similar lines.
It was noted that many micros use flash for program storage and cannot run programs from ram. There are some micros that can run from ram, but remember, most of these still only have small amounts of ram such that we are often talking about a cog's equivalent in ram.
I have many designs dating back to the 70's which download code and run from ram. Some of these used battery backed rams. The manufacturers gave a 10 year battery life that actually all failed after 5 years and a few months! Fortunately, they were all able to be rebooted (EPROM minimal boot code just like I use on the RamBlade) and reloaded remotely as I had 100's located all around AUstralia.
The reality is that Ram takes 2-4 times the die space than flash (this is about right isn't it Beau?) so most micros have much more Flash than Ram.
You are correct, RAM is generally larger than Flash.
... But I think we are confusing the issue... The Code being executed would be in RAM (No different than current Propeller Spin Code) and loaded from EEPROM, again no different than current operations.
The difference is being able to load RAM from a secondary external EEPROM and control where it goes in RAM before it gets executed.
I envision a large Spin program (limited only by the size of your external memory).
The DAT section could be loaded/executed/reloaded with another COG... double buffering could be used to make the code easier to sync to (i.e. no waiting until code is done and new code is loaded before proceeding)
The DAT section could be loaded via an external EEPROM or even from an external PROP (i.e. the High speed Prop-2-Prop communication program I wrote awhile back)
Right now, the Big language is C. Maybe this is a way of doing this with Spin?
We have plenty of external ram and plenty of code that can move fragments from eeprom or sd card. Many combinations and permuations there ranging from simple (a bigger eeprom to hub) up to complex (sd card to external ram, thence to hub).
I once tried to decode hex from the spin compiler but it didn't seem to make sense. Is it possible to compile a code fragment as a standalone entity that can exist as a binary file? A very simple example - a code fragment that you pass one value, the function adds one to that value and returns the value.
"Is it possible to compile a code fragment as a standalone entity that can exist as a binary file? A very simple example - a code fragment that you pass one value, the function adds one to that value and returns the value." - Yes ... I have an example I will have to post tomorrow (shutting down for the night)
PS - The HEX code isn't that hard to decode, you just have to break it down into singular commands and observe the results.
There are funny things that happen with defining values at bit breaks, nibble breaks, byte breaks, word breaks, and long breaks, but once you get past that you begin to understand why it was done this way. ... Also, depending on what side of the equal sign your operation is, the values are defined differently... i.e. is the variable referenced being written to or read from?
IF's and REPEAT's are tricky, but not impossible.
Reloadable spin routines without re-booting is a nice concept.
Yikes, what time do you work to? It is 8pm here and I think the US is 9 hours ahead?!
Anyway, that is great news. Actually, this could be a very interesting development. Given all the work with external ram and sd cards, I think external memory has gone mainstream. And just when I thought 512k was cool, along comes jazzed with 32Mb.
My view of the propeller, which is perhaps slightly biased, is that at a minimum it really needs a vga, keyboard, mouse, sd card and external memory. But if you say that we can do spin code fragments, that greatly increases what the prop can do.
You could have the slower cheap version that moves routines from eeprom to hub ram.
The slightly more upmarket version, which moves routines from an sd card to hub ram.
And the Rolls Royce version that moves routines from sd card to external ram at bootup, and thence to hub ram (but very quickly).
If you can do one routine, you could potentially do thousands. Are there limitations on this, eg did you say that individual routines need to fit into a certain size?
As I see it. It is 2 ways to go.
1. Have RUN stub and reuse most of available memory for that.
2. Use dimensioned buffer and load it with code snipets that need be run.
The main problem I saw with this is that it couldn't be used to build generic apps. Each piece of code that is loaded had to be compiled under the same framework as the main code. This would be OK for creating very large apps that could dynamically load and execute parts of itself, but it doesn't work well for executing independent apps.
I evolved C-DOS into spinix, which uses a different approach for loading apps. spinix uses an SD file system instead of EEPROM. The apps are independent from the main app, and they run in their own cog. The apps share the serial and SD drivers by communicating through known rendezvous locations.
Each method has it's pros and cons. It would be interesting to develop the single-cog large-app method further.
Dave
"Yikes, what time do you work to? It is 8pm here and I think the US is 9 hours ahead?!"
I'm usually up until about 1-2am ... but this I don't consider work, I did this under my 'play time'
I uploaded version 1.1 to the Top thread that shows passing global COG variables into the "DAT Code Snip" ... later I will show how to use local PUB/PRI variables.
Certainly, this opens up another way of thinking...
For example, we could "allocate" a single 2KB block for loading PASM programs (sort of done IIRC) into various cogs using a lock mechanism. We could load runtime spin code blocks and build the links as this is done.
The variables are reference by FIXED pointers that the interpreter understands.
For example:
The code 64 and 68 reference the FIRST and SECOND local PUB variables ... if there was a THIRD, it would be 6C... a FOURTH 70, etc.
Code 17 indicates the operation to perform... in this case a 'STRCOMP'
Code 41 references the FIRST global COG variable ... likewise if there were a SECOND global COG variable the code to use would be 45 ... 49, for a THIRD, 4D for a FOURTH, etc.
In BOTH cases for the variable there is an offset that indicates to the interpreter whether or not the variable is being read to or written from.
i.e.
+$0 for READ operation
+$1 for WRITE operation
+$3 for Address READ operation ... @var
So for 64 and 68 the offset would be +$0 since these locations are being read from. However in the case of 41, there is a +$1 offset here indicating that this variable is being written to.
I just threw the +$3 in there for good measure.
This writeup is a conglomeration from all sources including my own from re-writing sections of the Interpreter.
Thanks for the Interpreter codes, I have something similar.
I just made it real easy for someone who didn't want to mess with trying to figure out the HEX codes.
Version 1.2 at the top of this thread has more examples to show passing variables back-n-forth.
There is also a HEX extraction utility (<-PUB).
This PUB sends and formats the HEX code to a serial terminal with just the hex code for the PUB you want to extract. The format is such that you can just cut and paste the output from the serial terminal back into the DAT section of the example program.
How would I go about creating a temporary array of, say 500 bytes, for use by this PUB?
If I define that array in a VAR, the compiler goes and moves that to (I think) the beginning of the program and groups it with all the other VARs.
I am not sure what happens with DAT statement but I think the same thing happens.
If you define the pub
PUB mypub | a,b,c,d
I am not sure how a,b,c,d are handled - whether as temporary or permanent variables, but in any case, I'm not sure how to replicate that with arrays.
So I am thinking of an idea where you define a block of hub ram as 'common variable' space for use by all these PUBs that might exist in external ram. The variables are overwritten each time a new external pub is loaded.
How would you reference such variables? Do you define them a,b,c,d to z? That is a bit old-school but I think it would work.
In terms of compiling such code, could an IDE shell a command line compiler like Homespun multiple times to automatically create the hex code?
Treat a,b, c, and d as if the pipe "|" was not there. IOW the variables are all defined as long when placed in a PUB header and are consecutive on their location.
In the examples below, all of the variables a,b,c, and d are located the same in memory as longs.
For an array, I would create it in the main COG under the VAR section and pass the first address element into your PUB code.