PDA

View Full Version : DEMO: Self Modify Spin Code Example



Beau Schwabe
11-12-2010, 06:23 AM
I was inspired by this thread...

http://forums.parallax.com/showthread.php?t=126887

... Phil's post got me thinking about something that I had stumbled across but didn't quite comprehend.

What this Example does is basically allow you define "HEX code snips" (<-- interpreter code) in the DAT section and call them as if they were originally defined as a PUB/PRI routine.


It's kind of funny, because the heart of this Example only uses two lines of code, and that's to show two separate program snips that are residing in the DAT section...

Main PUB pointer magic


PUB Loader '' <- Change PUB pointer to point directly to DAT memory as if DAT was a PUB
byte[$18] := @Code - $10
byte[$1C] := @KitCar - $10

{{ 

Note: First PUB/PRI is always at $14 ... Second at $18 ... Third at $1C ... etc.. + $04

In this example:

'PUB Main' points to the BYTE address of $14
'PUB SelfModify1' points to the BYTE address of $18
'PUB SelfModify2' points to the BYTE address of $1C
'PUB Loader' points to the BYTE address of $20

Note: DAT blocks are stored in memory 'before' PUB/PRI blocks.


The value at this BYTE address represents the BYTE offset from where your program starts in memory
which is at $10, where all Spin programs start. To get the actual BYTE address, add $10.

Likewise if you are looking at an address already within the memory offset ... i.e. the address of
'Code' in the DAT section. You need to subtract $10 in order to properly point to the data.


}}


DAT section


DAT
Code byte $37,$03,$38,$17 ''<- HEX values determined using F8 and studying the HEX code results
byte $3E,$D6,$1C,$37 '' under bare minimum Spin code snips.
byte $03,$38,$17,$3E
byte $D4,$47,$3F,$91
byte $35,$C0,$37,$01
byte $F6,$EC,$23,$04
byte $6E,$32

KitCar byte $37,$03,$38,$17 ''<- HEX values determined using F8 and studying the HEX code results
byte $3E,$D6,$1C,$36 '' under bare minimum Spin code snips.
byte $37,$03,$38,$17
byte $3E,$B4,$37,$22
byte $08,$13,$36,$37
byte $03,$38,$17,$3E
byte $D4,$43,$3F,$91
byte $35,$C0,$37,$03
byte $F6,$EC,$23,$09
byte $6D,$37,$22,$08
byte $13,$36,$37,$03
byte $38,$17,$3E,$D4
byte $42,$3F,$91,$35
byte $C0,$37,$03,$F6
byte $EC,$23,$09,$6D
byte $04,$50,$32


There is enough information in the Example, that you could directly read portions of memory as 'PUB snips' and store that data on a larger external memory for future recollection and execution as a LMM (Large Memory Module) environment since it's relatively easy to transfer data externally to a defined section in the DAT block.

For those that want to go further, it is possible to decode the HEX and break it down into an "OP code for the interpreter" format and a pseudo-human readable instruction by instruction level but anyone who has gone that route knows that it a tedious adventure. Also very rewarding when you begin to unravel and make those ah ha connections.



code History - below


History:

11-11-2010 v1.0 - Initial BETA release
11-12-2010 v1.1 - Added example to show that passing variables defined in the VAR
section is possible.

Note: This method should work if the variables within the VAR
section are fixed to a specific/consistent (used defined)
location.

- Modified Loader field to allow for code snips larger than 256 Bytes

- Modified Loader filed to allow it to move the pointer in other objects
before was confined only to the top object. instead of '-$10' it was
changed to '- @@0'

Note: This change is not fully tested, but suggested by a forum
member

11-14-2010 v1.2 - Added more examples to show variable passing
- Added a PUB routine called 'HEXExtract' which will extract and format
the HEX data to a serial terminal that can be directly cut-n-pasted
into the DAT section below.

Duane Degn
11-12-2010, 07:32 AM
Beau,

This is amazing.

I once asked about pointers to methods in Spin and was told there wasn't any easy way to do it. If I understand you correctly, this technique is a big step towards method pointers.

This is just what I've been looking for.

I've been working on a menu for Raymans 3.5" Propeller Touchscreen Platform. I've been able to store most of the information for each menu in the DAT section (such as button color, size and actions to take with each corresponding button). I figured I could store this information in external memory if I needed. The limiting factor was the speciallized methods many of the menus required. I didn't know how I could store these externally. I think you just gave me the way to work this out.

This is awsome. Now there should be no limit to the number of menus I can use.

Thanks,

Duane Degn

Rayman
11-12-2010, 02:48 PM
That's really neat. If I'm reading it right, the you could use this trick to read in code from an SD card and then switch in and out routines...

Like Duane says, could be very useful switching between apps...

David Betz
11-12-2010, 02:54 PM
That's really neat. If I'm reading it right, the you could use this trick to read in code from an SD card and then switch in and out routines...

Like Duane says, could be very useful switching between apps...
This is one big advantage that the Propeller has over most other microcontrollers. It executes code out of RAM not flash. That means that you can update the code at runtime. With pretty much any other microcontroller, changing the code at runtime means reflashing the program store on the chip and possibly wearing out the flash. Of course, another microcontroller could easily be made to run a virtual machine like the SPIN VM out of RAM but the Propeller even runs its native assembly language code out of RAM.

Dave Hein
11-12-2010, 04:03 PM
Beau,

This is an interesting twist on modifying the call table to call relocated code. The idea of function pointers has been discussed a few times before. One thread is located at http://forums.parallax.com/showthread.php?t=120467 . You methods will work well for methods in the top object that don't access DAT or VAR variables (i.e., it only accesses stack variables.)

Dave

Batang
11-12-2010, 04:09 PM
David Betz
This is one big advantage that the Propeller has over most other microcontrollers. It executes code out of RAM not flash


You sure about that, with other micros RAM can also be used for code execution self modifying or otherwise:)

David Betz
11-12-2010, 04:10 PM
You sure about that, with other micros RAM can also be used for code execution self modifying or otherwise:)
Really? Which ones? I don't the the AVR or the PIC24 can execute assembly code from RAM can they?

Batang
11-12-2010, 04:15 PM
Ones that I have used Freescale 9S12's and ARM

Dave Hein
11-12-2010, 04:23 PM
Beau,

Your code only allows for small programs that fit within the first 256 bytes of RAM. You can handle larger programs by modifying words instead of bytes as follows:

PUB Loader
word[$18] := @Code - @@0
word[$1C] := @KitCar - @@0

I also changed $10 to @@0 so that it will work in any object, and not just the top object. Also, your code requires that the code snippets in DAT have the same number of local stack variables as the dummy methods. You could fix this my modifying word[$1A] and word[$1E]. They should contain the number of bytes used for local stack variables.

Another restriction is that your code snippets cannot use DAT or VAR variables unless it was compiled with the same variable offsets as your unmodified code.

Dave

Edit: I forgot to mention that the code snippets must have the same number of calling parameters as the dummy methods. These parameters are loaded onto the stack just before the method is called.

Beau Schwabe
11-12-2010, 05:34 PM
Thanks all...

This was just an idea, I'm sure there are lots of things to be discovered to this 'new door'.


Dave,

Your right about the VAR, but the DAT's can be a little more tricky. You just need to know where to look.

The VAR's are allocated at the end of your program space, while the DAT's are allocated immediately after the call tables that define where the PUB/PRI objects start.

Since the VAR's are relative to your 'top object' the interpreter knows where they are. The DAT's however are subject to change because of the self modifying nature of this example, and could be more tedious to locate.

BTW) thanks about the BYTE vs. the WORD reference to the program location ... something I overlooked since I was testing with small code snips.

Rayman
11-12-2010, 05:43 PM
Regarding issues of finding DAT and VAR data...

Can't pointers to these just be passed as parameters to the code being called?

Sapieha
11-12-2010, 06:06 PM
Hi Beau Schwabe.

Thanks

In my opinion that opens to RUN some SPIN code of fly from SPI/I2C SRAMS/Flash and EEProms.





Thanks all...

This was just an idea, I'm sure there are lots of things to be discovered to this 'new door'.


Dave,

Your right about the VAR, but the DAT's can be a little more tricky. You just need to know where to look.

The VAR's are allocated at the end of your program space, while the DAT's are allocated immediately after the call tables that define where the PUB/PRI objects start.

Since the VAR's are relative to your 'top object' the interpreter knows where they are. The DAT's however are subject to change because of the self modifying nature of this example, and could be more tedious to locate.

BTW) thanks about the BYTE vs. the WORD reference to the program location ... something I overlooked since I was testing with small code snips.

David Betz
11-12-2010, 06:12 PM
This looks really cool and it would be fun to create a framework for loading code dynamically. I have a question though. How much of the layout of memory is fixed by the way the SPIN VM in ROM works and how much is up to the Propeller Tool and hence could change between different releases of that tool?

Sapieha
11-12-2010, 06:23 PM
Hi David Betz.

Way I are thinking on it --- Made some buffer for dynamically loading some code snippets. RUN it and if needed reload other snippet to it and run.



Ps. That open possibility to RUN SPIN code that are some 100KBytes


This looks really cool and it would be fun to create a framework for loading code dynamically. I have a question though. How much of the layout of memory is fixed by the way the SPIN VM in ROM works and how much is up to the Propeller Tool and hence could change between different releases of that tool?

Cluso99
11-12-2010, 11:34 PM
This is an extremely interesting concept Beau.

It makes relocatable code and loading it on the fly easier to understand and I am sure there are some real tricks that can be done here along similar lines.

It was noted that many micros use flash for program storage and cannot run programs from ram. There are some micros that can run from ram, but remember, most of these still only have small amounts of ram such that we are often talking about a cog's equivalent in ram.

I have many designs dating back to the 70's which download code and run from ram. Some of these used battery backed rams. The manufacturers gave a 10 year battery life that actually all failed after 5 years and a few months! Fortunately, they were all able to be rebooted (EPROM minimal boot code just like I use on the RamBlade) and reloaded remotely as I had 100's located all around AUstralia.

The reality is that Ram takes 2-4 times the die space than flash (this is about right isn't it Beau?) so most micros have much more Flash than Ram.

Beau Schwabe
11-13-2010, 12:41 AM
Cluso99,

You are correct, RAM is generally larger than Flash.

... But I think we are confusing the issue... The Code being executed would be in RAM (No different than current Propeller Spin Code) and loaded from EEPROM, again no different than current operations.
The difference is being able to load RAM from a secondary external EEPROM and control where it goes in RAM before it gets executed.

I envision a large Spin program (limited only by the size of your external memory).

The DAT section could be loaded/executed/reloaded with another COG... double buffering could be used to make the code easier to sync to (i.e. no waiting until code is done and new code is loaded before proceeding)

The DAT section could be loaded via an external EEPROM or even from an external PROP (i.e. the High speed Prop-2-Prop communication program I wrote awhile back)

Heater.
11-13-2010, 07:20 AM
Overlays for Spin. Cool.

Dr_Acula
11-13-2010, 07:30 AM
Are we talking Big Spin here?

Right now, the Big language is C. Maybe this is a way of doing this with Spin?

We have plenty of external ram and plenty of code that can move fragments from eeprom or sd card. Many combinations and permuations there ranging from simple (a bigger eeprom to hub) up to complex (sd card to external ram, thence to hub).

I once tried to decode hex from the spin compiler but it didn't seem to make sense. Is it possible to compile a code fragment as a standalone entity that can exist as a binary file? A very simple example - a code fragment that you pass one value, the function adds one to that value and returns the value.

Beau Schwabe
11-13-2010, 07:44 AM
Dr_Acula,

"Is it possible to compile a code fragment as a standalone entity that can exist as a binary file? A very simple example - a code fragment that you pass one value, the function adds one to that value and returns the value." - Yes ... I have an example I will have to post tomorrow (shutting down for the night)

PS - The HEX code isn't that hard to decode, you just have to break it down into singular commands and observe the results.

There are funny things that happen with defining values at bit breaks, nibble breaks, byte breaks, word breaks, and long breaks, but once you get past that you begin to understand why it was done this way. ... Also, depending on what side of the equal sign your operation is, the values are defined differently... i.e. is the variable referenced being written to or read from?

IF's and REPEAT's are tricky, but not impossible.

Cluso99
11-13-2010, 10:03 AM
Yes I did understand Beau. I was just answering Dave and Batang.

Reloadable spin routines without re-booting is a nice concept.

Dr_Acula
11-13-2010, 10:26 AM
Re Beau "Yes ... I have an example I will have to post tomorrow (shutting down for the night)"

Yikes, what time do you work to? It is 8pm here and I think the US is 9 hours ahead?!

Anyway, that is great news. Actually, this could be a very interesting development. Given all the work with external ram and sd cards, I think external memory has gone mainstream. And just when I thought 512k was cool, along comes jazzed with 32Mb.

My view of the propeller, which is perhaps slightly biased, is that at a minimum it really needs a vga, keyboard, mouse, sd card and external memory. But if you say that we can do spin code fragments, that greatly increases what the prop can do.

You could have the slower cheap version that moves routines from eeprom to hub ram.
The slightly more upmarket version, which moves routines from an sd card to hub ram.
And the Rolls Royce version that moves routines from sd card to external ram at bootup, and thence to hub ram (but very quickly).

If you can do one routine, you could potentially do thousands. Are there limitations on this, eg did you say that individual routines need to fit into a certain size?

Sapieha
11-13-2010, 10:33 AM
Hi Dr_Acula.

As I see it. It is 2 ways to go.
1. Have RUN stub and reuse most of available memory for that.
2. Use dimensioned buffer and load it with code snipets that need be run.





Re Beau "Yes ... I have an example I will have to post tomorrow (shutting down for the night)"

Yikes, what time do you work to? It is 8pm here and I think the US is 9 hours ahead?!

Anyway, that is great news. Actually, this could be a very interesting development. Given all the work with external ram and sd cards, I think external memory has gone mainstream. And just when I thought 512k was cool, along comes jazzed with 32Mb.

My view of the propeller, which is perhaps slightly biased, is that at a minimum it really needs a vga, keyboard, mouse, sd card and external memory. But if you say that we can do spin code fragments, that greatly increases what the prop can do.

You could have the slower cheap version that moves routines from eeprom to hub ram.
The slightly more upmarket version, which moves routines from an sd card to hub ram.
And the Rolls Royce version that moves routines from sd card to external ram at bootup, and thence to hub ram (but very quickly).

If you can do one routine, you could potentially do thousands. Are there limitations on this, eg did you say that individual routines need to fit into a certain size?

Dave Hein
11-13-2010, 03:13 PM
I created an app that loads Spin code and calls it back in April. I named it C-DOS because it used the CLIB routines, even though it is writen in Spin. The thread is located at http://forums.parallax.com/showthread.php?t=121604 . This also used eefs.spin, which was an EEPROM file system I was working on.

The main problem I saw with this is that it couldn't be used to build generic apps. Each piece of code that is loaded had to be compiled under the same framework as the main code. This would be OK for creating very large apps that could dynamically load and execute parts of itself, but it doesn't work well for executing independent apps.

I evolved C-DOS into spinix, which uses a different approach for loading apps. spinix uses an SD file system instead of EEPROM. The apps are independent from the main app, and they run in their own cog. The apps share the serial and SD drivers by communicating through known rendezvous locations.

Each method has it's pros and cons. It would be interesting to develop the single-cog large-app method further.

Dave

Beau Schwabe
11-13-2010, 07:38 PM
Dr_Acula,

"Yikes, what time do you work to? It is 8pm here and I think the US is 9 hours ahead?!"

I'm usually up until about 1-2am ... but this I don't consider work, I did this under my 'play time'


I uploaded version 1.1 to the Top thread that shows passing global COG variables into the "DAT Code Snip" ... later I will show how to use local PUB/PRI variables.

Cluso99
11-13-2010, 08:08 PM
IIRC the PRI & PUB code is relocatable. However, running multiple cogs would have some pretty hard issues, if the "other" cogs were setup with a predefined block size, then that may resolve their problems. Then the "main" cog could be free to use the rest.

Certainly, this opens up another way of thinking...

For example, we could "allocate" a single 2KB block for loading PASM programs (sort of done IIRC) into various cogs using a lock mechanism. We could load runtime spin code blocks and build the links as this is done.

Beau Schwabe
11-13-2010, 09:12 PM
Here is a brief breakdown example of the HEX code... since the variables are not referenced by the placement or the order of code, it makes this code-in-lay approach possible.


The variables are reference by FIXED pointers that the interpreter understands.

For example:

The code 64 and 68 reference the FIRST and SECOND local PUB variables ... if there was a THIRD, it would be 6C... a FOURTH 70, etc.

Code 17 indicates the operation to perform... in this case a 'STRCOMP'

Code 41 references the FIRST global COG variable ... likewise if there were a SECOND global COG variable the code to use would be 45 ... 49, for a THIRD, 4D for a FOURTH, etc.



In BOTH cases for the variable there is an offset that indicates to the interpreter whether or not the variable is being read to or written from.

i.e.

+$0 for READ operation
+$1 for WRITE operation
+$3 for Address READ operation ... @var

So for 64 and 68 the offset would be +$0 since these locations are being read from. However in the case of 41, there is a +$1 offset here indicating that this variable is being written to.

I just threw the +$3 in there for good measure.

Cluso99
11-14-2010, 03:13 AM
FYI: I posted a writeup of the Interpreter codes in my "Faster Spin Interpreter" thread. See the "Tools" link below to find the thread.

This writeup is a conglomeration from all sources including my own from re-writing sections of the Interpreter.

Beau Schwabe
11-14-2010, 07:55 AM
Cluso99,

Thanks for the Interpreter codes, I have something similar.

I just made it real easy for someone who didn't want to mess with trying to figure out the HEX codes.


Version 1.2 at the top of this thread has more examples to show passing variables back-n-forth.

There is also a HEX extraction utility (<-PUB).
This PUB sends and formats the HEX code to a serial terminal with just the hex code for the PUB you want to extract. The format is such that you can just cut and paste the output from the serial terminal back into the DAT section of the example program.

Cluso99
11-14-2010, 08:22 AM
An easy way to see what bytecodes spin produces is to compile using bst or homespun and look at the listing.

Dr_Acula
01-27-2011, 11:12 PM
Can I just clarify something. Say I have a PUB. It is 50 lines of code and all of its variables are internal. It is passed one long (which might be just a variable, or it might point to an array of data). It returns one long (which might also point to more data). It makes no references to global variables within its code.

How would I go about creating a temporary array of, say 500 bytes, for use by this PUB?

If I define that array in a VAR, the compiler goes and moves that to (I think) the beginning of the program and groups it with all the other VARs.

I am not sure what happens with DAT statement but I think the same thing happens.

If you define the pub
PUB mypub | a,b,c,d

I am not sure how a,b,c,d are handled - whether as temporary or permanent variables, but in any case, I'm not sure how to replicate that with arrays.

So I am thinking of an idea where you define a block of hub ram as 'common variable' space for use by all these PUBs that might exist in external ram. The variables are overwritten each time a new external pub is loaded.

How would you reference such variables? Do you define them a,b,c,d to z? That is a bit old-school but I think it would work.

In terms of compiling such code, could an IDE shell a command line compiler like Homespun multiple times to automatically create the hex code?

Beau Schwabe
01-28-2011, 02:31 AM
Dr_Acula,

Treat a,b, c, and d as if the pipe "|" was not there. IOW the variables are all defined as long when placed in a PUB header and are consecutive on their location.

In the examples below, all of the variables a,b,c, and d are located the same in memory as longs.



PUB YourCode | a,b,c,d
...
PUB YourCode(a) | b,c,d
...
PUB YourCode(a,b) | c,d
...
PUB YourCode(a,b,c) | d
...
PUB YourCode(a,b,c,d)




For an array, I would create it in the main COG under the VAR section and pass the first address element into your PUB code.

Dr_Acula
01-28-2011, 08:32 AM
Thanks Beau,

I'm trying to think what this can do and what it can't do. Say you have a PUB function and it contains a few longs and maybe it reuses some commn hub memory for strings or whatever. That ought to work fine.

However, what about a self contained PUB function that calls other PUBs within its code? Is that a stack push/pop exercise and hence is it also possible to do this with relocatable code?

But how would you glue that all together with the naming of those sub-pubs? Dummy names? Or do you not have any sub-pubs and just use copy/paste a lot? (which is not as bad as it sounds when working with lots of external memory).

MagIO2
01-28-2011, 12:36 PM
I think one thing is missing ... the link to already loaded drivers.

I have a system running called COG OS. It has some nice features:
1. The upper EEPROM contains all kind of drivers, for example FullDuplexSerial, LCD driver, Keyboard driver, SD card ....
2. There is a COGOSCORE which contains functions to load the drivers needed by name and version number.
3. The drivers loaded are registered in upper RAM.
4. Another set of core functions is there to attach to those drivers.

The link between 2,3 and 4 is that the parameter RAM that you use for communication between main code and the driver is not an array, but taken and reserved in upper RAM.

So, with the "loadspin" you'd first have some code which attaches to the already running drivers.
This way each piece of code loaded can have access to SD card and so on

So, what I have for example is a prime.BIN file on SD card which simply attaches to the full duplex serial driver and spits out a bunch of prime numbers. Another demo uses the LCD.

It's also possible to pass any number (up to 10) of parameters to the loades SPIN function.

Why didn't I post it yet ... well ... because it would need a good documentation for setup of such a system. Maybe I should post a ZIP for all who are interested.

It's easy to create the external functions ... first develop them on it's own including load of drivers and stuff, then move the code to a stub which no longer loads the drivers, but uses the core functions to attach to the existing drivers instead. Compile this and save it as BIN.

Dr_Acula
01-29-2011, 04:53 AM
That sounds brilliant! The hard work is taking each of those obex objects and changing the code so that it works in a standard way. How many objects have you done?

This could link in nicely with the Big Spin work that Dave is doing. Big Spin could contain all the support code (which sometimes is a huge amount of spin and not much cog code).


It's easy to create the external functions

How quickly can you create one? I'm taking about 1 month! If you can do these quicker that would be great. The same driver could potentially be used by BigSpin, Catalina and Propbasic.

What is the most number of longs you have needed to reserve in upper ram? Is 10 enough per cog or would you need more for some drivers?

blittled
01-29-2011, 05:07 AM
MagIO2 that code sounds great! I would be interested it in trying on the Chameleon. It would probably work with the C3 as well since both have an extra eeprom on the SPI bus.

MagIO2
01-29-2011, 10:09 AM
Ok, I am going to prepare a long long post for you.
The drivers I have converted so far are:
TV
Keyboard
Full Duplex Serial
BenkyLCD
Spfd5408

The EEPROM is simply organized in slots of the same size (COG RAM size). The first slot is for the TOC. There is the name, the versionnumber and the number of longs needed in HUB.
The table of loaded drivers later on contains a hash value instead of the full name and the address of the Hub-buffer......
More has to come this evening.