design to support dynamically loading code

ags

I've hit a memory limit that I can't get past by pruning code. I expected it would happen sometime, and apparently that time is now.

I'm thinking that not all functions need to be available at once - so I can partition the problem, and load cogs with the necessary code, then swap it out for different code. The more I think of this, the more it becomes something like a lightweight file system with a dynamic loader. Here's the rough sketch:

• Store code in an EEPROM (or SD card) - virtually unlimited memory capacity.
• Create some type of index in EEPROM so I can find the location of each "module" as needed (this is the "lightweight file system")
• Have a loader always available in RAM that can copy from EEPROM to high RAM. If it's just PASM code, just use the top 512 longs from RAM. I could check to be sure there is enough room by taking the address of an object on the stack.
• Initialize a cog (finished with it's current purpose) with the code placed in high RAM. That RAM can now be reclaimed or used for the next module to load.
• If there is any SPIN code needed to interact with the PASM code, include it (for all modules that could ever be loaded) in the main program to avoid the next step.
• If SPIN code needs to be dynamically loaded, (more than one cog executing SPIN code at the same time) I suppose I could init a cog with the SPIN interpreter and point it at a non-standard start location in RAM, loaded from EEPROM. That would require better stack management than I currently am doing. And I would not be able to reclaim that space as I would with PASM code. That seems more complex to me, so I'm hoping to avoid that if possible.

To me, this is the "obvious" design. My questions:

• It is similar to what others have done?
• Are there problems that I haven't thought of?
• Is there a better way?

Thanks.

SRLM

Doesn't PropGCC do this for external memory models? I believe that it will cache a few KB of instructions in the hub. And of course automatically manage all the memory accesses.

One problem that I see with your system is functions that call other functions: it's easy to imagine a case where one function calls functions in many different modules, requiring those to all be loaded. Add a few levels of this and you'll suffer a performance hit.

What is your project? I'm always surprised by projects that hit the limit.

Dave Hein

Yes, PropGCC has various external memory models that can execute code from SD, EEPROM, flash and other forms of external memory. Spin code can be converted to C or C++ using the spin2cpp utility. I also have a spin-to-C converter that produces more readable code than spin2cpp, but it's not as complete and doesn't handle all the special tricks that are done in Spin.

If you stay with Spin, you could dynamically load PASM and Spin code from SD or EEPROM. In either case you would most likely talk to the dynamic code through mailboxes. It would be possible to modify the method tables to talk to dynamically loaded Spin call, but that gets pretty tricky, and probably isn't worth the effort. It would be easier to run the dynamically loaded code in it's own cog. You would just need to add the memory offset to the PBASE, VBASE, DBASE, PCURR and DCURR values in the header.

And yes, there are a few existing OSes that do something similar, but they are probably don't do exactly what you want to do.

max72

Dr Acula's cogjects are a possible solution.

http://forums.parallax.com/showthread.php/130537-Cogjects-load-cog-code-from-SD-and-save-hub-ram

ags

Dr Acula's concept is very close to what I was thinking. It's not just memory, but something like a Swiss Army knife: I have many things that I want to support - realizing they are not all available at the same time. If the SPIN portion is loaded into RAM at boot, then I can swap the PASM drivers in and out of cogs as needed to enable up to 7 (8-top level SPIN "event loop") types of capability at one time.

prof_braino

ags wrote: »

[*]Are there problems that I haven't thought of?
[*]Is there a better way?
Thanks.

I don't know if this is "better" but here's different and it works:
* Propforth suppports using EEprom or SD as simple fast storage. We can read, write and load "files" of text. Very simple.
* Propforth supports loading chunks of code on the fly. We can read and load new definitions off the storage, run them, and forget them (if desired) from the dictionary on the fly. We just have to load such that we always want the tail (newest) part of the dictionary eliminated.
* Propforth supports PAGED ASSEMBLER. The optimized assembler code loads into a buffer and executes. It stays there until we load something else over it. No memory clean up issues, ever.
* Propforth support running scripts directly from storage. A file need not contain new definitions, the app can be placed in a file. When that file is called and executed, the app runs. Such an app can call other files. Using this method, the app can be a big as you can handle. We haven't filled a 4 gig SD yet, usually apps only take a few K.
* the load time from storage is not a big factor (so far). The mechanism is so simple it is fast. When load time would have an impact, arrange the app to work with this. Requires thought, but still simple.

Of course the biggest draw back is that it uses forth, which many consider cheating, since it does so much so fast and so easily. But you might consider some parts of this as inspiration for your implementation.

ags

So many moving parts. Hard to make progress with so much to learn. Very intriguing for future consideration.

What is the development environment?