Big Spin - is it still a pipedream?
Dr_Acula
Posts: 5,484
With the C3 now supporting 64k of external ram, plus a number of other hardware designs that have external memory, I am wondering if we could brainstorm the idea of Big Spin, ie Spin that breaks out of the 32k memory size of the propeller chip.
I was inspired by this thread http://forums.parallax.com/showthread.php?127976-Spin-Simulator&highlight=spin+simulator
I'm also pulling in ideas from Zog, Catalina, the Zicog simulation and Bean's Basic.
The limitation, as always, is that we don't seem to have any open source code. But, that didn't stop those four projects listed above.
So why can't we write this ourselves?!
We already have "Big Pasm", in the form of LMM. It takes one cog and you can keep writing pasm code until the memory runs out.
So it seems to me that what we need is something to translate Spin into LMM, and I see two ways to do this. One is to translate to the bytecode that the Spin interpreter uses. I see most of this as possible but there are still bits missing, though the Spin Simulator might be able to plug those holes. Especially the bit about the flat 32 bit memory map.
The other option I see is Bean's Basic option where you compile to pure Pasm and don't run it through any simulator at runtime. This may well produce faster code and save a cog as well.
There have also been posts in the past about better IDE optios, eg auto indenting, autocompletion of all the methods available in an object once you type the . and other things. Maybe a new/improved IDE is an option.
But, the IDE may not be the highest priority.
So here is the idea. Take some existing spin code, typed into a text editor, or the Proptool or BST. Keep typing code, 500k or more. None of the above will be able to compile this, but they can at least save it and you will be working in a familiar environment.
Then run a simple batch file to compile this program to LMM pasm.
I've been thinking about how it might be possible to write a compiler using the collaborative input from many people, each of whom may have expertise in different languages, and I have come up with a concept that I'd like to run up the flagpole and see if anyone salutes.
The idea is to create a multipass compiler. I wrote a two pass compiler once, but it should be possible to do it with more passes and hence break the task up into smaller pieces. Each pass works on text files stored on the hard drive. The input is text file(s) and the output is text files(s). One pass could be written in C. The next in Java. The next in Basic. It does not matter as long as the program can read text files and write text files.
An example. The first pass might be to take the original Spin file, work out the names of the sub objects, and translate all instances of variables into a unique name that identifies that variable. For instance, in the main spin program there might be a variable "i", and in a sub object "serial" there might also be a variable "i". These are different variables, so you go through all the Spin code and replace "i" with "main.i" and in the sub object with "serial.i"
The next pass might simply combine all the sub objects and the main object into one big text file. Each pass is simply reading text files and then saving new text files.
The next pass might group all the CON parts together, and all the VAR parts, much as the Proptool compiler seems to do. Maybe this is not necessary?
The next pass could put all the CON values into a text file list and save as a separate text file.
The next pass could take all variables and create a list of those variables along with a token that goes with those variables, eg "myvariable" becomes "variable1" in a list. This list is now in a separate text file for use later on.
The next pass would do a global replace to put in "variable1" "variable2" in the code instead of the actual names. Or maybe just token numbers v1 v2 v3
Next pass would take all nested bracket code and expand it out to separate lines. Bracket code seems to have a hidden long that is passed from the inside to the outside - we could use "result" or something to move data to the next line.
The next pass might take all "myvariable:=1" and put in standardised spaces eg "myvariable := 1"
The next pass could search for text strings like "==" and ":=" and tokenise those.
The next pass could look for indents and convert (temporarily) into a format easier for a compiler to understand, eg nested "{" and "}" like in C.
And finally, we can start to do some compilation, eg variable1 := variable2 translated into LMM pasm.
We need a pass to search for loops, eg "repeat" and convert into conditional jumps.
And we need a pass to do some other complicated things like a cognew, which now would take the next 2k of data from external memory, move it to a fixed common location in hub memory and then execute pasm code to move it into the cog.
Now, the entire project is extremely daunting. But each of the above bits of code are not so hard. And they may well be a lot easier if anyone is free to program in their own language.
So I suppose the big question is whether there is enough demand for Spin programs that are more than 32k.
Thoughts, input, critique, offers of help, showstopper comments and even flames would be greatly appreciated.
Cheers, Drac
I was inspired by this thread http://forums.parallax.com/showthread.php?127976-Spin-Simulator&highlight=spin+simulator
I'm also pulling in ideas from Zog, Catalina, the Zicog simulation and Bean's Basic.
The limitation, as always, is that we don't seem to have any open source code. But, that didn't stop those four projects listed above.
So why can't we write this ourselves?!
We already have "Big Pasm", in the form of LMM. It takes one cog and you can keep writing pasm code until the memory runs out.
So it seems to me that what we need is something to translate Spin into LMM, and I see two ways to do this. One is to translate to the bytecode that the Spin interpreter uses. I see most of this as possible but there are still bits missing, though the Spin Simulator might be able to plug those holes. Especially the bit about the flat 32 bit memory map.
The other option I see is Bean's Basic option where you compile to pure Pasm and don't run it through any simulator at runtime. This may well produce faster code and save a cog as well.
There have also been posts in the past about better IDE optios, eg auto indenting, autocompletion of all the methods available in an object once you type the . and other things. Maybe a new/improved IDE is an option.
But, the IDE may not be the highest priority.
So here is the idea. Take some existing spin code, typed into a text editor, or the Proptool or BST. Keep typing code, 500k or more. None of the above will be able to compile this, but they can at least save it and you will be working in a familiar environment.
Then run a simple batch file to compile this program to LMM pasm.
I've been thinking about how it might be possible to write a compiler using the collaborative input from many people, each of whom may have expertise in different languages, and I have come up with a concept that I'd like to run up the flagpole and see if anyone salutes.
The idea is to create a multipass compiler. I wrote a two pass compiler once, but it should be possible to do it with more passes and hence break the task up into smaller pieces. Each pass works on text files stored on the hard drive. The input is text file(s) and the output is text files(s). One pass could be written in C. The next in Java. The next in Basic. It does not matter as long as the program can read text files and write text files.
An example. The first pass might be to take the original Spin file, work out the names of the sub objects, and translate all instances of variables into a unique name that identifies that variable. For instance, in the main spin program there might be a variable "i", and in a sub object "serial" there might also be a variable "i". These are different variables, so you go through all the Spin code and replace "i" with "main.i" and in the sub object with "serial.i"
The next pass might simply combine all the sub objects and the main object into one big text file. Each pass is simply reading text files and then saving new text files.
The next pass might group all the CON parts together, and all the VAR parts, much as the Proptool compiler seems to do. Maybe this is not necessary?
The next pass could put all the CON values into a text file list and save as a separate text file.
The next pass could take all variables and create a list of those variables along with a token that goes with those variables, eg "myvariable" becomes "variable1" in a list. This list is now in a separate text file for use later on.
The next pass would do a global replace to put in "variable1" "variable2" in the code instead of the actual names. Or maybe just token numbers v1 v2 v3
Next pass would take all nested bracket code and expand it out to separate lines. Bracket code seems to have a hidden long that is passed from the inside to the outside - we could use "result" or something to move data to the next line.
The next pass might take all "myvariable:=1" and put in standardised spaces eg "myvariable := 1"
The next pass could search for text strings like "==" and ":=" and tokenise those.
The next pass could look for indents and convert (temporarily) into a format easier for a compiler to understand, eg nested "{" and "}" like in C.
And finally, we can start to do some compilation, eg variable1 := variable2 translated into LMM pasm.
We need a pass to search for loops, eg "repeat" and convert into conditional jumps.
And we need a pass to do some other complicated things like a cognew, which now would take the next 2k of data from external memory, move it to a fixed common location in hub memory and then execute pasm code to move it into the cog.
Now, the entire project is extremely daunting. But each of the above bits of code are not so hard. And they may well be a lot easier if anyone is free to program in their own language.
So I suppose the big question is whether there is enough demand for Spin programs that are more than 32k.
Thoughts, input, critique, offers of help, showstopper comments and even flames would be greatly appreciated.
Cheers, Drac
Comments
(Having said that the Prop II will have us writing somewhat bigger Spin apps)
So, I'm seriously not up for this but:
1) Your first decision might be whether yo want to compile to spin byte codes or LMM. Given that LMM requires 4 bytes to be fetch from memory for each op and byte codes only requires one it might be more efficient to use bytecodes when our memory interfaces are so slow and only byte wide. On the other hand LMM gets the work done faster so it might be a wash. As an example Catalinia uses LMM and Zog uses bytecodes, turns out that Zog is still slower in most cases although that is not helped by the nature of the byte codes and the fact that Zog keeps his stack/data in ext RAM rather than HUB.
2) Where do you want to put your stack and data? Keeping it in HUB would of course be quicker.
3) I'm sure you don't need so many passes to write such a compiler. Have a look how Jack Crenshaw builds a Pascal like compiler here http://compilers.iecc.com/crenshaw/ A fascinating read.
No problem you can do COG new in C from Catalina and Zog and perhaps in other languages so I guess big Spin could do it as well.
BUT: I will assume that only one COG will be running one LMM or bytecode interpreter for big Spin code in external memory, otherwise things get very slow.
1. Converting Spin bytecodes to LMM PASM
2. Writing a Big Spin intepreter that can access more than 32k/64k of memory
3. Writing a Spin compiler that can create programs greater than 32k in size
The first idea is probably the easiest to implement. LMM PASM code could be written for each Spin bytecode and a simple program could be written that replaces each bytecode by a LMM PASM snippet. There is not much advantage to this since it would run slower and require more program space. The current interpreter does expend extra cycles to decode a bytecode, and many simple operators execute a lot of NOPs because they are combined with other simple operators. So this approach may break even in terms of execution time for certain operators.
The second idea would need to address the 16-bit bottlenecks in the current Spin interpreters. These are related to the memory pointers pbase, vbase, dbase, pcurr and dcurr. These are the basic memory pointers that determines where a Spin program's variables and instructions are located. These five values are stored as longs in the interpreter, but they exist as words in the 16-byte binary program header and in the call frame. The call frame contains the values of pbase, vbase, dbase and the return address used by the calling method. This would need to be increased from 16-bit values to a larger size. We could continue to use the 16-byte binary program header and add an additional "relocation" variable at run time. This would allow us to run 8 64K programs in 512K of RAM, where each cog uses 64K. A memory manager could be used to allocate and access much more memory beyond the 512K size.
The Spin interpreter would need to change it's memory access to accomodate external RAM. It currently contains 36 places where it uses a rdxxxx or wrxxxx instruction to access hub RAM. These 36 places would need to be modified so that they would call centralized routines, which would determine whether hub RAM or external RAM is being addressed. Of course, this would increase the size of the interpreter where it wouldn't fit in cog RAM anymore. A modified version of SpinLMM could be used to move a portion of the interpreter to hub RAM, which would provide more space in the hub RAM for the hub/external RAM access routines. This is certainly a doable task.
There are two approaches to a Big Spin compiler. Either a open source compiler needs to be developed first, or we would need to convince the owners of one of the three closed source compilers to add this feature.
I've taken the spin interpreter and put the main bytecode logic into HUB and load to COG space on demand. The math operations remain in COG without need for overlays. I've changed key read/write methods to use a cache interface running in a separate COG. With these changes and the 64KB limits, i've been able to run simple programs such as pin toggling. All code, stack, and data live in external memory. This implementation runs about 1/3rd the speed of spin for pin toggling.
As Dave mentions, the startup pointers are 16 bit and any program would have 64KB memory space until a BigSpin compiler is available. One of the other complications for getting more than 64KB is the object call lookup. Of course there has to be a way to address HUB memory from BigSpin code which will also require a compiler assist. As mentioned more than one interpreter accessing external memory is a problem.
Given all that, I propose:
- Only one BigSpin interpreter VM be runnable at a time
- The BigSpin compiler should put code start at $10000 to differentiate code/data from HUB access,
One possible consideration for a faster "not so big spin" is to use HUB for stack and variable storage. The compiler would have to be changed even more for this option, but it would allow executing from EEPROM or other non-volatile storage without severe language limits. Of course "not so big spin" program size would be limited by HUB memory because of stack/data and would allow fewer memory buffers for video, etc....change startup variables to be longs, and change method and object entries to be longs.
By contrast to "not so big spin", the BigSpin version would not keep stack/variables in HUB and allow using up to 2GB of external memory which I guess is more like HugeSpin.
The current compilers could be used to generate binaries for individual objects that fit within 32K. Each object would be compiled and linked with stubs generated from the objects that they reference. The stubs would contain small routines that would perform long jumps to the correct point in another object. It's similar to the technique I use with method pointers, except that the pointers would contain 32-bit values. The stub routines would just add 32-bit offsets to the pbase and vbase to get to the right place in the real object's code space.
I imagine Dave has solved some of this with the simulator.
I like Jazzed's idea of starting code at $10000.
The comments about 16 bit bottlenecks in the spin interpreter raises some issues I was not aware of. So... if there are problems with the existing spin interpreter, is it indeed going to be easier to compile to LMM opcodes?
Here is an idea. You have Big Spin running entirely in external memory, with its own stack in external memory. Yes, that is a bit slower. But... the upside is you can then be also running totally standard spin from hub memory, with multiple standard spin interpreters, thus preserving the advantage of parallel processing and also preserving all the existing obex code that runs spin in multiple cogs. Another advantage could be that the cogs could all be loaded from the Big Spin code, thus saving up to 14k of hub ram that would be used for cog code. You would need a small bootloader that would first load Big Spin to external memory and start its cog. Then that bootloader would chain a standard binary into hub ram.
So for speed, in parallel you can have slow but big spin, medium and standard spin and fast cog code. Then the speed limitation of serial ram does not matter so much.
I like the idea of code fragments. I'd also like to build those code fragments right from the start as 32 bits for everything, even if it means it takes a little more code space. With external memory, code space is no longer so precious. So calls to PUBs right at the end of the program exist in a flat 32 bit memory space.
Re jazzed's comments on the eeprom, I'd take it as a given that there is an sd card.
I need to check out the spin simulator. I just had a crazy idea about writing this big spin compiler in the spin simulator using spin.
heater
The problem I see is that while C exists, it can't use the vast majority of Obex code that is written in Spin. I've been working on porting a couple of VGA objects into C and it is not easy. I got one working, but the other one is very complex. It involves multiple shells into homespun then recombining the code back in as hex array data, and also copying C constants and translating them into CON data within each Obex code so that both languages can use common hub locations. RossH has ported a few and integrated them into the C language, but invariably one seems to need Obex code that hasn't been done yet. I think I could port maybe one Obex a month, but the whole Obex is daunting, and it is not easy to automate either.
The link to the pascal compiler is very interesting.
Probably not as many passes are needed through the multipass compiler - this was just an idea that might mean more people could contribute.
I'm thinking of something similar to Bean's compiler - mainly because attempts in the past to shoehorn big spin into the existing interpreter seem to have stalled in the past. But maybe the 16 bit problem is solvable in other ways?
I don't want to put you off - "Big Spin" is an excellent idea - but why waste your own time re-inventing the wheel? Parallax must already be a long way down the track to developing their own version of Big Spin, since they will need it for the Prop II - otherwise, how are they going to make effective use of the 128kb of Hub RAM on that chip?
Parallax cop enough flack as it is for expecting their customers to use Spin - they'd cop a whole lot more for releasing a new flagship chip without a language that could make effective use of of its entire memory space. So if its true that the Prop II is going to be released sometime this year, then Big Spin must already exist - at least in prototype.
Also, the last thing we want is a plethora of different and incompatible variations of Spin floating around - I think this is a positive benefit from Spin not being "open source". Even though we have other Spin compilers, Parallax still rules the roost - and this is a significant reason why the Spin OBEX succeeds where a C OBEX might not - no-one would write a Spin object intended for re-use that only works with one particular non-Parallax Spin (or C) compiler!
If I were you I'd wait till we see more details about what changes Parallax intend making to the existing Spin language to support the Prop II (I seem to recall a thread discussing this very thing a few months ago). When you know what the new language looks like, then "back port" a version to the Prop I. I very much doubt Parallax will be interested in doing this since every Prop I platform with XMM RAM - and you will need XMM RAM to run it at all! - is completely different.
On the (unrelated) subject of re-usable C components, I think you may want to rethink your approach before trying to reproduce the entire OBEX in C! While Spin users are likely to make heavy use of OBEX code, I'm not sure the same is true of C users. In my experience, C users make much less use of re-usable component libraries, and are much more to creating their own. Not necessarily just because they want to - but because they usually have to.
In the meantime, I'm with Heater - use C for large projects, and Spin for small ones. After reading or writing more than a couple of pages of Spin at a time, I begin to understand why Parallax called it that!
Ross.
The problem we are all trying to resolve is one that really requires one big program and up to 7 normal little programs for our 8 processor prop. I see that solution as C (or any other language) running mainly from the extended memory (sram/eeprom/etc). We already have some of those solutions but they need a better interface to the other spin & pasm programs.
So, what I see is some way to segregate the 7 spin/pasm processes from the main process, written in C or whatever. That is why I chose a second prop, but there are of course other ways. I believe a standard interface is required between the main process and the other standard processes, with the ability to import/export the interface so that the main process can be compiled completely separate from the other 'standard' processes.
The PASM Spin interpreter executes a heck of a lot of instructions to decode a Spin bytecode and execute it. The Spin instruction ldli0 (Load Long Immediate Zero) required 25 PASM instructions to load a 0 onto the stack. This operation could be done in as little as 4 PASM instructions, which is a factor of one-sixth. It may be that Spin bytecodes converted to PASM LMM would run at about the same speed as interpreted Spin.
Hi Humanoido,
My point is that Parallax has to develop Big Spin for the Prop II anyway - so we may as well wait till they release some specifics about the language changes. I'd be willing to bet they have already made a start on it.
To develop an incompatible version of "Big Spin" for the Prop I could end up being (at best) a waste of time, or (at worst) quite counterproductive.
Mind you, plenty of us do sillier things!
Ross.
{ld, st, la, ex}{l, m, w, b}{l, v, o, a, i}{c, x, 0, 1, m1}
Each field has the following meanings:
{ld, st, la, ex} - Load, Store, Load Address, Execute
{l, m, w, b} - Long, Medium, Word, Byte (Medium is 3 bytes)
{l, v, o, a, i} - Local offset, Var offset, Object offset, Absolute address, Immediate
{c, x, 0, 1, m1} - Compact, Indexed, zero, one, minus one
The third variable on a the stack would be loaded with "ldllc 12" or "ldll 12" The ldllc generates a one-byte code and the ldll generates two bytes. An indexed byte in the DAT area would be stored with "stbox $111".
Dave
I'm not sure why BigSpin would have to be incompatible with Spin. The entire Spin language specification can be supported with one exception. The only exception/real complexity at this time is multi-cog BigSpin which is solvable but impractical. As DR_A seems to suggest, one BigSpin that could start normal multiple Spin COG threads would be a requirement. BigSpin will obviously be able to start PASM otherwise it is useless.
BigSpin is interesting because Propeller people program in Spin - just check heater's poll if you want proof - everyone should participate in that poll to possibly prove otherwise.
@Dave thanks for your explanation. Some of the fields were confusing.
SpinLMM uses 8 instructions times to execute an LMM instruction, so this would take 232 instruction cycles to execute, which is a bit slower than the interpreter. The hand-coded PASM requires 116 bytes versus the original Spin bytecodes, which is only 10 bytes. So it seems like there isn't much advantage to converting Spin bytecodes to LMM PASM. The only practical use would be to access external RAM.
However, this might be useful for Small Spin, which would execute entirely within a Cog. Small Spin would still use the stack, and it could perform recursive method calls. It would also have access to VAR and DAT variables. The only restriction is that the program would have to fit entirely within a cog.
Small Spin could be optimized a bit by not using the stack unless it is absolutely necessary. The Spin statement x := y + z pushes y onto the stack, then pushes z onto the stack, executes an add instruction that pushes the result onto the stack, and then pops the result off the stack and stores it into x. This could be optimized to the point where temporary registers are used instead of the stack.
It may be possible to do further optimization where some Spin variables could be kept in registers as well. This would reduce the use of the stack even more. The nice thing about this is that PASM driver code could be written in Spin instead of PASM. There could be a Tiny Spin mode that doesn't use the stack at all. This wouldn't allow for recursive method calls, but it would be much faster.
Dave
SPIN LMM might be worth pursuing anyway as an alternative because the original SPIN spaghetti VM bytecode handlers do not execute all code since up to 4 functions are packed into one "jN" routine. The reason this is valuable is the overlay load time for BigSpin as I have it now may be longer than given LMM handlers would execute. Too bad Cluso99 is not interested in this topic.
1) Start another cog and run normal spin in parallel
2) Change from serial to parallel memory (8x speed increase?)
3) Use PASM
4) Use inline PASM within Big Spin
5) Open up a cog, load it with some PASM and then shut it down again
Option 4 ought to be an integral part of LMM Big Spin anyway
Option 5 opens up some new ways of programming. Because memory is no longer so precious you don't have to worry about using up 2k of memory space for every cog that is loaded.
As for waiting for the Prop 2, I'll probably get in trouble for saying this, but the Prop 2 has been "just around the corner" for as long as I have been on the forums. It isn't here now so we can't use it. And the world of microcontrollers changes rapidly. I just paid $115 including shipping for an Apad which runs Android and looks like an ipad. Then I paid another $39 for a licence for http://www.basic4ppc.com/index.html screenshot: http://www.basic4ppc.com/android/screenshots.html and now it is possible to program the device in VB.net. Other languages are possible too. Gorgeous display, touchscreen, wifi, USB, Cat5 network, speaker, microphone, heaps of code space.
The catch though is the Apad runs rather warm whereas a Propeller runs almost cold, and that is most relevant for battery powered applications. And a Dracblade plus 20x4 LCD plus freebie CRT VGA screen plus $3 keyboard is still cheaper than an Apad. So I'm keen to see what we can push the Prop to do with larger programs.
@Dave, can I just check - you have a simulator running in a flat memory space, right? Presumably each line of the simulation has to do some sort of translation to get the simulation to work. For example, taking that wonderful Pascal compiler tutorial listed earlier, to decode a := b*c / (d - e) you need to know BODMAS and you need to push things on and off a stack in the right way. If a simulator is doing this that is 3/4 of a compiler, so the next step is to translate this into something the propeller understands. This would be 1) spin interpreter bytecodes or 2) PASM for LMM. Which of these do you think will b easier.
The fastest relative implementation for a given set of constraints should be used.
Any luck looking at that Spin compiler?
I agree. Presumbably that is going to be code that is compiled to pasm rather than interpreted? And using parallel rather than serial ram?
Ok, let's push it to the extreme. Compiled and optimised code, running in LMM pasm with external sram. With latched ram, you can optimise it even further if you do things in blocks of 256 bytes so only one latch is changing most of the time. Or use Cluso99's idea of joining propeller pins to sram pins with no latches. That will be almost as fast as real spin.
Sounds like a reason why LMM is better?
I'm more inclined towards LMM Spin at the moment, but on the other hand, if most of the work has been done with interpreted spin, that might speed things up. On the other other hand, several discussion threads in the past have faded away when the complexities of interpreted big spin are looked at in more detail.
Which one do you mean? I'm deep inside the Pascal compiler at the moment. Maybe the answer is going to be to build this from scratch, like Bean's Basic?
That's why I suggested waiting to see how Parallax plans to extend Spin for the Prop II - the main benefit of Big Spin would be if it makes Spin programs that would normally only be executable on the Prop II (because it its larger memory) also able to be executed on the Prop I (albeit more slowly).
Of course people program in Spin - its the language (the only language) officially sanctioned by Parallax!
My own opinion is that the Spin language doesn't scale well, and is quite unsuitable for the size of applications that will be possible on the Prop II (or the Prop I with Big Spin).
But if nothing else, Big Spin would give us a terrific opportunity to find out once and for all!
I think it is slightly easier to write a Spin compiler that generates Spin bytecodes. However, it would probably be about the same effort to generate LMM PASM codes. I played around with this idea about a year ago, but I didn't get very far with it. I spent a couple of hours today modifying a copy of the SpinSim VM to generate PASM code. I'm hoping to spend another couple of hours on it and try converting a small Spin program into PASM. I'll post some results once I get it working.
Dave
I agree. The motivation for this is the ability to drop in any Obex code into a program. Indeed, if you had a big enough memory chip (eg jazzed's chip), you could put the entire obex in the memory chip, one after the other. The only coding difference would be that you can't have more than 8 cogs running. But all the obex code I have looked at all has a cogstop method, so if you want to free up a cog, stop it, start a new method etc. Your 'main' program will need to keep a record of what is running where which maybe you don't need to worry about if you only ever load up 8 cogs.
From a compiler point of view, the jumps are bigger. eg you call a PUB that is 500k away from where you are. But in a flat 32bit memory space that should not be hard in LMM pasm.
So: -
no 'special' code.
no 'special' cogs, as the program plus its stack plus any other internal thing are entirely in external memory.
no 'special' hub access that could potentially clash with existing obex code. (of course if the user wants talk to hub ram that is their program)
re one platform, the default platform is maybe the C3, but ideally the LMM driver includes the few lines of code needed to talk to any external ram
re 'one compiler', I think there are advantages to a command line compiler that deals with txt files. Keep it modular and open source. If it is command line driven it can be integrated easily into any existing IDE with just a few lines of code.
@ Ross
Can you explain a bit more?
Here's a beginning list of "usage" so that we may understand the constraints of "implementation."
BigSpin is by necessity a superset of Spin which among other features will include:
- Should be syntax compatible with SPIN + some BSTC/Homespun extensions.
- Source code files will use .bspin, that is: SomeDemo.bspin.
- Will allow running normal SPIN code to run in HUB space as required.
- Will allow starting normal PASM code as required.
- Will be loaded and run in external memory by a normal SPIN program.
- Will maintain variables and stack in external memory.
- Will have access to normal HUB via HUB address + $10000000 (or other).
- May support simple BSTC or Homespun extensions.
- Allow non Propeller Tool extensions to be easily converted.
- May support by a simple read/write external memory interface.
- May support a COG cache access external memory interface.
- A modified compiler will generate code and data sections at a start address > $FFFF.
I'm sure there are more "usage" requirements that I missed. Please feel free to add them.I see DR_A has started a requirements list too. I think the list is compatible except for maybe BSTC/Homespun extensions which I really think is important to have in their primitive forms.
Here is a (partial) list of things you might consider adding to "Big Spin" if you were actually free to develop a replacement for Spin itself. Most of these things don't matter a jot in programs up to a few thousand lines of code - but they all contribute to making larger or more complex projects needlessly difficult (in some cases nearly impossible!):
- Spin has no memory management. It is hard to predict (or control) where in RAM a variable is going to end up. This will matter much more in Big Spin as you want to be able to specify whether things are in internal or external RAM.
- Spin has no library management. This makes it is very hard to organize components even within your own program - everything has to live in the same directory (not exactly true, but nearly so).
- Spin has no "externals" and no linker. This means spin programs or components cannot be distributed in binary form, and also that every program has to be recompiled from a complete set of source every time. It also makes it difficult to interoperate between Spin and other languages.
- Spin has no conditional compilation - jazzed suggests adopting the Homespun/bstc extensions - but if Parallax themselves doesn't do so, then most existing Spin users (who use the Parallax tools exclusively) are left to mess around with preprocessors etc. Most of them will simply not bother with anything so arcane - and there goes your user base!
- Spin has no include file processing - same consequences as previous point. We've all discovered that just being able to include a common "object" is not good enough - e.g. you can't include a PASM subroutine and then call it from existing PASM code. Also, including objects incurs overhead, whereas include files do not.
- No macro definition capability - same consequences as previous point. Even the simplest macros can save a heap of complex and repetitive coding, and also improves the efficiency of the resulting programs. It makes large programs much easier to prototype and to maintain.
- Spin has very small limits on numbers of symbols, and the size of symbols - same consequences as previous point (although this is an implementation rather than a language issue). Most Spin programs are currently small enough that this doesn't matter. But these limits would become major problems in larger programs. Of course, these limits can be increased - both bstc and Homespun do so - but again unless Parallax themselves does so then it means your program will compile only under one particular compiler.
- Spin has no function (method) pointers. This makes it much more difficult to implement a whole bunch of elegant programming techniques designed to make large complex programs more maintainable. Of course, many other languages exclude pointers to functions as well - but they generally provide other techniques to make up for the lack (e.g. inheritence, templates, interfaces or generics).
- Spin has no structures or pointer types. You have to simulate these on your own (and be aware of Spins propensity to re-order things). In a small program (where you can keep track of all references yourself) this is ok. In a large program, this is a maintenance nightmare.
- Spin has very few variable types (ditto constant types and enumerated types) and no type extension capability. Languages that are "type poor" only really work well when you can "see" the whole source at once. Once a program becomes large enough that you struggle to remember which file contains the explanation of the use of a particular data type or structure or pointer (because none of it is evident in the code itself) then such languages are a positive menace.
These are the main ones - I have tried to exclude things that are just my "personal preference".Ross.
My spin interpreter will give you a 20+% increase in performance and the maths get a real boost. Also, there is some cog available which I had started to use to improve further code. It does however use hub for the bytecode decode table and this is where it gains some speed boost and space.
Now, here are some thoughts to improve the performance of user spin bytecode (i.e. the output from the compilers). However, to do this would require access to the spin compiler code (homespun, bst or Proptool and note I believe mpark has released the source of the prop sphinx compiler. So, a post compiler stage could rip through the bytecode and remove the push/pop pairs that are unnecessary. This would give another performance increase.
Most of the items on your list are not discouraging and has a workaround. The number of variables and methods per object is the only one that comes close to a problem and would need to be fixed. Even function pointers are possible more than one way in standard Spin.
Also I tested #include "somepasm.inc" as in-line PASM in Homespun and it worked just fine for me.
I did not test it extensively with large files though.
As you suggest the "typical" Parallax customer probably wouldn't use BigSpin.
Many of the a-typical adventurers would however.
But this situation will change when the Prop II arrives - some of these issues will have to be sorted, or Spin risks becoming unusable as the mainstay programming language of the Propeller. We're not talking here about programs that are merely 4 times larger than current programs (assuming the Prop II has 128kb of Hub RAM). Most of this RAM is currently used for non-code anyway (e.g. buffers). We're talking about a chip that is capable of running programs between 10 and 100 times larger. Do you really think Spin is up to this? I don't!
If we can convince Parallax to adopt at least some of the existing solutions (e.g. those which are already implemented in the non-Parallax tools) then that will help - otherwise "Spin II" (I will use this term to distinguish "whatever Parallax does to Spin for the Propeller II" from "Big Spin") will be a problem.
Exactly - so who is the target market? The "a-typical adventurers" are currently using other languages anyway. Are you suggesting they would migrate back to Big Spin if it were available? Why would they?
Ross.
The key word there is 'superset' Big spin ought to be 100% backwards compatible with standard spin. But it can include other things too if you want and that list is a great list. I'd support all those things. And Ross's list is great too, because those are things that can be fixed.
I particularly like jazzed's idea "Will be loaded and run in external memory by a normal SPIN program."
You could take that one step further and allow normal spin to load and reload big spin programs.
Maybe I'm wrong with this statement, but I assume all the compilers out there still have bits that are not open source? So if one has to start from scratch, that pascal compiler earlier might be where I might start. A somewhat daunting, but not impossible task.
I think up till now the target market has been the adventurous ones, but I think that may be changing with the C3. We now have a nifty board with lots of memory and we have Spin that maxes out at 32k and at the moment people can't use that external memory to store a program bigger than 32k.