PropellerForth
Cliff L. Biffle
Posts: 206
Over the past week or so, I've been working on PropellerForth, a mostly ANS-compliant Forth for the P8X32 chip. This is a trial balloon to see if there are any Forthies in the audience. (Or, for that matter, anyone who would like to have an interactive development environment on the Propeller itself, and is willing to learn Forth to get there.)
RATIONALE
I really dig the Propeller, but I don't really like writing large applications in assembler. For a while, I considered developing a typesafe, optimized version of Spin (which has very nice attributes, like the component [noparse][[/noparse]n
RATIONALE
I really dig the Propeller, but I don't really like writing large applications in assembler. For a while, I considered developing a typesafe, optimized version of Spin (which has very nice attributes, like the component [noparse][[/noparse]n
Comments
Interactive Forth sound cool.
I worked with RSC-Forth long time ago, I will be happy to work with this kind of solution.
Eviatar
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Paul Baker
Propeller Applications Engineer
Parallax, Inc.
I think if you could make this stand-alone, it would be especially interesting. Host computers are always orphaning projects. One that would still be accessible 30 years from now might be useful.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Chip Gracey
Parallax, Inc.
Who's your father? Chuck Moore?
Seriously, this looks really cool! Forth can be an excellent bridge language between the user and the metal when speed is of the essence. It's often gotten a bad rap for being unreadable and, in truth, it's very easy to write unreadable code with it. But that's true of almost any language.
One thing I'd like to see, unless you're an ANSI Forth purist, is a wrap-around expression stack. Stack maintenance is the bane of Forth programmers. It would be nice just to be able to abandon a non-empty stack when you're done with it, rather than having to remember how much stuff to pop off. Implementing the stack like a queue might solve this problem, in that stuff that's not needed eventually just gets pushed off the end. Also stack overflow and underflow go away -- although this would be a bad thing for debugging. An alternative would be what Postscript does with its mark and cleartomark operators. Postscript is strongly typed, though, so a mark can be unique. In Forth, you'd have to keep separate track of any that were on the stack. (It's been awhile since I've done anything with Forth, so maybe this has been addressed in a later standard.)
Anyway, good luck with the project! I'm anxious to see more! And I've already got an app (four-axis mill) I wrote years ago in Forth for Zilog's Super-8 that I'd love to port to the Propeller.
-Phil
That's the goal. It's already standalone if you have a serial terminal.
I'm working with your TV and VGA drivers, trying to embed them. (As I mentioned in a previous post, the main stumbling block is propasm's lack of constant expression support.) I'm not confident that Forth will be fast enough to drive the display itself -- not at 1.4m words/sec. It'll have to be native code.
My ultimate goal involves TV, PS/2, and SD mass storage -- but I won't overcommit until I get the thing shipped.
No, though they do have stories about each other.
Forth is the essence of what we now call "domain-specific languages." It can be evolved to look like just about anything. If a programmer doesn't put thought into it, it can become far worse than Perl.
But in skilled hands, it produces some of the most elegant code I've seen. (Yes, I would rank it with or above Smalltalk.)
I'm not an ANS purist, but that's pushing it.
It's pretty straightforward to define words that reset the stack to a previously stored location, like in Postscript. They're used in some of the try/finally type constructs for writing guard blocks.
(Incidentally, what the hell were you doing writing Postscript? You've passed even my geek threshold here.)
There are other approaches -- ANS forth supports local variables, for example -- but in general, if stack management is becoming a chore, your word definitions are too long. This is something I keep relearning over and over, and never really applying.
Hey, now. I don't think PropellerForth is so bad.
The directory on my disk is actually called P8x32Forth, so it can certainly be worse....
*Peter*
When PropellerForth is released (after I do some more testing and iron out some kinks), it will be under an open source license, and hosted on one of the collaborative development sites (probably Google Code, but possible SourceForge). At that point, I'm happy to accept enhancements, bugfixes, additions, and the like from anyone on the 'net.
I will be passing out prerelease archives on the forums shortly, and I encourage you (and everyone else) to bang on them and tell me what I screwed up.
This will be wonderful. I have always liked the interactive development in Forth.
Really looking forward to getting my hands on it.
Mike
-Phil
Chris
I've got most of the useful words from the ANS core set working, and have been doing a lot of reworking and simplification.
What Works:
- Interpreted interactive exploration (including defining words)
- Most necessary primitives (including hub operations and doubleword multiplication)
- Host "compilation" of words into the image (really just a Ruby preprocessor to assembler, I'm not using a Forth cross-compiler).
- Most interesting Propeller-specific functions (I/O, control of the counters, Cog control)
- A small subset of the ANS String word set.
What Doesn't:
The main stumbling block at this point is storage. I don't have any extra EEPROMs strung off the Propeller and the SD module I wanted (at SparkFun) is out of stock. Without mass storage, there's no realistic way to store, compile, and recompile the sources on the Propeller itself -- so this is taking priority over other self-hosting issues (such as keyboard and TV).
I'm considering writing elementary serial file transfer support (XMODEM or ZMODEM) and faking it that way, but we'll see.
Other elementary things that aren't working include:
- Multitasking, either within a Cog or across Cogs. (Not all the USER vars are being accessed through USER currently, so I can't separate tasks. This should be fixed this evening; I'm most of the way there.)
- Handy programming tools like DUMP, FORGET, and SEE.
- Direct EEPROM access.
I'm doing this primarily for fun and to learn the architecture, so I'm hesitant to involve others too early; however, I encourage interested parties to contact me. I'll have code for testing soon.
Just out of curiosity, how are you storing dictionary symbols? The first three letters and a length, or the whole enchelada?
Also, my own preference as a potential user would be to forget the Propeller-resident editing and file ops for now (which I'd likely never use), in favor of a complete and rich set of native words, including strings and conversions.
Thanks,
-Phil
Length+7, case preserved (though user input is case-insensitive). My older Forths tended to use length+3, but I kept running into conflicts; a lot of Forth-83s used Length+7 and I think I've got the RAM to spare here.
I am currently storing all symbol information in the dictionary, but I plan to split it out so an image can optionally be stripped. (My embedded applications rarely use EVAL or INTERPRET.)
Well, that's certainly where I'm headed now, but it's mostly for lack of spare parts.
We currently have decent String support (.", S", COMPARE, and some other goodies). I'm eschewing counted-string support in favor of the c-addr +u strings that the ANS folks like -- of course, any user could add counted string support trivially.
I don't have any of the pictured numeric output words working (<# #>); I only recently got division working, so base conversion and formatting were pretty far down on the list.
So, keeping in mind that the only way to compile Forth properly is Forth, how would you want to do your development? I can certainly document the dictionary layout and primitives, which would allow you (or others) to write a hosted cross-compiler in some other Forth.
The way I'd prefer to develop for a system like this is to use an editor like UltraEdit. If you had a shell command which then uploaded the raw source to the Propeller (e.g. fterm -f mysource.forth), your resident firmware could compile it as it came in. fterm would dump the stuff that comes back to STDOUT (which UltraEdit captures). Once uploading is complete, it could pop up an interactive terminal window (maybe only if a -t modifier were present on the command line) for testing words, etc. Anything entered or received would also get dumped to STDOUT.
Forth itself would have to have a built-in word to transfer the compiled image to EEPROM. That way it could be done interactively in the terminal window. Or the "store" command could be placed at the end of the source code.
-Phil
Post Edited (Phil Pilgrim (PhiPi)) : 11/6/2006 8:02:26 AM GMT
UltraEdit?! Holy Smile, is it still around? Neat! I actually remember it from back when I was on Windows!
I'm a vi guy myself for everything but Java (mmm, Eclipse), so I can't blame you.
This is pretty much the hack I was proposing when I referenced XMODEM, though I was going to use an actual transfer protocol (with error correction and such) rather than just piping it.
I could hide that behind the script, of course. I'll see what I can do.
For me, of course, I'd rather write all my code in a Forth, even if it means I have to give up vi. Filing in Forth sources can have a lot of unintended side effects, since the source includes executable commands (or, really, is executable commands). If there's a syntax error halfway through, you can't roll the previous changes back.
(Sure, there's FORGET, but that's not complete. This pickiness probably comes from my day job working on transactional systems.)
Creating a word at a time in Forth helps prevent this.
I would even be happy to contribute in some small way to the programming, if you felt that would
be worthwhile.
The SD card setup is just the SparkFun socket with a .1" header soldered to it so it can be plugged
in to a breadboard, along with all the resistors needed to make it all fly on the demo board.
You can email me at rokicki at gmail.com if you are interested.
(Actually at some point it might be interesting to do a Bay Area Propeller Group or some such,
maybe in conjunction with the robot club or maybe separately.)
I think XMODEM would be overkill here and waste too many precious Propeller resources. But I do get your point about unintended side effects. One bad word, and you get a whole cascade of errors. Perhaps the PC-resident uploader could me made just a little bit smart and stop uploading when it saw the first error. Or define a Forth word that sets a flag that terminates compiling and enters flush mode on the first error.
In this environment, I'm not terribly concerned about rolling changes back, or even using FORGET, since I can just make my corrections in UltraEdit and upload the whole thing again. If I wanted to be interactive, I'd just run fterm -t without the -f and enter stuff by hand.
The main point, I guess, is to keep from having to store any raw source in the Propeller. You'll run out of room way too soon. Hence the upload-with-compile-on-the-fly suggestion. This is the way I developed with Forth on the Zilog Super8 (may it rest in peace), and it worked quite well.
-Phil
Rock on. Which SD board do you have -- the raw breakout board, or the DOSonCHIP FAT controller? (I'm looking at the latter; all the automation of the GHI controllers, with a less stupid protocol. I'm trying not to have a FAT implementation on the Propeller if I can help it.)
I'd be interested.
XMODEM is remarkably simple, too much so for most purposes. It's either that, or reinvent it (write my own protocol with some basic handshaking and CRCs).
Using XMODEM would also let Windows users use HyperTerminal or whatever, and allow Unix folks to use standard tools. (Why anyone uses HyperTerminal for anything is beyond me, I admit.) I'm not super interested in writing a cross-platform terminal emulator -- or even a platform-specific one -- when there are so many good open source ones available. I'd rather just use Minicom.
The scheme I'm working on would have the PC upload a chunk of source, and the Propeller compile it; repeat as necessary. I'm using the word "chunk" to distinguish it from Forth's traditional "block" -- this would not be a 64x16 grid of characters, but rather a text segment with normal line endings.
At the most granular level, this would be a single colon def, variable definition, etc. More realistically, individual source files of a kilobyte or three could be sent. A simple script could automate your transfer program to upload a directory of source files in a defined order.
I assume, even with the XMODEM option, that a straight-ASCII, line-oriented, compile-on-the-fly mode will still be there by default, since this is Forth's normal mode of operation — right? Will the compiler be fast enough not to require serial handshaking at, say, 9600 baud, if it were being force-fed?
-Phil
Yes -- in fact, this works right now, though it has a couple deviations from ANS that I'm working on.
Currently the terminal is 19200 8n1, and works just fine. I've tried pasting sources into HyperTerminal (to simulate what most users will experience) and it mangles them really, really badly. I blame HyperTerminal; I'll try it in minicom when I get home to the hardware.
Feeding sources this way (from a competent terminal emulator) should be possible, but like most traditional Forth compilers, mine is O(size of dictionary) for speed -- each lookup must traverse a linked list. So, you might want to introduce a brief delay between lines (a few bytes worth of bit periods should be fine).
I'm going to play with this more tonight when I'm home, and I'll post results. (And also possibly pass out binaries.)
Currently, files can be "typed" into PropellerForth by using your terminal emulator's "send as ASCII" feature. The compiler will process your file a line at a time, storing no more than 80 characters in RAM at any point. This lets you build up the dictionary in-place without any mass storage.
It's less than ideal, but it works.
There's one main caveat: the terminal routines are in Forth, have not been optimized, and are not terribly fast. At 19200bps (the default terminal speed), you'll want a character-to-character delay of about 1ms, and a line-to-line delay of 50ms. Most terminal emulators support this.
Yes, this is annoying. Yes, it will be fixed.
Currently, it has the ability to execute no-argument words on other Cogs, but I'm working on cooperative multitasking within the Cogs as well. (They're based on the same mechanism, after all.)
In terms of ANS Forth, I've implemented most of the Core words, many of the Core Ext words, all of the Exception words, and a number of the Programming Tools words.
Here's a binary suitable for loading with Propeller Tool. It will bring up a serial console (19200, 8n1) on the standard pins (31 and 30, as used on the demo board).
Demo board users should see some LEDs come on and a prompt. Users of other boards beware: PropellerForth currently takes great liberties with pins 16-23, which drive debug LEDs on the demo board. These would be bad pins to use for, say, your atomic death ray.
I'll post some documentation soon, and work on cleaning up the sources for release. In the meantime, to help folks see results, here's some demo code to try. A note on the non-standard Propeller words:
- All the Propeller registers (CNT, CTRA, etc.) are defined as Forth words that return the in-Cog address.
- In-Cog addresses like this can be read with L@ and written with L!, by analogy to @ and ! for the shared RAM. (L is for Local, because C for Cog was taken.)
Paste this into your terminal emulator, or into a text file and send it as raw ASCII. Remember that my terminal code sucks, so set a character-to-character delay of at least 2ms, and an end-of-line delay of at least 50ms.
Once that's all entered, attach speakers and try:
Note: it will be loud. I'm not doing PWM yet, as you can see.
Last time I was interested was when the Harris forth chip came out.
Looking·forward to forth on the prop.
A lot of the early space stuff was writen in forth, they seemed to work without crashing into things.
Hmm Prop robot running forth?
And the price has droped on the chip too.
Hmm ballbot with prop controller?
*Peter*
I ran your code and it blow up my house by miss-firing evil rays of death from my demoboard.
Forth looks cool!
The sad thing is I never programmed in it before, nor do I know its syntax.
But I'd be excited to see some Propeller-IDEs.
Yeah, WORDS and a few others are part of a source file I'm manually loading each time I flash the board, as a test of the compiler. So they're not in the standard image. (FORGET is missing too, you'll note.)
I'm absolutely interested in whatever help I can get. I've been hoping to do most of the kernel fundamentals myself, since it's been a while since I've written one and it's about as far as I can get from my day job -- but that's mostly done now, so I'll be releasing what I've got, and I'd love your help.
I'm actually pretty impressed that you put together WORDS without any docs. Can you post your source?
Dude! What'd I tell you?!
Nobody listens.
Edit: Fixed some markup problems. mCode is silly. Why every CMS on the internet needs its own markup language is beyond me.
Remember that PropellerForth is case-insensitive for user input, so these words don't have to be entered in SCREAMING UPPERCASE.
( Edit: My snarky comment about every forum needing their own markup language? Well, one of the advantages of a standard one is that I know which HTML tags stop my lines from wrapping. The code tag here apparently does too. Fixed. )
The USER area is Forth's per-task storage area (C/C++/Java programmers would call it Thread Local). PropellerForth's current USER area layout is:
Offset
$00 BASE - task's initial number base for I/O
$04 STATE - task's initial interpretation (0) or compilation (-1) state.
$08 >IN - task's initial parse position in its input - should be 0
$0C SOURCECOUNT - number of bytes initially in task's input - should be 0
$10 SP0 - pointer to base of task's data stack
$14 RSP0 - pointer to base of task's return stack
$18 FIRST-WORD - execution token (right-justified in a cell) of the first word this task runs
$1C TIB - task's terminal input buffer
All these are cell-sized variables, accessible with @ and !, except for TIB -- which is an 84-byte character buffer.
For the purposes of starting a new task, one must
1. Pick a block of RAM for the task's USER area. PropellerForth leaves most of high RAM, up to around $7000, free; I tend to use $6000 for the second task's USER base.
2. Pick sections for your data and return stacks. The space required will depend on your task, and no, from that other thread, we can't statically determine it.
3. Set at least SP0, RSP0, and FIRST-WORD, like this:
4. The USER area is passed to a new Cog in its PAR field, which is a 14-bit address in the high-order bits of coginit's argument. Using this knowledge and 'KERNEL, we can construct the word:
5. And start the cog. I don't have coginit packaged into a word, because (as you've noticed) this process is laborious and stupid. PropellerForth's COGINIT word will do most of this work for you.
It'll display the ID of the cog you just started; you can stop it again by entering "the-cog-id-goes-here cogstop".
Happy hacking!