Propforth 5.0 preview
prof_braino
Posts: 4,313
Please note: As of 2012-03-04 proforth 4.6 is no longer the current release
Please the the new thread
http://forums.parallax.com/showthrea...e-for-download
- - - - - - - - - - -
Release is expected in January 2012
20111204 update
PropForthEEPROMfilesystemOptimizeKernel - EVERYBODY (that uses propforth) will use this
- PropForthBootOptimizeKernel with EEPROM file system supposrt built into the kernel.
THIS VERSION DOES NOT INCLUDE the SAVEFORTH word. INSTEAD, the file system has a file .boot.f which is automatically loaded at boot. Sp all your extensions are re-read and re-compiled at start up. YEP, now you don't have to worry about corrupting the kernel, as the kernel itself becomes "read-only"
From this kernel, we add the extension for SD support, and can load and re-load scripts of any size, on the fly.
Number base can be forced using the prefixes B, D, H for binary Decimal, hex. Other flag character, such as #, $, %, etc, are not standard, and not consistently used across all processors and environments, and I always though using weird characters was kind of silly when B, D, and H always make perfect sense.
b0101 = decimal 5
d010 = decimal ten, even if the base is Hex or binary
h123 = decimal 291
Also, the lower case z will be used to designate base 64. This allows a long to be expressed in six characters. This is used all the time with the assembler, as we will be dealing with lots of longs. But we don't have to worry about it, the assembler is supposed to take care of this automagically.
20111012 Preview 2 for 5.0
CHANGES
- hex values MUST be in UPPERCASE
- ability to force hex using 'x' prefix: x00 xADB
- ability to force decimal using 'd' prefix: d00 d123
- line buffer option: does not echo until CR is input. Used for ethernet connection, or for text file input
- backspace is x08, DEL xF7 is not used
- CR x0D is the line termination, LF x0A is not a line termination
- CR is ouput as x0D, setup teraterm to receive CR+LF, and to transmit CR. This was done to allow redirection of files easily to and from IO. (Telenet and serial terminal etc all behave the same.)
- kernel is modified and optimized
- added ST@ ST! - fast instructions to access and modify stack
- assembler optimized routine can be dynamically added
- assembler words are now dynamically loaded and cached; provides huge optimizations and space efficiency
- spin constants which allow:
specififying tx pin
specififying rx pin
specififying baud rate
specififying cog used for the console serial driver
specififying which cog starts up forth
http://code.google.com/p/propforth/issues/detail?id=53
There is now a multi-step process to build the kernel. This is to facilitate modifications for custom hardware (Hive, C3, dracblade, etc) and custom functionality. The intent is that modified kernels will still be compatible with the remaining propforth developer extensions. In the following,
"slow" means "as slow as 4.5 was". Uses the original, straight forth definitions without optimization. These are easier to modify
"fast" means forth routines identified as "bottlenecks" have been optimized in assembler. These are more difficult to modify unless the starting point is the original forth definitions.
PropForthStartKernel - MOST folks will NOT use this
- slow io, slow multiply and divide, slow to interpret and compile,
but very small and code runs fast. Error checking in place, but no error
reporting. This kernel has extra symbols which are necessary for recompile
but will be removed to shrink it still further.
PropForthBootKernel - MOST folks will NOT use this
- slow io, slow multiply and divide, slow to interpret and compile,
but very small and code runs fast. Error checking in place, but no error
reporting. This is minimum size. This kernel can be used for specialized
apps.
build 2011OCT04 10:10 20,400 bytes free in dictionary, 222 longs free per cog
PropForthBootOptimizeKernel - EVERYBODY will use this
- Assembler optimizations, fast kernel. Error checking in place, but no error
reporting. This kernel can be used for specialized apps.
build 2011SEP29 14:00 19,822 bytes free in dictionary, 176 longs free per cog
This kernel is the base for most kernels and applications.
ADDED 20111204 - PropForthEEPROMfilesystemOptimizeKernel - EVERYBODY will use this
- PropForthBootOptimizeKernel with EEPROM file system supposrt built into the kernel.
THIS VERSION DOES NOT INCLUDE the SAVEFORTH word. INSTEAD, the file system has a file .boot.f which is automatically loaded at boot. Sp all your extensions are re-read and re-compiled at start up. YEP, now you don't have to worry about corrupting the kernel, as the kernel itself becomes "read-only"
From this kernel, we add the extension for SD support, and can load and re-load scripts of any size, on the fly.
- the stuff below may no longer be needed due to the change above, but I did not sort through it yet -
*** Normal Dev/App system runs on ANY prop chip (EEPROM 32k and greater).
Load PropForthBootOptimizeKernel.spin into eeprom, reboot
paste contents of PropForthDevKernel.f into teraterm
saveforth
**** eeprom filesystem kernel. Runs on prop chip EEPROM 64k and greater.
BootOptimizeFsrdKernel
This kernel is designed to be a base system with an eeprom file system. The system comes up with a
read only capability, searches for a file called boot.f, and executes it. Holding down the ESC
key while rebooting prevents teh execution of boot.f
Modification should be done by modifying boot.f, and loading the files and capabilities as needed.
The first thing boot.t does is to define forget, and forgets the seriial driver. When needed the serial
driver - serial.f can be loaded from the file system. So if you add words and do a saveforth, you will mess
up this kernel.
The file system is read only, and fswr.f must be loaded before the file system can be written.
***** eeprom filesystem kernel + sd card file system
This now provides a large file system, boot modification should be done by modifying
sdusr.f in the root directory of the system partition
Please the the new thread
http://forums.parallax.com/showthrea...e-for-download
- - - - - - - - - - -
Release is expected in January 2012
20111204 update
PropForthEEPROMfilesystemOptimizeKernel - EVERYBODY (that uses propforth) will use this
- PropForthBootOptimizeKernel with EEPROM file system supposrt built into the kernel.
THIS VERSION DOES NOT INCLUDE the SAVEFORTH word. INSTEAD, the file system has a file .boot.f which is automatically loaded at boot. Sp all your extensions are re-read and re-compiled at start up. YEP, now you don't have to worry about corrupting the kernel, as the kernel itself becomes "read-only"
From this kernel, we add the extension for SD support, and can load and re-load scripts of any size, on the fly.
Number base can be forced using the prefixes B, D, H for binary Decimal, hex. Other flag character, such as #, $, %, etc, are not standard, and not consistently used across all processors and environments, and I always though using weird characters was kind of silly when B, D, and H always make perfect sense.
b0101 = decimal 5
d010 = decimal ten, even if the base is Hex or binary
h123 = decimal 291
Also, the lower case z will be used to designate base 64. This allows a long to be expressed in six characters. This is used all the time with the assembler, as we will be dealing with lots of longs. But we don't have to worry about it, the assembler is supposed to take care of this automagically.
20111012 Preview 2 for 5.0
CHANGES
- hex values MUST be in UPPERCASE
- ability to force hex using 'x' prefix: x00 xADB
- ability to force decimal using 'd' prefix: d00 d123
- line buffer option: does not echo until CR is input. Used for ethernet connection, or for text file input
- backspace is x08, DEL xF7 is not used
- CR x0D is the line termination, LF x0A is not a line termination
- CR is ouput as x0D, setup teraterm to receive CR+LF, and to transmit CR. This was done to allow redirection of files easily to and from IO. (Telenet and serial terminal etc all behave the same.)
- kernel is modified and optimized
- added ST@ ST! - fast instructions to access and modify stack
- assembler optimized routine can be dynamically added
- assembler words are now dynamically loaded and cached; provides huge optimizations and space efficiency
- spin constants which allow:
specififying tx pin
specififying rx pin
specififying baud rate
specififying cog used for the console serial driver
specififying which cog starts up forth
http://code.google.com/p/propforth/issues/detail?id=53
There is now a multi-step process to build the kernel. This is to facilitate modifications for custom hardware (Hive, C3, dracblade, etc) and custom functionality. The intent is that modified kernels will still be compatible with the remaining propforth developer extensions. In the following,
"slow" means "as slow as 4.5 was". Uses the original, straight forth definitions without optimization. These are easier to modify
"fast" means forth routines identified as "bottlenecks" have been optimized in assembler. These are more difficult to modify unless the starting point is the original forth definitions.
PropForthStartKernel - MOST folks will NOT use this
- slow io, slow multiply and divide, slow to interpret and compile,
but very small and code runs fast. Error checking in place, but no error
reporting. This kernel has extra symbols which are necessary for recompile
but will be removed to shrink it still further.
PropForthBootKernel - MOST folks will NOT use this
- slow io, slow multiply and divide, slow to interpret and compile,
but very small and code runs fast. Error checking in place, but no error
reporting. This is minimum size. This kernel can be used for specialized
apps.
build 2011OCT04 10:10 20,400 bytes free in dictionary, 222 longs free per cog
PropForthBootOptimizeKernel - EVERYBODY will use this
- Assembler optimizations, fast kernel. Error checking in place, but no error
reporting. This kernel can be used for specialized apps.
build 2011SEP29 14:00 19,822 bytes free in dictionary, 176 longs free per cog
This kernel is the base for most kernels and applications.
ADDED 20111204 - PropForthEEPROMfilesystemOptimizeKernel - EVERYBODY will use this
- PropForthBootOptimizeKernel with EEPROM file system supposrt built into the kernel.
THIS VERSION DOES NOT INCLUDE the SAVEFORTH word. INSTEAD, the file system has a file .boot.f which is automatically loaded at boot. Sp all your extensions are re-read and re-compiled at start up. YEP, now you don't have to worry about corrupting the kernel, as the kernel itself becomes "read-only"
From this kernel, we add the extension for SD support, and can load and re-load scripts of any size, on the fly.
- the stuff below may no longer be needed due to the change above, but I did not sort through it yet -
*** Normal Dev/App system runs on ANY prop chip (EEPROM 32k and greater).
Load PropForthBootOptimizeKernel.spin into eeprom, reboot
paste contents of PropForthDevKernel.f into teraterm
saveforth
**** eeprom filesystem kernel. Runs on prop chip EEPROM 64k and greater.
BootOptimizeFsrdKernel
This kernel is designed to be a base system with an eeprom file system. The system comes up with a
read only capability, searches for a file called boot.f, and executes it. Holding down the ESC
key while rebooting prevents teh execution of boot.f
Modification should be done by modifying boot.f, and loading the files and capabilities as needed.
The first thing boot.t does is to define forget, and forgets the seriial driver. When needed the serial
driver - serial.f can be loaded from the file system. So if you add words and do a saveforth, you will mess
up this kernel.
The file system is read only, and fswr.f must be loaded before the file system can be written.
***** eeprom filesystem kernel + sd card file system
This now provides a large file system, boot modification should be done by modifying
sdusr.f in the root directory of the system partition
Comments
The preview is to start a discussion on the way it is taking shape.
The changes are in response to user requests and the team's stated intent to "refactor and do it right" per the current design decisions.
I have not completed scrubbing the issues list.
Can't wait to get my time challenged little hands on it and start playing!
If this is really the final major structural change, 5.0 is where I'll start documenting and tutorializing (of COURSE it's a word!!).
When 5.0 is out, there will be a 5.1 release tailored to the final standard testbed hardware configuration.
The 5.1 testbed will allow all propforth kernel functions to be excersied, and including multiprop, ethernet, SD, remote interactive diagnostics; and automated tests.
Also, the final version will support adding extensions and drivers for new hardware, and including automated tests for these.
The final goal is to support continuous regression test during development, including both kernel and user applications on custom hardware.
All this is present in 5.0, but the peices for automation will not be packaged up nicely until the 5.1 hardware is finalized.
Prepare to commence tutorialization!
WRT the hex x0D standards, don't most modern languages support 0x0D (i.e. the hex prefix is 0x)? Might this be more prudent?
I've always hated the 0x prefix method and much prefer the $ prefix or h suffix. For the sake of standards can't we just use $ for hex, % for binary and why not # for decimal? The prefix (or suffix) overrides the current conversion base.
In Forth when the number is being compiled or interpreted it is processed as any other word by scanning the dictionary and if a match is not found it will then try to convert it to a number in the current base and if this fails then it's an error. The trick is to further process numbers to look for identifiers but if you used 0x then you couldn't just look at the prefix as it is already a digit. Now that just makes it that little bit more complicated for a compiler that is target resident and not residing in 100's of megabytes on a multi GHz PC. The Prop is very limited in terms of memory and I'm sure the extra memory of "Prop II" will prove to be a real boon for Forth in general.
FORTH is a simple, high level language, easy as basic (once you get the RPN syntax) but as powerful and (nearly) as fast as assembler. It tends (used to tend?) to be faster than compiled languages like C, but modern optimizing compilers that approach assembler speed can be faster in execution. The main advantages are interactive command line, language extensibility, small footprint, and ease of writting drivers.
One may extend that language to add new contructs, like those found in pascal or C, but usually these slow things down and take space, so this is kept to a minimum. The famous FPC implementation for PC is an ANSI forth that implements everything including the kitchen sink (far as I can tell), so if you can remember all the stuff an example can be found.
About the hex and decimal designators: I will open and issue to examine the suggestions. Thanks for these, hopefully the discussion for the preview will get us ironed out by release. The end solution should be "simple". I believe Sal's routine looks for x follwed by 0-F, and d followed by 0-9. Of course it can be changed to look for anything we want, but this may have been selected for shortest, faster code. Sal is picky about adding bytes to the kernel, and clock cycles to the interpreter.
http://code.google.com/p/propforth/issues/detail?id=58
So far we have three suggestions, if you have a preference or alternative, please post.
Gforth allows you to override the value of base by using a prefix2 before the first digit of an (integer) number. The following prefixes are supported:
- & – decimal
- # – decimal
- % – binary
- $ – hexadecimal
- 0x – hexadecimal, if base<33.
- ' – numeric value (e.g., ASCII code) of next character; an optional ' may be present after the character.
...the "if base < 33" comment makes me chuckle.I can't find what ANS forth says but shouldn't it be done how most Forth implementations are done? this seems like a small price to pay for portability between Forths at least.
Just fuel for the fire.......
For example, direct threaded vs indirect threaded; software multitasking vs single tasking; 8 bit vs 16 bit (vs 32 bit) tokens; etc. These are all selected per the characteristics of the processor and target application.
In this case, Sal started out with everything HEX. Then added decimal, to make it easier to talk about it with me (I don't speak in hex well). Since each cog has its own number base register, Sal gradually decided its easier just to specific the base in the literal in the source, rather than keeping track of BASE on the cog handling the text, and the cog handling the interpreter, and the cog running the forth command line. Now the question is, whats the EASIEST way for Sal to do this, in a way that we can understand, such that it takes minimum bytes and clocks, preferably zero.
Which is easier, prefix or suffix? I think Sal has decided on prefix, due to the way the interpreter is implemented. Examples supporting suffix can be considered.
So, while special character can be used, what is simplest and easiest? If $, #, % can be shown to be associated with hex, decimal, and binary, then these can be candidates for tagging the number base. BUT I think the symbols are NOT always used this way, AND that the majority of the time, these symbols are used to designate other than number base, so might not be the best choice in this case. Of course, I'm open to seeing examples for or against.
In my opinion, h (or x) for hex, d for decimal, and possibly b for binary are the way to go. These make sense, are easy to implement, and are already there. I've always felt using $ for hex when it had already been established as meaning 'address of' was a blunder, but that's just me and another reason why they call me "brain? no!".
As a note: ANSI forth is great because it is a standard, but it is more for workstations, not so much for micro controllers. The rumor is that Chuck Moore said "in addition to being the best thing that happened to forth, the ANSI standard is also the worst thing that happened to forth". The stuff in the standard is optional, and is a measure of degree of compliance to the standard; not necessarily a measure of forth.
All this is my opinion, if we decide we like a different method better, Sal will probably go along with it.
kuroneko did some analysis on timing starting in 4.5 which resulted in significant improvements
The analysis continues to the software logic analyzer, Sal added pf5.0 routines to get better data
Once this get ironed out, we will update the LA tests.
Disregard the comments below, I didn't read the whole thread, just the top post. It looks like others have concerns too.
FWIW I can't see why we can't just set the base to what we prefer to use and leave it at that.
Is this set in concrete ?
Other flag character, such as #, $, %, etc, are not standard, and not consistently used across all processors and environments.
b0101 = decimal 5
d010 = decimal ten, even if the base is Hex or binary
h123 = decimal 291
b0101, d010, and H123 could equally be interpreted to be Hex values.
Proforth is not forth for all processors and environments. All of those chars have special meaning in Prop code.
Ron
PF5 fix "PF4.6 with SD-function cannot assemble asm-code"?
PF5 can use SD-function?
Pf4.6 actually cannot use SD-card(saving file or reading file etc).
When is Sal due back?
Nothing is set in concrete, but I whine the loudest, as I am the only one Sal hears.
HEX digits (ABCEDF) MUST be in UPPER case
forcing the base with a prefix character is done with LOWERCASE only
IE b010 d0101 h010 are valid
hABC is valid
dABC and bABC would result in and UNDEFINED WORD error, etc
Sal will fix this when he does the 5.0 regression tests. He guesses he will send 5.0 beta to me for testing in the next couple weeks, I will forward the beta to you (and the rest of the guys too if they are willing). We can work through each issue.
He got back last week (he said Buenos Aires was awesome) and we had our call last Sunday, but he's going to be occupied until January (Sunday the 8th is our next scheduled call). Something he calls "family" and "work", but I am unfamilliar with these as they are not in the standard kernel.
I'll post when I get the next beta, and folks can PM then if they want to see it and make comments. This should help dilute my influence on the quirks of propforth.
See the new thread of the PF5.0 release
http://forums.parallax.com/showthread.php?138399-Propforth-5.0-is-available-for-download
Thanks for the update.
I should have said "interepreted by the computer industry in general"
Look forward to seeing the next beta.
Cheers Ron
Glad things are moving forward well!
I also am looking forward to seeing the next beta.
>>Bob
I am in favor of b, d, and h as as number base designators although it would be nice if that was an option that a user could set to their personal preference. Not too happy with having to use upper case for the hex digits though. Is there a reason for this?
There is an option for changing this, and anything else in the kernel. The user just has to edit the source, and change to check for a different character (and rebuild the kernel, this is in the ADVANCED chapter of 5.0). HOWEVER, if you do that, you are totally on your own, because all of Sal's source is written using b for binary, d for decimal, h for hex and x for base 64.
The reason for using uppercase: We want it to be completely obvious when the number base is changing. Each cog can have its own number base, including the cog you are working on (e.g. cog 6), and the cog that is handling communications (usually cog 7) and the cog running remote communications (.eg. cog 4) and the you are talk to remotely (e.g. cog 5). So at minimum we must worry about number base in two places, and worst case can be four or more places. Sal noticed this got out of hand and caused errors, so this scheme completely eliminated those errors.
In general, Sal's rule of thumb is every single op-code and data byte has to be the minimum to get the job done. (This allows the user to make all decisions on resources allocated to error checking, etc). In this particular case we determined that exactly one character was needed to determine the number base; and to prevent contradictions, we decided to use lower case for the flag characters and upper case for the values. We considered using %, $, #, etc, but there was no universal "standard" we could point to, and spin only allows for forcing binary and hex, and does not allow switching the number base; and so does not allow for forcing values into decimal. Also, we need to specify base 64, which spin does not allow, so we figured the spin "standard" wasn't going to work for us in this case. We were discussing this since last spring and finally reached the decision for 5.0
Sal mentioned that CR-FL is handled differently in some terminal programs. Teraterm can translate sent or recieved lines from CR, LF or CR-LF into CR, LR, or CR/FL according to user needs. Sal needed to do development on another machine, and found the common tools did not support this, and/or did not support them in the same way. So CR-LF translation has been added to the serial driver. This should not affect anything that is already in place, it is just another configuration option that is available. (I think Brian ran across this in 4.3, but we did not properly identify the cause)
Not all serial hardware supports flow control, these days it appears that the assumption is the PC is fast enough, and no flow control is needed. This may be an issue on older or single core PC systems, the tiny dual core netbooks Sal uses have no problems. I think I'm the only one that still has 286's in the lab, so I will just be careful.
LOADING Source into the interpreter: Some folks have reported issues loading source into the interpreter. To clarify, source is buffered into HUB memory, so there is a physical limit as to much we can stuff it in one gulp. About 10K characters is about as big as you want to go on an empty kernel. Quick Solution - take multiple gulps. The bigger the application, the less space there is for buffering source to HUB memory. The un-optimized kernels can choke from the interpreter being slow, and will give a buffer overrun message, but are fine for rebuilding the kernel (so be careful if you want to go nuts and build the whole application into the kernel, most folks shouldn't).
Sal tried circular buffers for the SD file write, an it works fine, but not all cards perform the same, so this method proved unreliable due to the variation in the physical SD card speed.
The real solution is to load over IP. The IP support contain flow control by nature of the ethernet hardware, so its built in and we don't have to worry about it. This is to be laid out in the tutorials, I've got an issue open to track it.
(Issues 94 and 95)
I might have a release candidate in my hands next week. The main task is finishing the tutorial suite, as the tutorials point to the HTML documentation pages (that I will need to create after release).
Sal reports he same not seen a bug in the kernel since the initial passes at 5.0. So far, everything is stable and fast.
The issue is with packaging the release. Propforth very powerful and complex, and we are trying to make it as simple as possible.
As of now, we have three spin options included: the regular kernel, the kernel with EEPROM filesystem support, and the kernel with EEPROM filesystem plus SD card filesystem support. These are the configurations that Sal uses in development. Other configurations are possible, for example SD file system + VGA on the demo board, but these are out of scope for the time being.
It turns out the testing and documentation (explaining things so other than the developer can use them) takes much longer than the actual development.
We are working on release candidate 4 this week. We remain on the brink of release, just short of finished.
I noticed that too, this is one of the reasons we spent so much time on the release package after the kernel has been stable for so long.
This is the new plan:
Release the kernel, there will be three spin files that need no additional <forth>.f files to get going-
- propforthOptimized.spin - propforth with assembler optimization
- propforthEEPROMfswr.spin - propforthoptimized with the EEPROM file system built-in, read-write by default (previous read only proved to involved for new users)
- propforthSD.spin - propforthoptimized with SD support built into kernel.
We release this and start documenting what's here. Sal has a nice chunk of it already, starting with the words list.
The regression tests are removed to the advanced section. We had previously tried to redo the regression tests as tutorials (as they demonstrate the proper use of each and every function) but making them "new user" friendly proved to be a mass of contradictions. So now all the regression tests and the parts for rebuilding the kernel from scratch are move to the ADVANCED section. The advanced stuff is not ready yet, so we will continue development on that in private.
IF anybody want to do something advanced, I can send them the kernels source directly, I suspect only Ron and Brian would want it early. Believe you me, there's enough to play with in the easy section before futzing with the advanced stuff.
When it first comes out, there may be very little documentation, expect to work out the details in the forum or via email for a short while.
I'd like the kernal source also. I learned most of what I know about propforth by going through the kernal for 4.5/4.6 !
I'll be happy to play, futz, and write* where I go wrong and document what I learn.
*yes, I intended this 'write' and not this 'right'
Thanks
Bob
The PF5.0 kernel is pretty much finished, and has been stable since forever it seems. BUT the documentations is in flux. The prop architecture, forth, and forth on the prop architecture are significantly unique and deserve proper documentation. We know from previous questions and conversations that propforth appears deceptively simple, users can easily get lost without clear, concise documentation. Propforth is a very sharp knife. Forth itself can leave folks baffled, unless they have proper mentoring or just "get it" right off the bat. Many folks on these forums have shared horror stories of forth project gone out of control, the root cause usually seems to be poor communication. Since we cannot personally mentor everyone and don't wish to rely on luck, we are focusing on creating sufficient material so users can discover how to use forth on their own.
To address the issue, the core team is "doing the docs". Impatient as I am, I continue to agree that the kernels should be released with completed docs, and not before.
Outside pressures are causing the normally fast release cycle to move slowly. (Read: "dayjob"). Each time we find something, we check for impacts to the "user persona" and "use cases". Each "tweak" triggers a complete respin of the release cycle. Each time I run the tests (as "voice of the user"), I find more things that "could be clarified", causing more specific decisions to be made, triggering more changes. This cannot be avoided when one is breaking new ground, and we are resisting the temptation to "just ship the prototype".
We're still here and working. Hopefully the end result will be worth the wait.
FYI - I am premitted to send out a pre-release only AFTER I have completed cycle with no new changes.