Personally, I think the interactive aspect of Forth is overrated. I would much rather use a good off-line editor/pre-compiler and just upload all the resulting threaded code at once when I want to try something. With such a system, dictionary entries can be as long as you want them, since the target doesn't have to include the dictionary.
Yes, I'd agree with that - and more so on a Memory-constrained Prop.
Target Interactive, as in using a line-editor or something low level on target only, is rare.
Most SW design keeps a source copy in an editor, and on some storage device.
Better to have a 'somewhat smart' editor/pre-compiler, that can include the download.
A system with a very short char count can be used for simpler systems, with user care, but also works in larger systems.
The thing is that 12 characters is very generous, it's not cramped or clipped at all, but neither is it too wasteful. We are not talking about those 4 letter Forths etc and bringing up the horrors of some systems is not really comparing apples with apples, but neither is comparing useful. Having a 16 byte fixed header will use up more hub RAM initially but it's not meant to stay there as once the system is extended it can move that dictionary to EEPROM or SD. At present once it is moved into EEPROM it can take up to 10ms to read that block that the string should be in if I don't already have it cached. This works but it is a complicated hybrid arrangement yet if I use fixed length records I can store all of them in EEPROM and even without caching I expect to be able to find a match using a binary search in around 1 to 2 ms.
Some Forths are tethered and are totally compiled on the PC but the strongest points of Forth are lost I feel. It's not the language so much as the environment. I really depend upon the interactive nature of Forth to debug and develop incrementally on a system, to see what really works and what doesn't. It removes all the guess work, the plans of mice and men who after having devised a grand scheme press the GO button and then puzzled as they watch the system crash or make it's way agonizingly slowly to a non-event. Anyway I love exploring ideas and I can do this instantly in a way that includes all the debugging at my fingertips. Take not my words from Forth away!
btw, some systems will have most of the words stripped away for security reasons, but not for me. Tachyon borrows pub and pri from Spin to define headers in place of : so that headers can be marked as private to be reclaimed later but V4 adds pro which is a protected (and prioritized) name so that all the others can be stripped leaving only these few functions.
I don't think you guys understand FORTH. The dictionary, editor, interactive mode is the language.
Where space is cramped, one just clears out words, or consolidates them or redefines them as needed.
YES - @potatohead, (@jmg, @phipi)
having 'dictionary & interactive mode' is essential in FORTH as I like it.
While you can create a FORTH system which uses an external (PC) compiler and the old EDIT/COMPILE/LINK/LOAD/DEBUG ... cycle,
this would lose the best of the FORTH world.
e.g. the Tachyon HTTP/FTP/TELNET servers and my extension for dynamic HTML with embedded FORTH scripting all require
access to the dictionary, the interactive parser and 'read-eval-print' loop.
I've added another special opcode for task variables so that each cog may have its own but share common routines that use these task variables. This means that all word codes are only one word long except for a LONG literal which of course requires 32-bit operand and 16-bit word code.
So far so good, says the eternal optimist.
Now the dilemma I face is what to do with the dictionary as I need to mix word aligned code with byte aligned characters. Currently Tachyon stores each record in the dictionary in this format:
That works well enough and the count also helps to search faster and to skip to the next record in ascending memory since the dictionary build down towards code memory which builds up.
Now to code a dictionary entry for V4 wordcode in the Prop tool requires an entry like this:
byte 0,"WORDS", hd, (@WORDS+s)>>8,@WORDS+s
Since I'm lazy I don't bother working out the count byte as Tachyon will fix up the counts on a cold start.
But that looks messy so I could just enter @WORDS+s as a word like this:
byte 0,"DUMP", hd
word @DUMP+s
So that is a lot cleaner but depending upon the byte alignment it might add one extra byte between the atr and the wordcode which has to be factored in when skipping to this field or the previous name.
Now the other way to format this is to bite the bullet and make each dictionary record a fixed length given that it is rare that Forth names exceed more than 12 characters since_we_don't_use_long_names as they are a pain to interactively type plus they take up memory. One advantage of a fixed format is that it lends itself to begin accessed easier in slow memory such as I2C EEPROM. At present the dictionary can take up a lot of precious hub RAM but I have a scheme which drops these names into 1 of 64 hashed index blocks of EEPROM/SD. However if I use fixed length records I could just store them as is in EEPROM without any special tricks or hashed index blocks except perhaps sorting them. Using a binary search it becomes quick and easy to locate a name without having to read a whole block of 384 bytes in.
The fixed record looks like this:
1 byte = count + 4-bit vocabulary id
12 bytes = name with unused bytes as nulls
1 byte = attribute (smudge, private, immediate, etc0
1 word = word code
So the main reason for the fixed length record is that the dictionary, or most of it really belongs somewhere other than hub RAM but normally that somewhere else is slow. The hashed index block uses around 24kB as some blocks are only half full yet the 16-byte fixed record approach would use less memory overall.
Thoughts?
Hi Peter,
I had a look at your and my dictionaries and it looks 12 characters should be ok -
BUT I dislike fixed limits ...
where I see this could be a restriction is in using the dictionary and interactive mechanisms to create language extensions,
like what you did with FTP / HTTP&HTML / TELNET - or what ever a user might want to come up with (e.g. NEMA parsing ...)
There a fixed name length could be annoying ...
so ... while I see the benefits of a fixed dictionary structure, I generally prefer the flexibility ...
@MJB - if we are going to have all that functionality we need as much code space as possible and hub RAM is the sensible place for that which means EEPROM for the dictionary. There is only one name I came across that was 13 characters long and changing to less is neither here nor there so I feel we will not be restricted with this scheme but it will allow us to handle it more efficiently in EEPROM. Bear in mind too that V4 will include vocabularies so names can be more descriptive, such as FILE OPEN vs PORT OPEN, where there are two OPEN functions that work totally differently.
Now FIXED LENGTH does not mean fixed length name, it means fixed length RECORD, that is, memory allocated to allow up to 12 characters although the count does allow 15 to be recognized though not stored.
CNT+VOCAB(1),NAME(12),ATR(1),WORDCODE(2)
So it makes no difference if you have ! as a name or CONNECTED? since they all fit nicely in that record, there are no practical restrictions.
@MJB - if we are going to have all that functionality we need as much code space as possible and hub RAM is the sensible place for that which means EEPROM for the dictionary. There is only one name I came across that was 13 characters long and changing to less is neither here nor there so I feel we will not be restricted with this scheme but it will allow us to handle it more efficiently in EEPROM. Bear in mind too that V4 will include vocabularies so names can be more descriptive, such as FILE OPEN vs PORT OPEN, where there are two OPEN functions that work totally differently.
Now FIXED LENGTH does not mean fixed length name, it means fixed length RECORD, that is, memory allocated to allow up to 12 characters although the count does allow 15 to be recognized though not stored.
CNT+VOCAB(1),NAME(12),ATR(1),WORDCODE(2)
So it makes no difference if you have ! as a name or CONNECTED? since they all fit nicely in that record, there are no practical restrictions.
vocabularies : great - missed that in previous post
I agree, as said above that for FORTH use fixed 12 should be plenty.
where I see this could be a restriction is in using the dictionary and interactive mechanisms to create language extensions,
like what you did with FTP / HTTP&HTML / TELNET - or what ever a user might want to come up with (e.g. NEMA parsing ...)
There a fixed name length could be annoying ...
but this might not be a central design concern - and for short words like the FTP commands (seem to be max 4 chars ?? ) it will be OK.
Unless supercalafragalisticexpialidocious is a command I don't think we will have any problems ever at all plus remember the 15 character count trick but I could also play another trick and link the next record's memory for those improbable extra special cases if I ever come across that, I will figure it out then.
@MJB - if we are going to have all that functionality we need as much code space as possible and hub RAM is the sensible place for that which means EEPROM for the dictionary.
.
.
.
.
More code space please, everything is allot: HTTP, FTP, TELNET, FILE IO, application drivers and application code oh my.
I will gladly refactor any of my source to get more code space.
I don't think you guys understand FORTH. The dictionary, editor, interactive mode is the language.
e.g. the Tachyon HTTP/FTP/TELNET servers and my extension for dynamic HTML with embedded FORTH scripting all require
access to the dictionary, the interactive parser and 'read-eval-print' loop.
MJB, did you get your dynamic HTML working with the IOT5500 modules?
I don't think you guys understand FORTH. The dictionary, editor, interactive mode is the language.
e.g. the Tachyon HTTP/FTP/TELNET servers and my extension for dynamic HTML with embedded FORTH scripting all require
access to the dictionary, the interactive parser and 'read-eval-print' loop.
MJB, did you get your dynamic HTML working with the IOT5500 modules?
All I can do from here is help test or something when you ask. It was so useful to marry <?fth ." Hello World /> syntax with javascript back on the spinerette.
But I'm not doing "the work" so I will not complain, just encourage when the time comes.
Could a NULL first char in the name be a signal that the name exceeds 12 bytes, and then use the remaining 11 bytes to redirect to the actual name string? Coder discipline would keep names short, but longer names are still possible. Might need garbage collection ..
However if I use fixed length records I could just store them as is in EEPROM without any special tricks or hashed index blocks except perhaps sorting them. Using a binary search it becomes quick and easy to locate a name without having to read a whole block of 384 bytes in.
I think this runs counter to interpreted Forth, does it not? What you really want is a linked list. That way, by scanning back through the list until you reach a match, you will get the most recent definition. If you FORGET that definition, anything prior to it with the same name will still be in the list for use. That allows you to try new stuff without deleting an earlier definition if the new stuff doesn't work out. Plus, the dictionary can be interlaced with the the word definitions, so that FORGETting requires but a single pointer adjustment.
The other advantage of a linked list is that word lengths can be variable. The disadvantage, of course, is the requirement for a linear search, which takes extra time.
I realize this runs counter to my notion that a target-hosted editor/interpreter is overrated. I'm only suggesting this in the present context that nearly everyone else seems to agree on. IOW, it's still an interesting problem, even if I don't entirely buy the premise.
Unless supercalafragalisticexpialidocious is a command I don't think we will have any problems ever at all plus remember the 15 character count trick but I could also play another trick and link the next record's memory for those improbable extra special cases if I ever come across that, I will figure it out then.
Vocabularies will make a huge difference.
Could a zero character count be used to indicate a link to the next record's memory?
A case of a word being longer than 8 characters will definitely happen. The case of it being more than 10 characters might possibly happen. From my experience and I find that 12 characters is in all reality all that is needed. There are programmers familiar with other languages who then name functions as they would in other languages which is ugly even in those languages. Some might define TURN_GREEN_LED_ON whereas I would rather factor these functions and end up saying ON GREEN LED and all those words are totally reusable and interchangeable. In fact both ON and GREEN are just constants. So programming style has a lot to do with it.
I noticed in some old Forth source code I came across that they had defined "add-reference" and "add-definition" which are 13 and 14 character length names but even so these would still be accepted with the last couple of characters truncated. However elsewhere they are quite happy to say "one-char" which is short for "one-character" so why didn't they just say "add-def" and "add-ref"?
To tell the truth I had some doubts about this fixed length record business but the more I've explained it the more I'm convinced that it is the best way to go for EEPROM use. I have options to squeeze more characters into the same space or linking another record but the reality once again is I don't need it. The important thing is to optimize the memory for code but also allow for efficient handling of names which means we impose some reasonable restrictions. I don't use LFN in my file-system, it's a blight and although 8.3 names are just a little short they work fine and are nice and efficient. If I designed a version of LFN I would have just used the next directory entry(s) suitably tagged for this rather than the convoluted and laughable patented microslop methods.
A case of a word being longer than 8 characters will definitely happen. The case of it being more than 10 characters might possibly happen. From my experience and I find that 12 characters is in all reality all that is needed. There are programmers familiar with other languages who then name functions as they would in other languages which is ugly even in those languages. Some might define TURN_GREEN_LED_ON whereas I would rather factor these functions and end up saying ON GREEN LED and all those words are totally reusable and interchangeable. In fact both ON and GREEN are just constants. So programming style has a lot to do with it.
I noticed in some old Forth source code I came across that they had defined "add-reference" and "add-definition" which are 13 and 14 character length names but even so these would still be accepted with the last couple of characters truncated. However elsewhere they are quite happy to say "one-char" which is short for "one-character" so why didn't they just say "add-def" and "add-ref"?
To tell the truth I had some doubts about this fixed length record business but the more I've explained it the more I'm convinced that it is the best way to go for EEPROM use. I have options to squeeze more characters into the same space or linking another record but the reality once again is I don't need it. The important thing is to optimize the memory for code but also allow for efficient handling of names which means we impose some reasonable restrictions. I don't use LFN in my file-system, it's a blight and although 8.3 names are just a little short they work fine and are nice and efficient. If I designed a version of LFN I would have just used the next directory entry(s) suitably tagged for this rather than the convoluted and laughable patented microslop methods.
For LFN I would have used 28+4 or perhaps 56+8. Existing 8+3 would just slot right into this the same as a file name of say 4+3 fits into the 8+3. Agreed the current patented method is total Smile. Perhaps that's why they were able to patent!
Now we have a licensed exFAT system which is now going to be used on SD Cards. Surely someone could come up with an agreeable free open source alternative???
exFAT is another licensed proprietary bridge that we will have to cross at some point.
I've come full circle with my dictionary approach, well kinda. For entries that are created in hub RAM including the precompiled dictionary I am using variable length names to keep RAM usage down. But when I transfer these to EEPROM they become a 16 byte fixed length record allowing up to 12 characters to be stored (and up to 15 characters to be specified). The 16 byte record length then becomes easy to do a binary search with and fits into EEPROM page sizes but any words that are longer than 12 characters will be treated as an exception and left in hub RAM since this will be a hybrid dictionary allowing new words in general to be created in RAM until a COMPACT operation is performed to also add them to EEPROM.
So best of both worlds plus the EEPROM is nice and easy and quick to search.
edit: this does mean that I need to compensate for byte to word alignment gaps but that is easy enough to handle.
Just a progress update as I have been plodding along with this in between some real work. So far, so good. Everything seems to be working fine as I am working with Dawn at the interactive prompt although I haven't quite allowed for new definitions quite yet as I think about some changes. Tachyon previously used a giant loop within the kernel to process user input whereas I am avoiding anything like that in V4. Each character is processed via an action table so that a CR will mostly call an ENTER function to process the compiled line of code/commands.
There are also special functions tied to many of the unused control keys so you don't have to type anything to view or init the stack, list words, debug etc. Even numbers are preprocessed so that if the number of digits equals the number of characters processed then it is automatically accepted as a number without any further processing or dictionary searching. That means that the $ # and % prefix also modify what is accepted as a digit.
So each wordcode can represent a PASM address in cog space $0000...$01FF or an interpreted wordcode address in hub memory $0200...$7CFF or a local per cog task variable $7E00...$7EFF for up to 256 bytes or a conditional relative jump instruction $7F00...$7FFF for +/- 128 word displacement. I did originally have another wordcode for unconditional jumps such as AGAIN or ELSE but these are handled just as well with the normal hub address call except we set the otherwise redundant lsb to indicate a jump rather than a call. The lsb is also handy for saving two bytes for functions that EXIT so instead of calling HOLD then EXIT we simply jump to HOLD and save some time and two bytes.
Rather than the traditional "ok" prompt at the end of a line that is executed and not part of a definition I am just using indented line prompts which may include additional information as necessary.
Previously the kernel dictionary was manually aligned in the source to place it high in memory so it could grow down towards the code but in V4 I simply have a routine which besides calculating the count of each name also calculates how to move the whole dictionary right up against the buffer area and set all the pointers correctly.
Wordcode is very efficient putting aside that we always need two bytes even for cog addresses in the first 256 longs, and the only function that is longer than a word is a LONG LITERAL of 32-bits plus a wordcode. Take a look at this simple function that before it is executed with an CR we type a ^D for DEBUG to dump among other things the compiled codes for the current line:
The first wordcode is $800A which corresponds to 10 with the msb set to indicate a 15-bit literal. FOR, DO, and LOOP are all wordcodes that do not require any calculated branch address as they use a branch stack. So this whole broken down looks like this:
V4 can now compile new definitions and all the structured branching IF ELSE THEN BEGIN AGAIN WHILE REPEAT etc is working fine. Constants and variables are identical in that they take up no code space at all (in most cases) but simply store a literal value in the header so that any reference to a constant or a variable will simply compile a literal wordcode. Variables no longer occupy codespace but simply point to dataspace which works in a similar way to the earlier ORG DS style variables. So variables are contiguous in dataspace so if we define:
128 BYTES gpsbuf
LONG gpsnum
Then the address of gpsnum is 128 bytes after gpsbuf (whereas previously they were both long aligned with a preceding bytecode).
Since it is relatively easy to code Forth as wordcode in DAT statements I have added a lot to the basic kernel including a full feature DUMP word with variations for words, longs, characters, cog mem, etc. Also include is both WORDS for a simple listing and WWORDS for a more detailed listing of words.
Many functions such as listing words or dumping memory or stack etc are tied to control keys so it is very quick to debug.
Since numbers encountered in the input stream are processed immediately as numbers then this speeds up compilation as well. There was always a problem with table entries previously where the long list of numbers would cause the dictionary to be searched for each number so the line delay had to be long enough or else extra blank lines inserted between the table rows. There are other improvements being made as this is a totally new version with only the core components of the V3 PASM kernel remaining intact but expect this version to self-compile Forth source code much faster which may mean a much smaller line delay or hopefully none at all with just the basic buffering.
Since no high-level code can exist at the same address value as cog code then this area from $0000 to $01FF in hub memory is used for various buffers and system and Forth variables. Also the area occupied by the cog images for the serial and ping-ping code plus the PASM kernel cannot be used for high-level code since it cannot be backed up and overwrite the cog images in EEPROM, so this is used instead for file system and other buffers that do not need to be backed up into EEPROM.
Current Memory Map:
$0040 - system variables
$00C0 - spare
$0100 - cog 0 task registers, word and number buffers
$0180 - spare
$0200 - start of high-level wordcode
$1240 - currently the end of the wordcode kernel at present
---
$7400 - end of relocated dictionary
$7400 - start of receive buffers etc
$7600 - default data space
$7800 - file system (four open files) and general buffers
The kernel is still not ready for prime time yet but it is interactive and easily compiled with any of the Prop compilers. Default clock config around line 70 is set for 6MHz x8.
Propeller .:.:--TACHYON--:.:. Forth V4.0 DAWN 400161128.1830
Cold start - no user code - setting defaults
Setting up dictionary
--------------------------------------------------------------------------------
0001 1240 V4>
DICTIONARY WORDS @6CB6
DUP OVER DROP 2DROP SWAP ROT BOUNDS 1+ 1- + - MIN MAX * UM* U/ U/MOD / */ UM*/ ABS -NEGATE ?NEGATE NEGATE ADO DO LOOP +LOOP FOR NEXT AND ANDN OR X
OR ROL ROR SHR 8>> SHL 8<< 2/ 2* REV MASK >N >B 9BITS 0= NOT = > U< 0< C@ W@ @ C+! C! C@++ W+! W! +! ! CLOCK CLKIN CLKOUT OUTSET OUTCLR OUTPUTS INP
UTS SHROUT SHRINP RESET 0EXIT EXIT NOP 3DROP ?DUP 3RD 4TH CALL JUMP BRANCH> >R R> >L L> (WAITPNE) I SPIWRB SPIWR16 SPIWR SPIRD SPICE UM/MOD64 UM/MO
D32 RUNMOD (WAITPEQ) CMOVE (EMIT) (EMITX) CMPSTR LOADMOD !RP !SP COG@ COGREG COG! COGID REBOOT DELTA WAITCNT LAP .LAP HIGH LOW FLOAT NIP 0<> <> C~
W~ ~ SET? WITHIN SET CLR HEX DECIMAL BINARY READBUF KEY WKEY (KEY) EMIT CLS SPACE SPACES BELL CR <CR> .HEX .BYTE .WORD .LONG PRINT . ZPRINT @PAD HO
LD >CHAR #> <# # #S <D> PRINT$ LEN$ U. .DEC .DP " SEARCH FINDSTR NFA>BFA VER .VER TACHYON ERASE FILL ms seconds second DISCARD <CMOVE ALLOT ALLOCAT
ED HERE @NAMES names REG BUFFERS KOLD WORDS WWORDS DUMP DUMPW DUMPL DUMPC DUMPA DUMPAW .S IF ELSE THEN ENDIF BEGIN UNTIL AGAIN WHILE REPEAT --- \ '
' ( { } IFDEF IFNDEF " ." (.") == : pub ; ' NFA' V4 *end*
Still sorting out some quirks with the way the compiler works in that I am avoiding any special action that stops and waits for another word in the input stream such as when we go to create a new definition etc as the new version will set an action for the next word encountered as well as actions for characters themselves. I'm also thinking I have to distinguish this version of Tachyon from previous versions, although it is still "Tachyon" but wordcode based with other enhancements so I'm thinking I might just refer to it as Tachyon+. Nonetheless I have been compiling extensions to the kernel successfully but I would like to get to the stage where I can load the file system and more to really test it out.
A lot of the experimental stuff in it may change or be discarded as I test it out but the kernel seems a lot simpler than the old Tachyon kernel did, both in coding method and in the way all the parts are connected together.
Well, it probably won't be long now before I get to testing a fully loaded system, I am already running various devices.
Why use Tachyon+? Because it is designed to be faster and more compact overall than the "old" Tachyon as well as an improved and faster compiler.
So after T+ we will have the object oriented version T++ then ;-)
I was thinking I had a pretty good understanding of Tachyon internals,
but seems I have to relearn a bit :-)
When T+ is up we can check the old code still runs ... or adapt if needed.
Yes, I am thinking of the ++ version too
The good thing is that most of the PASM code is identical but using 16-bits for the address/code might mean we waste 8-bits for all calls to the first 256 locations but for most other operations it is faster and at its worst it is no slower. Knowing that loading a value of 1,000 used to involve 3 bytecodes and 3 hub accesses but now that is only a single word and access and good for up to values of +32767. High level calls were mostly handled with 2 bytes and a 16-bit vector or once the vector table was full involved 3 bytes for a call. Now there are no vectors or overheads, just one word to call or jump anywhere. Remember that hub access would slow down bytecode which needed extra bytecodes so the single hub access is both fast and more compact overall.
My experiments with the compiler are more interesting though and certainly slowing me down as I attempt to decouple all the various processes to work as a team rather than a bureaucracy, in a manner of speaking. I want this part to be fast and flexible so I can change the way it operates without having to bypass whole sections.
The EEPROM dictionary access needs to be transparent too so it can be done incrementally as required to make as much of the hub RAM as possible available for code. Having a separate data space has helped too to cut down on code memory usage since all variables and most constants are simple one word literals directly compiled into the code. It is easier to create structures with this approach.
I'm hoping that I won't make too many changes that require learning new rules etc. Will removing the traditional "ok" and using the new permanent line and status prompt be a good thing? A lot of things are experimental so feedback is always appreciated as not only does it need to perform, it also has to feel right.
My current terminal session. (need to fix my number radix back up again)
OK can be replaced by an informative prompt.
So V4> is fine for me.
What I would not like to loose is the possibility to paste in part of a terminal session.
Like you made a dummy definition for OK.
For a variable informational prompt this might be a little more complicated.
I think I would redefine the prompt to s.th. very simple like >
This V4> is quite distracting for the eye,
since the automatic focus of attention is on the start of the line.
And in this respect OK seems more 'ergonomic' ...
What about making the CR being part of the prompt definition.
Then we can have
" <CR>T4> " or " OK<CR>" as the prompt ...
The prompt info has been enclosed in parentheses so the screen can be copied and pasted.
The kernel is quite usable even at the moment.
Propeller .:.:--TACHYON--:.:. Forth V4.0 DAWN 400161201.0930
Cold start - no user code - setting defaults
Setting up dictionary
--------------------------------------------------------------------------------
( 0001 1400 ok ) $7F $20 DO I EMIT LOOP !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
( 0002 1400 ok ) : ASCII $7F $20 DO I EMIT
( 0003 1400 ++ ) LOOP
( 0004 1400 ++ ) ;
( 0005 140E ok ) ASCII !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
( 0007 140E ok ) TACHYON Propeller .:.:--TACHYON--:.:. Forth V4.0 DAWN 400161201.0930
( 0261 1772 ok )
End of source code, 0000 errors found Load time = 215080.808us
( 0262 1772 ok )
( 0263 1772 ok )
( 0269 1772 ok ) 4 FOR CR ASCII NEXT
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
( 0270 1772 ok )
Comments
Target Interactive, as in using a line-editor or something low level on target only, is rare.
Most SW design keeps a source copy in an editor, and on some storage device.
Better to have a 'somewhat smart' editor/pre-compiler, that can include the download.
A system with a very short char count can be used for simpler systems, with user care, but also works in larger systems.
Where space is cramped, one just clears out words, or consolidates them or redefines them as needed.
Some Forths are tethered and are totally compiled on the PC but the strongest points of Forth are lost I feel. It's not the language so much as the environment. I really depend upon the interactive nature of Forth to debug and develop incrementally on a system, to see what really works and what doesn't. It removes all the guess work, the plans of mice and men who after having devised a grand scheme press the GO button and then puzzled as they watch the system crash or make it's way agonizingly slowly to a non-event. Anyway I love exploring ideas and I can do this instantly in a way that includes all the debugging at my fingertips. Take not my words from Forth away!
btw, some systems will have most of the words stripped away for security reasons, but not for me. Tachyon borrows pub and pri from Spin to define headers in place of : so that headers can be marked as private to be reclaimed later but V4 adds pro which is a protected (and prioritized) name so that all the others can be stripped leaving only these few functions.
having 'dictionary & interactive mode' is essential in FORTH as I like it.
While you can create a FORTH system which uses an external (PC) compiler and the old EDIT/COMPILE/LINK/LOAD/DEBUG ... cycle,
this would lose the best of the FORTH world.
e.g. the Tachyon HTTP/FTP/TELNET servers and my extension for dynamic HTML with embedded FORTH scripting all require
access to the dictionary, the interactive parser and 'read-eval-print' loop.
I would stay with Tachyon 3 in this case.
Hi Peter,
I had a look at your and my dictionaries and it looks 12 characters should be ok -
BUT I dislike fixed limits ...
where I see this could be a restriction is in using the dictionary and interactive mechanisms to create language extensions,
like what you did with FTP / HTTP&HTML / TELNET - or what ever a user might want to come up with (e.g. NEMA parsing ...)
There a fixed name length could be annoying ...
so ... while I see the benefits of a fixed dictionary structure, I generally prefer the flexibility ...
Now FIXED LENGTH does not mean fixed length name, it means fixed length RECORD, that is, memory allocated to allow up to 12 characters although the count does allow 15 to be recognized though not stored.
CNT+VOCAB(1),NAME(12),ATR(1),WORDCODE(2)
So it makes no difference if you have ! as a name or CONNECTED? since they all fit nicely in that record, there are no practical restrictions.
I agree, as said above that for FORTH use fixed 12 should be plenty.
but this might not be a central design concern - and for short words like the FTP commands (seem to be max 4 chars ?? ) it will be OK.
Vocabularies will make a huge difference.
More code space please, everything is allot: HTTP, FTP, TELNET, FILE IO, application drivers and application code oh my.
I will gladly refactor any of my source to get more code space.
All I can do from here is help test or something when you ask. It was so useful to marry <?fth ." Hello World /> syntax with javascript back on the spinerette.
But I'm not doing "the work" so I will not complain, just encourage when the time comes.
The other advantage of a linked list is that word lengths can be variable. The disadvantage, of course, is the requirement for a linear search, which takes extra time.
I realize this runs counter to my notion that a target-hosted editor/interpreter is overrated. I'm only suggesting this in the present context that nearly everyone else seems to agree on. IOW, it's still an interesting problem, even if I don't entirely buy the premise.
-Phil
Could a zero character count be used to indicate a link to the next record's memory?
I noticed in some old Forth source code I came across that they had defined "add-reference" and "add-definition" which are 13 and 14 character length names but even so these would still be accepted with the last couple of characters truncated. However elsewhere they are quite happy to say "one-char" which is short for "one-character" so why didn't they just say "add-def" and "add-ref"?
To tell the truth I had some doubts about this fixed length record business but the more I've explained it the more I'm convinced that it is the best way to go for EEPROM use. I have options to squeeze more characters into the same space or linking another record but the reality once again is I don't need it. The important thing is to optimize the memory for code but also allow for efficient handling of names which means we impose some reasonable restrictions. I don't use LFN in my file-system, it's a blight and although 8.3 names are just a little short they work fine and are nice and efficient. If I designed a version of LFN I would have just used the next directory entry(s) suitably tagged for this rather than the convoluted and laughable patented microslop methods.
For LFN I would have used 28+4 or perhaps 56+8. Existing 8+3 would just slot right into this the same as a file name of say 4+3 fits into the 8+3. Agreed the current patented method is total Smile. Perhaps that's why they were able to patent!
Now we have a licensed exFAT system which is now going to be used on SD Cards. Surely someone could come up with an agreeable free open source alternative???
I've come full circle with my dictionary approach, well kinda. For entries that are created in hub RAM including the precompiled dictionary I am using variable length names to keep RAM usage down. But when I transfer these to EEPROM they become a 16 byte fixed length record allowing up to 12 characters to be stored (and up to 15 characters to be specified). The 16 byte record length then becomes easy to do a binary search with and fits into EEPROM page sizes but any words that are longer than 12 characters will be treated as an exception and left in hub RAM since this will be a hybrid dictionary allowing new words in general to be created in RAM until a COMPACT operation is performed to also add them to EEPROM.
So best of both worlds plus the EEPROM is nice and easy and quick to search.
edit: this does mean that I need to compensate for byte to word alignment gaps but that is easy enough to handle.
keeping the cake and eating it ...
I am very happy with this
There are also special functions tied to many of the unused control keys so you don't have to type anything to view or init the stack, list words, debug etc. Even numbers are preprocessed so that if the number of digits equals the number of characters processed then it is automatically accepted as a number without any further processing or dictionary searching. That means that the $ # and % prefix also modify what is accepted as a digit.
So each wordcode can represent a PASM address in cog space $0000...$01FF or an interpreted wordcode address in hub memory $0200...$7CFF or a local per cog task variable $7E00...$7EFF for up to 256 bytes or a conditional relative jump instruction $7F00...$7FFF for +/- 128 word displacement. I did originally have another wordcode for unconditional jumps such as AGAIN or ELSE but these are handled just as well with the normal hub address call except we set the otherwise redundant lsb to indicate a jump rather than a call. The lsb is also handy for saving two bytes for functions that EXIT so instead of calling HOLD then EXIT we simply jump to HOLD and save some time and two bytes.
Rather than the traditional "ok" prompt at the end of a line that is executed and not part of a definition I am just using indented line prompts which may include additional information as necessary.
Previously the kernel dictionary was manually aligned in the source to place it high in memory so it could grow down towards the code but in V4 I simply have a routine which besides calculating the count of each name also calculates how to move the whole dictionary right up against the buffer area and set all the pointers correctly.
Wordcode is very efficient putting aside that we always need two bytes even for cog addresses in the first 256 longs, and the only function that is longer than a word is a LONG LITERAL of 32-bits plus a wordcode. Take a look at this simple function that before it is executed with an CR we type a ^D for DEBUG to dump among other things the compiled codes for the current line: The first wordcode is $800A which corresponds to 10 with the msb set to indicate a 15-bit literal. FOR, DO, and LOOP are all wordcodes that do not require any calculated branch address as they use a branch stack. So this whole broken down looks like this:
Here's a quick session dump:
Propeller .:.:--TACHYON--:.:. Forth V4.0 DAWN 400161118.0000 Cold start - no user code - setting defaults Setting up dictionary ---------------------------------------------------------------- 00001 V4> 00002 V4> 1234 2* . 2468 00003 V4> $7F $20 DO I EMIT LOOP !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijkl mnopqrstuvwxyz{|}~ 00004 V4> 00005 V4> CR $7F $20 DO I EMIT LOOP !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ 00006 V4> 10 FOR CR $7F $20 DO I EMIT LOOP NEXT !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ 00007 V4> LAP 1000 FOR NEXT LAP .LAP 404.800us 00008 V4>V4 can now compile new definitions and all the structured branching IF ELSE THEN BEGIN AGAIN WHILE REPEAT etc is working fine. Constants and variables are identical in that they take up no code space at all (in most cases) but simply store a literal value in the header so that any reference to a constant or a variable will simply compile a literal wordcode. Variables no longer occupy codespace but simply point to dataspace which works in a similar way to the earlier ORG DS style variables. So variables are contiguous in dataspace so if we define: Then the address of gpsnum is 128 bytes after gpsbuf (whereas previously they were both long aligned with a preceding bytecode).
Since it is relatively easy to code Forth as wordcode in DAT statements I have added a lot to the basic kernel including a full feature DUMP word with variations for words, longs, characters, cog mem, etc. Also include is both WORDS for a simple listing and WWORDS for a more detailed listing of words.
Many functions such as listing words or dumping memory or stack etc are tied to control keys so it is very quick to debug.
Since numbers encountered in the input stream are processed immediately as numbers then this speeds up compilation as well. There was always a problem with table entries previously where the long list of numbers would cause the dictionary to be searched for each number so the line delay had to be long enough or else extra blank lines inserted between the table rows. There are other improvements being made as this is a totally new version with only the core components of the V3 PASM kernel remaining intact but expect this version to self-compile Forth source code much faster which may mean a much smaller line delay or hopefully none at all with just the basic buffering.
Since no high-level code can exist at the same address value as cog code then this area from $0000 to $01FF in hub memory is used for various buffers and system and Forth variables. Also the area occupied by the cog images for the serial and ping-ping code plus the PASM kernel cannot be used for high-level code since it cannot be backed up and overwrite the cog images in EEPROM, so this is used instead for file system and other buffers that do not need to be backed up into EEPROM.
Current Memory Map:
The kernel is still not ready for prime time yet but it is interactive and easily compiled with any of the Prop compilers. Default clock config around line 70 is set for 6MHz x8.
Propeller .:.:--TACHYON--:.:. Forth V4.0 DAWN 400161128.1830 Cold start - no user code - setting defaults Setting up dictionary -------------------------------------------------------------------------------- 0001 1240 V4> DICTIONARY WORDS @6CB6 DUP OVER DROP 2DROP SWAP ROT BOUNDS 1+ 1- + - MIN MAX * UM* U/ U/MOD / */ UM*/ ABS -NEGATE ?NEGATE NEGATE ADO DO LOOP +LOOP FOR NEXT AND ANDN OR X OR ROL ROR SHR 8>> SHL 8<< 2/ 2* REV MASK >N >B 9BITS 0= NOT = > U< 0< C@ W@ @ C+! C! C@++ W+! W! +! ! CLOCK CLKIN CLKOUT OUTSET OUTCLR OUTPUTS INP UTS SHROUT SHRINP RESET 0EXIT EXIT NOP 3DROP ?DUP 3RD 4TH CALL JUMP BRANCH> >R R> >L L> (WAITPNE) I SPIWRB SPIWR16 SPIWR SPIRD SPICE UM/MOD64 UM/MO D32 RUNMOD (WAITPEQ) CMOVE (EMIT) (EMITX) CMPSTR LOADMOD !RP !SP COG@ COGREG COG! COGID REBOOT DELTA WAITCNT LAP .LAP HIGH LOW FLOAT NIP 0<> <> C~ W~ ~ SET? WITHIN SET CLR HEX DECIMAL BINARY READBUF KEY WKEY (KEY) EMIT CLS SPACE SPACES BELL CR <CR> .HEX .BYTE .WORD .LONG PRINT . ZPRINT @PAD HO LD >CHAR #> <# # #S <D> PRINT$ LEN$ U. .DEC .DP " SEARCH FINDSTR NFA>BFA VER .VER TACHYON ERASE FILL ms seconds second DISCARD <CMOVE ALLOT ALLOCAT ED HERE @NAMES names REG BUFFERS KOLD WORDS WWORDS DUMP DUMPW DUMPL DUMPC DUMPA DUMPAW .S IF ELSE THEN ENDIF BEGIN UNTIL AGAIN WHILE REPEAT --- \ ' ' ( { } IFDEF IFNDEF " ." (.") == : pub ; ' NFA' V4 *end*A lot of the experimental stuff in it may change or be discarded as I test it out but the kernel seems a lot simpler than the old Tachyon kernel did, both in coding method and in the way all the parts are connected together.
Well, it probably won't be long now before I get to testing a fully loaded system, I am already running various devices.
Why use Tachyon+? Because it is designed to be faster and more compact overall than the "old" Tachyon as well as an improved and faster compiler.
So after T+ we will have the object oriented version T++ then ;-)
I was thinking I had a pretty good understanding of Tachyon internals,
but seems I have to relearn a bit :-)
When T+ is up we can check the old code still runs ... or adapt if needed.
Thanks for sharing the development process with us.
Gives some good insight.
Yes, I am thinking of the ++ version too
The good thing is that most of the PASM code is identical but using 16-bits for the address/code might mean we waste 8-bits for all calls to the first 256 locations but for most other operations it is faster and at its worst it is no slower. Knowing that loading a value of 1,000 used to involve 3 bytecodes and 3 hub accesses but now that is only a single word and access and good for up to values of +32767. High level calls were mostly handled with 2 bytes and a 16-bit vector or once the vector table was full involved 3 bytes for a call. Now there are no vectors or overheads, just one word to call or jump anywhere. Remember that hub access would slow down bytecode which needed extra bytecodes so the single hub access is both fast and more compact overall.
My experiments with the compiler are more interesting though and certainly slowing me down as I attempt to decouple all the various processes to work as a team rather than a bureaucracy, in a manner of speaking. I want this part to be fast and flexible so I can change the way it operates without having to bypass whole sections.
The EEPROM dictionary access needs to be transparent too so it can be done incrementally as required to make as much of the hub RAM as possible available for code. Having a separate data space has helped too to cut down on code memory usage since all variables and most constants are simple one word literals directly compiled into the code. It is easier to create structures with this approach.
I'm hoping that I won't make too many changes that require learning new rules etc. Will removing the traditional "ok" and using the new permanent line and status prompt be a good thing? A lot of things are experimental so feedback is always appreciated as not only does it need to perform, it also has to feel right.
My current terminal session. (need to fix my number radix back up again)
0004 1400 V4> .S Data Stack (0) 0005 1400 V4> 0006 1400 V4> : DEMO 0007 1400+ V4> 128 32 DO I EMIT LOOP 0008 1400+ V4> ; 0009 140E V4> DEMO !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ 0010 140E V4> : DEMOS FOR CR DEMO NEXT ; 0011 1418 V4> 4 DEMOS !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ 0012 1418 V4> $2A . 30 0013 1418 V4> $2A .WORD 001E 0014 1418 V4> HEX 000F 1418 V4> 2A 0010 1418 V4> . 2A 0011 1418 V4> pub STAR 2A EMIT ; 0012 141E V4> STAR * 0013 141E V4> pub STARS 0014 141E+ V4> FOR STAR NEXT 0015 141E+ V4> ; 0016 1426 V4> 8 STARS ******** 0017 1426 V4> 4 longs mynums 0018 1426 V4> 4 bytes myflags 0019 1426 V4> mynum ??? ??? 001A 1426 V4> mynums .WORD 7400 001B 1426 V4> myflags .WORD 7410 001C 1426 V4> ' DEMO 20 DUMPW 1400: 8080 8020 00CA 00C5 06C0 00D1 0021 00CB ............!!.. 1410: 0600 1400 00D7 0021 802A 06C0 0021 00CB ......!!....!!.. 001D 1426 V4> 0022 1440 V4> DECIMAL 0035 1440 V4> 22 == *LED 0036 1440 V4> long rgb 0037 1440 V4> : LED! rgb ! rgb 3 *LED MASK DUP OUTCLR 4 COGREG! [WS2812] RUNMOD ; 0038 145A V4> HEX 0027 145A V4> 400040 LED! 0028 145A V4>So V4> is fine for me.
What I would not like to loose is the possibility to paste in part of a terminal session.
Like you made a dummy definition for OK.
For a variable informational prompt this might be a little more complicated.
I think I would redefine the prompt to s.th. very simple like >
This V4> is quite distracting for the eye,
since the automatic focus of attention is on the start of the line.
And in this respect OK seems more 'ergonomic' ...
What about making the CR being part of the prompt definition.
Then we can have
" <CR>T4> " or " OK<CR>" as the prompt ...
The kernel is quite usable even at the moment.
Propeller .:.:--TACHYON--:.:. Forth V4.0 DAWN 400161201.0930 Cold start - no user code - setting defaults Setting up dictionary -------------------------------------------------------------------------------- ( 0001 1400 ok ) $7F $20 DO I EMIT LOOP !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ ( 0002 1400 ok ) : ASCII $7F $20 DO I EMIT ( 0003 1400 ++ ) LOOP ( 0004 1400 ++ ) ; ( 0005 140E ok ) ASCII !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ ( 0007 140E ok ) TACHYON Propeller .:.:--TACHYON--:.:. Forth V4.0 DAWN 400161201.0930 ( 0261 1772 ok ) End of source code, 0000 errors found Load time = 215080.808us ( 0262 1772 ok ) ( 0263 1772 ok ) ( 0269 1772 ok ) 4 FOR CR ASCII NEXT !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ ( 0270 1772 ok )