SED - The Simple Sector Editor for TAQOZ
Peter Jakacki
Posts: 10,193
My TAQOZ threads are way too big and there are many useful utilities embedded in TAQOZ that make it useful simply as a utility on the P2 such as the disk reporting and formatting etc.
So likewise this is a useful utility for SD cards and FAT32 files in that sectors can be edited in text or hex mode. Originally I wrote this simply to edit config files which had fixed position text and so all I need to do is to have a fixed width format and be able to use the PC terminal cursor keys to move around and overwrite text.
But SED has also been useful for editing file headers and even directly editing the directory sectors and the MBR etc. I'm still enhancing this in between doing some actual real work and assembling and testing boards. The boot splash and sound that I added to TAQOZ was all part of the built-in test procedures I am using but of course are very handy to have.
Here is a screen-shot of SED editing the MBR and because it is directly editing a sector rather than a file, it defaults to 16 character wide hex mode. Other modes up to 128 characters wide remain in pure text mode though since the byte dump would be too wide. BTW, I purposely made it so that changes are never written back to the SD unless explicitly asked to do so with a ^S for save.
SED is designed to work with ANSI/VT102 terminals or automatically adapt if being used on the VGA modes that are built into TAQOZ. (photos later). I have yet to map function and cursor keys from the PS/2 keyboard module.
btw - the editor has climbed to a massive 1300 byte code footprint :)
Comments
I'm trying to follow your SED development and I've noticed you must have patched (extended) the TAQOZ version you're running with the new RTC functions for SED to compile :) .
Those patches don't really affect SED at all although it uses QTIME for a test. At 921600 baud it is very fast to recompile all modules. I use the scripts that I have in my TAQOZ folder, specifically the "s" script to send a file.
In the serial terminal type COLD (or ^Z)
then in the Linux terminal: (assuming that you are using USB0, and wait a couple of second between each command).
./s EXTEND 0
./s FILE 0
./s P2ASM 0
./s MEDIA 0
./s SED
Then backup the system after which you only need to send SED again when it has been updated.
Thanks.
The reason I posted was that when I tried to upload the newest SED onto the full TAQOZ it thew an error at exactly the QTIME entry while the previous SED version compiled without problems.
I got sidetracked with testing QTIME and decided to add it to SED as well. Sorry about that :)
EDIT FILES, SECTORS, FLASH, RAM, or DEVICE
So I can edit any file or any sector on an SD, but what if I want to edit Flash, or even RAM, or any other device just as easily?
Now you can. I have added device support to SED the same as DUMP can access chip registers and memory. Just type the address $1BE00 in this format `$1BE00 SED RAM". Same goes for Flash or I2C devices such as the RTC or CLKGEN or even the UB3 Flash on the P2D2.
I still have some more things to tweak on it but together with the disk utilites and formatting I suppose the binary could be treated as a general-purpose utility even though it has a TAQOZ heart beating away, ready to do more.
Indeed, it is very handy.
It's amazing how such a powerful, feature rich tool can be squeezed in less than 170 lines of code that takes up so little program space.
Makes me wonder how many thousands of hours of a skilled programmers' time must be used to fill with forth programs the entire 16MB of flash that sits on the P2D2 board.
I've also added a soft cursor to the VGA display to help with editing too but I still have to extend the Flash module to use a 4kB page so I can buffer and erase pages of Flash. Once I do that I may as well extend the SD card into that 4kB buffer area too and display in a full screen. My hex edit isn't complete since I need to be able to switch to hex input mode because at the moment it shows the hex but it can only be edited in the text column. Bit by bit it is shaping up very nicely.
SED is not just an Simple Editor - since it sits on a TAQOZ base it is fully extensible. By adding a couple of lines of code I now can specify the "sector" size, and in this case 2kB was all I needed to fill the screen at 360p (106x36 in 5x7). Next thing to add is syntax highlighting.
@"Peter Jakacki" looking for the last TAQOZ following your signature's link, Dropbox returns a 404 error
True.
It doesn't work for me either.
How about using the one that's posted to SourceForge ?
But wait, that one might not incude SED if that's what you're after. I haven't checked.
The pCloud and Source Forge files have been updated and there is a new TAQOZ binary with the current version of SED which is still a WIP but very usable. Later on I will add support for inserting and deleting rather than overwriting and this may end up morphing into a standard text file editor too.
@"Peter Jakacki" thanks for the updated link!
I never worked with SED and there now is another step to climb. But the idea to just edit a SD- file in the propeller is to exciting. So I try to get more acquainted to TAQOS understanding this version of SED. Is anybody else engaged in SED?
I am, from a user perspective but haven't really had much time recently for this.
I had this idea of creating a user manual for it at some point but found it not to be necessary since a quick peek into the SED source code gives you all you need to know. And you can even add features as you please however I have not had a need for it. It is super useful and has great potential in real life (on site) situations. All you need is a terminal connection and off you go.
While I'm not unwilling to decipher this message, I would appreciate a short example for every application: how to edit a file?, flash, ... Would free some clocks and memory for other tasks ;-)
OK, found it and it was as simple as can be: SED SEE.FTH opens the file in the editor
That's the way.
Use ^X as the command sequence key and then H for hex, or T for TEXT etc. Look at the source code as these sequences are very easy to spot. Use standard ANSI settings on your terminal and just use your keyboard's navigation and arrow keys.
I still have to add the part that edits a hex byte and allow a Flash page to be erased and rewritten. Make sure your terminal has enough lines
pri SED^X a>A SWITCH WIDE_ 'W' CASE 128 TXTBLK BREAK TEXT_ 'T' CASE 64 TXTBLK BREAK NARROW_ 'N' CASE 32 TXTBLK BREAK HEX_ 'H' CASE HEXBLK BREAK CLEAR_ 'Z' CASE @TEXT $20 FILL BREAK FORMAT_ ^Z CASE FORMAT.BLK BREAK CLRLN_ 'X' CASE CLRLN BREAK REVERT_ 'R' CASE tcur@ tblk + SED@ DROP BREAK BLANKS_ $20 CASE !TEXT BREAK ;btw, any word with a trailing _ is treated as a single word comment
OK, Thanks Peter, I didn't realize ^X to be the DLE ;-) I use Teraterm on Windows7 and maybe I have not the latest version of SED. My screenshot doesn't show the file name. As I gave it another try: it may depend on VT100 control characters... Have to investigate and understand SED.
The first invoke didn't show the file name in the first row, as your screenshot shows.
After clearing the screen and reinvoke of SED I had the file name at the bottom. I didn't check, if my SED is the latest version, as following the code doesn't make a difference. Would be nice to come up with the version in the head line..
The SED version you can easily find at boot screen here:
The numbers after the EDITOR word are showing the version (date and hours) so it's there and you can compare it with yours.
It's the terminal program thing that you do not see exactly the same graphical output as shown by Peter above. In minicom (under linux) I can see it like that:
So it is not showing the file name at file closure. It really doesn't matter to me.
I will check it out under TeraTerm just to make sure it's formatted correctly.
btw, I also have EDIT as an alias for SED, just it case it seems more natural to say MYFILE.FTH
I'm comfortable now, I can start SED and try to incorporate TAQOZ by going through the source step by step. In contrast to conventional programming where subroutines are lengthy, FORTH calls for subroutines to be very short, like "tcur!" for "tcur !". That allows to write 10xmove for 10 xmove for 10 x move and maybe is the way to create very compact code. On the other hand you have to know all of those words.
Excellent!
I see three good points captured here about forth:
1. comfortable - true
2. very compact code - true
3. you have to know all of those words - well, the point is, you actually don't.
It's like with the natural human language. While having a huge vocabulary at you disposal you only use the small subset of words for the majority of communication tasks most of the time and the more often you use them the better you know them. And then you can actually create words that feel natural to you or look up a word that does what you need and give it an alias that you are comfortable with.
This is the version I use:
Yes, a word should be obvious and while there is no harm in having tcur ! throughout the code, I find that when it is common I may make a word out of it which shouldn't need explaining, so tcur ! becomes tcur!
In fact the word is private meaning the name is not necessary once it has been compiled. However tcur! is also used directly by tcur+! and tcur-!. Mind you, this was q&d code and when I get around to tidying it up, it may look totally different.
SED is also written with short lines so that it can readily fit into a 64 character wide block if need be, and then SED can EDIT SED.
I'm fine with the q&d style, as I said: if we have a function: move_X_by_10 we would write it by_10_X_move and when I send forth program as a command to an axis controller I can write 10 X MOVE to a serial device that can move any distance on any axis. Next I can enumerate all tokens so I send 1 2 3 and I can even enumerate 1 2 3 to 4 so the complete command for this special purpose is just "4" . There is no common understanding that forth and Tachyon can change the world once th P2D2 is mass fabricated ;-)
This is SED
This is SEDKEY
And a look behind the scenes:
Still a lot to do and to learn...
Just a question: is there a difference between "private" and "pri" or "public" and "pub"
Yes and no. The public and privates word sets the default attribute for the names of variables and constants whereas pub and pri are used in place of the colon : to create a new word that will be set to public or private at creation (but does not affect the default settings). The colon : is always public which is the default.
Rarely do I reclaim dictionary memory during development and while I have plenty of memory on the P2, the dictionary searches are nonetheless faster with less clutter when private headers are stripped with RECLAIM. With the MEDIA and EASYNET modules loaded on top of all the other modules it takes 588us @320MHz to find the end of the dictionary. When I RECLAIM it releases 5,201 bytes of memory and now it only takes 288us.
TAQOZ# " END" @WORDS LAP FIND$ LAP .LAP --- 92,072 cycles= 287,725ns @320MHz okI should really optimize the search though to search common kernel words first.
Last last for tonight: [C] prevents preemption of the next word, but what means [G] ?
And, just to be sure: there is no check for EOF, the terminal window block shows a page of 2048 and there is no scrolling? Just q&d?
Correct, the [C] forces compilation of the next word that follows and is a compile-time directive. However [G] for want of a better name is compiled so that when it is used in a pre word such as SED, which is used as a command, it can effectively grab any parameters out of the compile buffer by executing and releasing the buffer. BTW, the temporary compile buffer (let's call it the TCB) is simply at the end of code memory where the next new code definition would go anyway, except it doesn't permanently allocate any memory.
Ok, that may be a head-spinner so let's look at what happens when you type in "commands", that is, anything not within a definition that we just want to "interpret" such as "@ROOT SED SECTOR" to edit the root sector. When we type @ROOT then it is found in the dictionary and compiled in the TCB since unlike Forth in general which interprets word by word, TAQOZ will compile a complete command line before executing it (at full runtime speed)
So far we have the code address of @ROOT in our TCB but when we come to SED, after TAQOZ has found the name in the dictionary it sees that it is a preemptive word and instead of compiling it, ends up executing it. Now if SED just went straight ahead and waited for the next word SECTOR, and then figures that this is one of its directives rather than a file name, then which sector? We know we passed a parameter to it already, except it's not on the stack, it's in the TCB. So what SED needs to do to grab whatever was passed to it, is to execute the TCB which then executes @ROOT and leaves the sector value on the stack. This is what [G] does at runtime and if we had typed 0 or @ROOT 4 + or any calculation etc, then SED would have a sector number on the stack to use.
Have a look at what SED is at runtime by seeing inside it and notice that there is no [C] FOPEN just FOPEN (OPEN-FILE) but [G] is there as its alias GRAB (just as EDIT is an alias for SED).
EDIT: You know what, maybe I will make OPEN-FILE a runtime pub function and leave FOPEN as a terminal command.
Ok, I changed it so that FOPEN is an immediate word and also returns with console messages, whereas OPEN-FILE returns with the sector handle where 0 is false.
I could do the same with some other words too. However I could make [C] also compile a [G] by default.
Here's a commented listing for SED
pre SED ( SED <file> or 0 SED SECTOR or 0 SED FLASH ) --- set defaults FCLOSE 64 TXTBLK blksect ~ --- grab any optional parameters from tcb before opening a file [G] [C] FOPEN @FILE IF 0 SED: BC@ BC! THEN --- not a file, then check for keyword FLASH FLASH? IF SED: FLASH@ FLASH! THEN --- or RAM RAM? IF SED: C@ C! THEN HEXBLK --- or SECTOR SECTOR? IF OPEN-SECTOR 0 SED: BC@ BC! THEN --- don't know PRINT" BAD COMMAND " ;If you don't know, you know now:
All the private words are not callable. After a RECLAIM. But if you reload a module, they are here. And you can easily test your normally hidden words. I think you should avoid using a name twice anyway as it may hurt at compile time. Funny things might happen if there is WORD1 public WORD1 private, WORD2 may reference WORD1 (which one) and after RECLAIM the late WORD1 is taken from the dictionary while now the early WORD1 is found. Or something like that, I will not test it if not by accident