Screen editor suggestions
Peter Jakacki
Posts: 10,193
in Propeller 1
Although I've been laid up this past week with a nasty flu I've been busy expanding the functions of my Tachyon system. I wanted to build in a clock-by-clock Prop emulator via the virtual memory layer of the file system and include a disassembler as part of that. Well I started on that interesting road well enough, and then I thought of writing a nice PASM assembler that could take handle Prop tool compatible files etc and then I thought about editing text files on the Prop using an ANSI terminal for display. I also looked at writing HTML to a file(s) via serial on the PC/tablet and have the browser auto-refresh as well.... However the ANSI terminal approach is easier for the moment so I wrote a nice little text file viewer in about 1KB as a prelude to introducing editing functions and at present it displays line numbers, title, position, size etc. BTW, I have the baud rate set at 2M baud for faster screen refresh and function and navigation keys that generate escape sequences are supported too and of course you can escape to the command line to directly access Forth commands.
Now this is the problem, while it is easy enough to delete characters by temporarily substituting them with a suitable non-printing code, it is a different matter when we insert characters as we cannot read the whole file into RAM and it is way too slow to insert character by character directly into a file. My initial thinking is to mark this insertion point with a special 8-th bit character such as $C1 etc, with a unique one for every insertion point. This code can point to a virtual memory buffer where the inserted text is kept and the screen refresh always diverts to and displays these buffers when it encounters an insert code. Whenever the file is saved it basically rewrites the whole file into a new file and uses the old one as a backup.
This is what I'm working on right at the moment but maybe somebody has some experience with doing something similar in a memory constrained system. Ideas?
Now this is the problem, while it is easy enough to delete characters by temporarily substituting them with a suitable non-printing code, it is a different matter when we insert characters as we cannot read the whole file into RAM and it is way too slow to insert character by character directly into a file. My initial thinking is to mark this insertion point with a special 8-th bit character such as $C1 etc, with a unique one for every insertion point. This code can point to a virtual memory buffer where the inserted text is kept and the screen refresh always diverts to and displays these buffers when it encounters an insert code. Whenever the file is saved it basically rewrites the whole file into a new file and uses the old one as a backup.
This is what I'm working on right at the moment but maybe somebody has some experience with doing something similar in a memory constrained system. Ideas?
1854 x 1488 - 96K
Comments
When a line was modified, or a new line inserted, it was allocated from the end of the file image in virtual memory, and the linked-list pointers were updated to point to the new line. At the end of the editing session the linked-list image in virtual memory would be written to an output file in a normal text-file format.
I never finished the editor, but I did get it to a point where I could modify, delete and insert lines.
One thing I should mention is that this is not a stand-alone screen editor with all the hub memory to play with, it's just an app that runs on top of the Tachyon system with assembler, emulator etc and as such is limited by the available memory. I'm trying to keep this to just the sector buffers that are used by the file system, only 2K altogether.
I have never found any SD FAT32 files that are fragmented ever, plus when I create a file I scan the FAT for contiguous clusters, and maybe this is what FAT32 is doing on a PC as well. So I work safe in the knowledge that once I have the starting sector for a file that I can then access it as linear memory. So that does simplify things.
But what you were doing sounds somewhat similar except there is the overhead in building a double-linked list file when you go to edit. How did you find it in terms of responsiveness? With my code substitution method I am simply substituting characters with insert or delete codes and maintaining the combined insert texts in another file. So the original file doesn't change in size, it remains static. It is only when it is saving that it might pause for a second or so while writing the output file etc.
How were you displaying the text? ANSI terminal?
insert this at pos whatever
delete from-to.
for a text editor, you can chunk it down to lines, but say for a hex editor lines are not needed at all, since just data.
and a simple diff file can work byte-wise so no need to care about lines and track them.
Your code substitution method needs some escaped character and just works for deletions, as far as I could follow the idea.
with some diff-file approach you don't need any escaped characters. So you also could use it for HEX instead of ASCII...
Since your file system uses predefined big files, size should not be the problem for a diff file and with your skills and the power of FORTH you might be able to code the solution to reduce fragmentation in that tmp-diff-file.
just pondering, you asked for,
Mike
My static edit scheme only ever overwrites the original file with delete or insert codes in the range $80-$FF. The file size remains the same. There are no escape sequences since I am not trying to edit UTF-8, just 7-bit ASCII.
"The quick brown fox leaps"
Backspace on "leaps" but only overwrites "deleted" code character
"The quick brown fox ~~~~~"
The ~ is used to represent the otherwise non-visible delete code so the display only shows:
"The quick brown fox "
We now insert "jumps" in its place and in this case only delete codes are overwritten effectively one for one.
"The quick brown fox jumps"
Now we really do insert some extra text"
"The quick brown sly fox jumps"
But the original file looks like this:
"The quick brown <INS1>ox jumps"
Where the "f" in fox has been replaced with an <INS1> code, maybe $C1 perhaps.
<INS1> points to the string "sly f" in the insert buffer file and so when the text is displayed the <INS1> code will display the insert text.
When we save the file we essentially display the whole processed file as it would appear on the display but written to the output file instead so there is no setup time required nor much in the way of runtime processing and writing the output is a straightforward process.
Editing the "edits" are another matter but if they are small snippets with plenty of virtual memory allocated then they can be handled more conventionally.
think outside the box.
You are focused on a line editor, because you already have it. Do a small side step, for a moment.
You are half ways there, with your
"<INS1> points to the string "sly f" in the insert buffer file and so when the text is displayed the <INS1> code will display the insert text."
Just add
"<Del>points to no string, just skips 'N' chars of the original file and displays nothing"
and keep all <ins> and <dels> in your buffer file, not changing the original one, just using pointers to the original file, it is static.
So do for delete, what you do for insert, somehow.
Mike
My thinking with using substitution codes was to minimize the display overhead so that the file could just be read sequentially. If I were to maintain pointers to the text file then it would mean that I would have to scan the pointers for every line at least and perhaps relax and simply type out the line if no pointers were found.
I know how I would do it if I had more processing power and memory, even P2 style, but P1 fully loaded already is where it gets tricky.
When the insert/delete file is fairly small it wouldn't be much bother I suppose. What happens when it gets a bit heavy? Do I autosave and clean it up? Maybe. It's food for thought.
At start, your change-buffer contains one entry, 0 for file start, 'nnn' for file length.
if you insert or delete something, the list gets divided in not-changed and changed chunks. But still kept in address order?
You can then display/save all chunks by following the change-buffer.
either it is a chunk of the original file, or a chunk out of your change buffer.
Mike
Yes. That is one of the reasons they invented red Wine.
Mike
Once I moved to a full screen editor I was impressed (Vedit was my first screen editor). However, it proved not so reliable, so I always saved to a new file before editing. Once this saved my life when the second half of the file was corrupted, and I didn't discover this for a many edit sessions later.
My file system is based on FSRW, which could result in fragmented files over time. However, the VM file is created once and never deleted, so it is guaranteed to be contiguous.
I haven't noticed any slowness due to the double-linked list. However, most of the files I have edited are fairly small. Even on a large file I don't think there will be much of a speed penalty. The speed could be improved by having an index array that points to every N-th line.
The editor that I was working on was a version of the Unix ED line editor. I added a mode that repaints the screen after each edit. Once I got this working I was going to port the VM stuff to my VI editor. My VI editor supports text, PST and ANSI terminal modes.