Gap Buffer - Success!!!
CassLan
Posts: 586
I have a gap buffer working, you can change the buffer values, point within file and it still splits apart·a file into: File, Mem, Mem, File...and can then put them all back together properly (and pretty quickly too)
So working off the ABCDE model, what other functions aside from the initial split up and rebuilding should this buffer have?
Moving data·from B -> D, including the neccesary file writes to A (appending) and the re-writing of D (due to removing data from the begining of this file)
D->B
Anything else?
I've had enough coding for today but will be thinking about this for tommorow.
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
NYC Area Prop Club
Prop Forum Search (Via Google)
Post Edited (CassLan) : 12/7/2009 9:20:44 PM GMT
So working off the ABCDE model, what other functions aside from the initial split up and rebuilding should this buffer have?
Moving data·from B -> D, including the neccesary file writes to A (appending) and the re-writing of D (due to removing data from the begining of this file)
D->B
Anything else?
I've had enough coding for today but will be thinking about this for tommorow.
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
NYC Area Prop Club
Prop Forum Search (Via Google)
Post Edited (CassLan) : 12/7/2009 9:20:44 PM GMT
Comments
I added slots to the interface of virtual memory. So, now there are 4 independent sector buffers. So, you can work in 4 different areas of virtual memory.
And I still think that virtual memory would be helpful for you. Gap buffers can easily be implemented with it without the need to rewrite file for buffer D. Say you allow files with 1MB size. You simply define address $00_0000 - $0F_FFFF being buffer A and address $10_0000 - $1F_FFFF being buffer E. You only initially load the file in buffer E and never have to take care of A and E unless you want to save the changes.
Accessing those memory locations will automatically copy stuff in HUB-RAM buffers which are then buffers B and D. C is all the space between A+B and D+E.
All you need are two pointers which show at which memory location your left of cursor text ends and your right of cursor text starts. If you move your cursor one position you simply do a vbPeek on one side and a vbPoke on the other and update the pointers.
Using peek and poke with single bytes is also safe when you cross sector boundaries.
Are you interested?
Post Edited (MagIO2) : 12/3/2009 11:11:33 PM GMT
using fsrw and the gap buffer (and a sector-reversed file for E) but if you don't need it, let me know. Essentially writesector() and readsector() would manage
the FAT chains for you (and extending the file as needed) without giving the full complexity of a seekable-read-then-write file (which is quite doable but has some
complexities; after a seek, you don't know if the file is open for "read" or "write" until the next operation).
I'm excited about this and can't wait to see what you come up with! Gap buffers are *elegant*.
Yes very interested, what your basicly saying is if I use the VM functions, I can treat A and E just like memory, and shift them around with out implicitly re-writing files as house keeping? Because the VM functions take care of that for me?
Rokicki,
If what I think MagIO is talking about is really what he means
Is the E sector-reversed in order to append to the *top*?
It sounds to me like your saying some of those housekeeping functions will be made much easier with your new functions, honestly though I really don't know what you mean by "managing FAT chains"
Reading and writing sectors, I just don't understand what that means. I am very appreciative of your help and everyone elses also. I need to do some reading on how FAT really works, as well as sectors, clusters etc.
Night all,
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
NYC Area Prop Club
Prop Forum Search (Via Google)
·
The rationale for the new functions is to give you a file (or rather, a set of files) that you can both
read and write from randomly, as long as it's in sector-sized chunks, without having to worry about
saving space on the volume or anything like that. It is so you *don't* have to learn anything about
the FAT filesystem or sectors or clusters or any of that.
I'm not familiar with MagIO's specific VM implementation, but it should probably do what you
need. If you use it, I recommend you keep "E" reversed just so you don't need to worry about
it growing down into A.
Please keep me posted on what's happening and how I can help. I look forward to a hosted editor.
-tom
Yes. The swap-file is fixed. There is no need to extend or shrink it. You simply define where you want to have the A buffer and where to have B buffer. Of course the file size is then limited to the size of the buffers, but you can be very wasteful with that limit, as a swap-file can have the size of the whole SD card if you want. Just to mention it ... FSRW is still full functional, so you can open files with FSRW while accessing vMem.
Here is what I have up to now.
Somehow the vwMemRevCopy seems to be not working correct ... have to fix that ... but earliest in 10 hours
vwMemRevCopy is exactly meant for initial writing of the file into vMem (E buffer) in reverse order, as that's where a freshly opened file has to start. Now the virtual memory has 4 slots. You need them to minimize writebacks. In the previous version accessing different areas would have led to writeback to SD card with each and every change, because buffer A is at a completely different location than buffer E....
Hmmmm in terms of vMem we should maybe change our naming. You have 2 adress pointers. One pointer that points to the end of text in A+B and one pointer that points to the start of text in D+E. You always point to the end/start of the text, so in fact we could say there is no B and no D for the editor. Writeback of A and E is managed by vMem. B and D are hidden in vMem. So, I'd suggest to talk about left hand buffer (LHB) and right hand buffer (RHB) instead.
Having 4 slots means that now 1 slot can be used to work in the LHB, another slot can be used to work in the RHB, one slot can be used as copy'n'paste buffer and one slot can be used for reading from anywhere for example for refreshing the screen after colsing a drop-down menu or the file-save dialog. You could also put all your strings that you need in your editor into vMem and gain some valuable HUB-RAM this way. Or maybe you want to put complete dialog boxes into vMem.
Moving the cursor to left or right should be pretty easy:
move to the right -> vbPoke( 0, LHBpointer++, vbPeek( 1, RHBpointer-- ) )
move to the left -> vbPoke( 1, RHBpointer++, vbPeek( 0, LHBpointer-- ))
(not tested yet ;o)
I will definately keep you posted, I'm sure I will need help
MagIO2,
Sounds good, although I DL'd what you posted and the SD card won't mount and is returning -13.
I did have to change the pll to 16x and the xtal to 5Mhz, and SDBasepin to 0; I'm using a demo board.
note: I tried with a 1GB SanDisk, and a 2GB Kingston, and also tried re-formatting the Kingston; same result
Maybe the problem is that I have no "swap.sys" file on the SD card. Can you give me an idea how I can make one?
I think a function to create one should be implemented.
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
NYC Area Prop Club
Prop Forum Search (Via Google)
Post Edited (CassLan) : 12/4/2009 12:49:12 PM GMT
You can use any file you find which has the desired size, make a copy and rename it. Filesize will then limit the address that you can use. (You have to take care of this limit by yourself as checks cost additional runtime. We don't do hobby programming here! ;o)
And I guess my SD card setup is no longer standard, since I had problems to run version 2.5 of FSRW. I soldered my SD card socket on a prototyping PCB. And now the order of signals is different. You need to have a look at the mount function of the sd card driver. There the mask is created according to the base pin you give. I swapped the lines CLK and CS. Just swap them back and it should run. Sorry for that. Guess I should rewire to match the common setup again.
pub start(basepin, lcdcmdbuf, mask)
'
' Initialize the card! Send a whole bunch of
' clocks (in case the previous program crashed
' in the middle of a read command or something),
' then a reset command, and then wait until the
' card goes idle.
'
do := basepin++
cs := basepin++ <<<<<<<< swap these
di := basepin++
clk := basepin++ <<<<<<<< swap these
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
NYC Area Prop Club
Prop Forum Search (Via Google)
·
I have made a demo to excersize my understanding of the vm components as well as the gap buffer implementation.
This takes a file ("edittext.txt"), and given a point, splits the file into A & E, or LHB & RHB.
then when complete, it makes a new file "backup.txt" and attempts to re-create the original file by putting the pieces back together.
I did this all within one slot of the VM interface, using an offset of $100_000 for the RHB side.
my swap.sys is 4mb.
While the resulting "backup.txt" is the correct size, the data is incorrect.
Also in my initial attempt, it got stuck when trying to write to a different slot, which is why I kept all operations in slot 0
Also I would like to be able to check for the existance of "swap.sys" and its file size, but I am unable to do so.
Do you think you could take a look, and see if I'm using the functions incorrectly?
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
NYC Area Prop Club
Prop Forum Search (Via Google)
Ok so I was using a buffer to move between VM and RAM of 2k...thats a no-no
I switched the buffersize to 512 and it works great! (reading instructions works wonders)
This also fixed my inability to to use more than one slot.
I have a question about the slots...are they overlaping each other, with the reason for their existence so that you may perform operations on a slot to another?
Or are they completely seperate slices of memory that do not over lap? I'm assuming the former but I want to ask.
Still curious about checking the existence of swap.sys on the card and its filesize, the reason being, I would like to be able to make sure people have one, that its big enough, and also to create one for them if they dont have it.
Thanks,
Oh here is the working Gap buffer split-rebuild demo utilizing different slots.
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
NYC Area Prop Club
Prop Forum Search (Via Google)
Now we move the cursor again. Again first VM reads the sector of LHB because the sector in the buffer is one coming from RHB. And the sector would be most likely the same that has been loaded in the previous peek. But even more annoying is, that before loading that sector, the changed one has to be written back to SD card, so that the change is not lost.
To avoid unnecessary reads AND WRITES I implemented those slots.
Only thing you have to take care of is that it's a bad idea to operate in 2 slots that use addresses that are close together and are placed in the same sector. This is currently not checked. Let's see what experiences we will have with that - maybe it turns out to be a must have. But currently I'd like to avoid adding code for that as code means making things slower.
Once the SPIN version of VM is finalized I plan to code it in PASM. Then you'd call a peek or poke and before your SPIN function returns, the value is available ;o)
If swap.sys is available or not can be put into the FSRW startup, as there I already try to open that file, as I need the information where to find the first sector of that file. Creating that file on the propeller makes no sense, as this file has to be copied to the SD card after formatting it (just to make sure that swap.sys will not be fragmented).
How to check the size I currently don't know. Maybe FSRW will help? In the directory entry the size should be available.
What I found is:
repeat sdcard.pread(@buffer,buffersize) sdcard.vwMemCopy(0, filelocation, @buffer, buffersize) filelocation := filelocation + buffersizepread returns the bytes really read. You should change the loop to repeat until this returns something smaller than the buffersize.
And of course you then can add the real number of bytes to the filelocation.
For the real Gap Buffer you have to load the file into RHB, because when you load a file most editors start with a cursor pointing to the top of the file and not to the end ;o) Being on top of the file means LHB empty and RHB full. Being at bottom of file means LHB full and RHB empty.
AND... I need to get the RevCopy running, as it is very useful to store the text in reverse order in the RHB. Meaning that your RHB offset would start at $1F_FFFF.
But VM uses a pre-defined address space and of course we can put all the RHB stuff at the end and 'grow' to the inside. We simply decrease the RHB pointer. To save the changes into a file, we simply write from LHB adress $00_0000 up to LHB pointer and then append from LHB pointer up to the end of LHB.
Understand what I mean? Yes ... now I understand, what I mean ;o)
Maybe your code is already doing exactly that and I did not understand while reviewing. But I found a bug in VM during this understanding/testing process. Please replace FSRWC.spin.
Post Edited (MagIO2) : 12/4/2009 9:13:38 PM GMT
I was thinking the same thing about the reverse...but I didn't think about decreasing where the RHB starts, thats a good one!
I'm going to work on a scroll test now, I may not finish it today though.
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
NYC Area Prop Club
Prop Forum Search (Via Google)
·
I added a new version to the virtual memory thread using FSRW 2.5, which is much faster than the 1.7 version I used in old vMem.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
NYC Area Prop Club
Prop Forum Search (Via Google)
·
I am having an issue with moving bytes from the upper portion of memory (LHB) to the lower portion of memory (RHB).
It seems to work well, but you will notice that at some point the data in the lower portion gets replaced by something else.
I have posted a video illustrating this if you don't feel like running in on your setup.
http://www.youtube.com/watch?v=M7jHslA2bbg
The code that is happening during the scroll loop that affects the data is:
sdcard.vrMemCopy(0, LHB, @Charbuffer, 1) LHB-- RHBOffset-- sdcard.vwMemCopy(1, RHBOffset, @Charbuffer, 1)I know that MagIO said Peeking or Poking is safe even if you cross a sector boundary, this is using the MemCopy functions, and perhaps I am crossing a sector and thats the issue.
I will try using Peek/Poke instead to see if that resolves the issue, any input is appreciated.
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
NYC Area Prop Club
Prop Forum Search (Via Google)
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
NYC Area Prop Club
Prop Forum Search (Via Google)
·
I'll have a look at your code now.
Just a tip: You can put the SD card in a PC SD card reader and see if the file has been loaded correctly into the swap-file. I use an editor that displays the file in HEX - like UltraEdit or HexEd.
Post Edited (MagIO2) : 12/6/2009 7:33:22 PM GMT
sdcard.vrMemCopy(0, LHB-19, @buffer, 20)
sdcard.vrMemCopy(1, RHBOffset, @buffer[noparse][[/noparse]20], 20)
seem to be problematic as well. I told you that you have to take care about the 512byte boundaries by yourself, but you don't.
I'll add a function that allows you to read/write across boundaries.
When I use the SlotTest in the newer release of VM, I get "Yest4" 4 times.
I misunderstood what you meant; so if I want to copy 1500bytes starting at VM address 255, I have to make·4 commands?
256bytes in address (255-511), 512bytes in address (512-1023), 512bytes in·address (1024-1535), and last 220bytes (1536-1456).
Is this the correct way to do this?
You don't have to do that, I understand your desire for as much speed as possible, I will work with Peek/Poke until I get the concept done, then I will switch to MemCopy with sector checking.
I see I have been not using vMemFlush..this fixed some things for me.
How often should I be using it?
Thanks for your help,
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
NYC Area Prop Club
Prop Forum Search (Via Google)
·
But to make life easier for other usecases it makes sense to have the NL versions.
Please find the latest versions in virtual memory thread.
Flush is normally included in all the memory access functions. So, if you do a transfer from somewhere to virtual memory it's only needed after transfering the last chunk - before switching off or maybe before reading the data again using a different slot.
Post Edited (MagIO2) : 12/6/2009 9:51:25 PM GMT
Video of the below program:
http://www.youtube.com/watch?v=6ZHuU-VC1zw
I have tried to break it in everyway I can..changing the values of the LHB/RHB, changing the direction in which I shift bytes, using different file sizes etc etc.
It still loads the file the way it's supposed to
It still moves the data the way its supposed to
It still re-assembles the data to an exact match!
PS - I have been using this nifty program to compare files...it's been very helpful ( http://winmerge.org/·)
I am·pretty confident this will be the basis of Prediter!!!
Thanks to all of the people who have been helping!
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
NYC Area Prop Club
Prop Forum Search (Via Google)
The vFlushAll is only needed AFTER the repeat loop in your scrolling demo. Whenever poke leaves a sector the changes are saved automatically. Currently you would write the whole sector for each byte change.
Nice to see that it works.