Of course to make this work well you *really* do need seeks. If you want I can hack up a
version of fsrw to give you seeks (although no guarantees that they will be super fast).
Also of course fsrw should support r+ and w+ (so you can read and write from the same
file).
Also you want two files open . . . some work to go there still.
1. If the file fits in memory, in total, then automatically it becomes a totally
in-memory editing system. Indeed, the first step to realizing this
system is to ignore the file prefix and suffix and just make the in-memory
part work.
2. Note that I can open a 1MB file, insert 314 chars at the top, and not
have to write any data until the file is closed (or the user scrolls down
a bunch) because my file "hole" can be smaller than my memory
"hole".
3. The amount of I/O done is proportional to the cursor movement;
during most inserts, you're just sticking characters into the in-memory
hole so no file I/O.
4. No need for a progress bar; things will be plenty fast.
5. The hole approach also pretty much automatically takes care of the
alignment requirements; you can make the file prefix and suffix
*always* be on sector boundaries, and the in-memory buffer
can take care of the odd sizes.
If you go this way let me know and we can talk about making the
needed changes to fsrw to support what you need without waiting for
the full next release.
Here is an incomplete version which I'm trying to get up to just a scrolling file Viewer (With no FileBuffer) before I get into the editing part, I could use some help on speed, I have attached the textfile that its set to open upon startup.
Here is whats up:
No FileBuffer to speak of (1 byte)
Uses Baggers TV_Text_Half_Height Driver (Slightly Modified)
FSRW of course
KeyBoard Only for now
DropDownMenu works but most of the Commands generated as a result do nothing
PageDown Works to Scroll Down one page
DownArrow Key Works to scroll Down 1 line
File ->Exit Command Reboots
Help ->About Command throws a pop-up window up
File-> Open lets you scroll through files on SDCard, but has not been resized for new display driver and is very slow, and does not really open a file anymore
Its just been changed to not use a file buffer so wordwrap is permanently off right now
I have no idea how to make it scroll sideways when wordwrap is off for very dense documents like the one I included.
I also have all the object/file names very short so it can compile under sphinx later without too much hassle.
There is probably some dead code in there for things that did at one time work..but since further changes have broken,,,please excuse that.
So rokicki,
I dont know if you've had a chance to take a look at what was posted but I wanted to run some ideas on improving speed by you:
I think this document is an extreme example in terms of filesize and line length, but then if this can be done quick almost any file can.
Currently, I keep track of the location within the file on SD with a Long, and when you scroll, the program puts one character at a time into a 1 byte throw away var, this is to increment the internal counter that fsrw uses until I get to where I need to in the file. I'd like to pread the entire amount I dont want away...is there a way to pread into thin air, as a 1 command way to jump to the file location I'd like to be at before I start putting data onto the display? Is this what you mean by file seek?
Or another way I was thinking to speed things up would be to pread by 10 or 20 at a time until I got close, then do single characters.
Another question, what is a good way to move through the file to detect the end of a line other than the single char read&check that I have going on. I'm heading into the office to do some more on this.
After some work on the File-> Open function to use the now available screen realestate with Baggers TVTHH driver, I did some tests to see how I can boost speed in traversing the SD Files.
Instead of moving ahead by 1 byte until I reached the point I need to be at (RED LINE), I did a bunch of tests to see where the best options where:
The best result ended up using a 1000byte garbage buffer to either move through the file 1000Bytes at a time...or if my place is less than that, jump immediately to that (BLACK LINE).
The test was to open the file and scroll down 1 line at a time, 50 times
I have more work to do but progress is being made.
Try adding these two routines to fsrw. These are read-only seek/tell, and will only seek within the
current cluster. Otherwise they return -1. Make sure to test the return code and bail if it's -1.
Since the cluster size on SD cards for the prop *should* be 32K, this should enable you to keep
moving forward as I add a more comprehensive seek.
pub seek(pos) | delta
if (direntry or pos < 0 or pos > filesize)
return -1
delta := (floc - frem) & ((SECTORSIZE << clustershift)-1)
if (pos < delta or pos > delta + (SECTORSIZE << clustershift))
return -1
if (pos < floc - bufend or pos => floc - bufend + SECTORSIZE)
' must change buffer
delta := floc + frem
floc := pos & (SECTORSIZE - 1)
frem := delta - floc
pfillbuf
else
bufat := pos & (SECTORSIZE - 1)
return 0
pub tell
return floc + bufat - bufend
Will do rockicki, I will post results thanks!!
I had a mini-breakthrough and am having some small alignment issues, as soon as thats done I will hit the fsrw with those changes!
Great; if you want quicker response, mail me at rokicki at gmail.com.
I can add more to fsrw if you need; in particular, adding full seek would
not be *terribly* difficult (I'd have to add a firstcluster variable though).
hey rokicki,
Thanks for that code, I'm not having too much luck with it atm, however I wouldn't take time away from your scheduled release cycle becuase I just took a peek and my garbage buffer seek system only takes up .5 mil clocks out of a 30mil+ routine. [noparse]:)[/noparse] It does grow the farther into the file you are, but still its nothing compared to what else needs to be done.
Is it possible to have another cog prepping stuff for me in the background? I'm thinking thats one way to go, but who knows.
No sweat. I think you just want to keep it as simple as possible. The SD routines should be plenty fast for what you need, and if they are not
right now they will be when you need them.
I can give you a "pread" that, if you pass it a 0 for a buffer, it doesn't "use" the buffer. Would that be better?
Have you thought about my "hole" system? It's really sweet a simple, just a few integers giving the size of each portion.
Couple of questions..thanks about the mounting everytime! Duh, its the simple things we miss [noparse]:)[/noparse]
I have been opening/closing the file everytime I pull data from it, in some respects so that I may "reset" fsrws counter into the file because I cannot read backward (which would be really sweet).
But your saying that I can leave the file open.. and seek to whatever location within the file I want, without preading to that known location from the begining?
That sounds extremely handy, I can omit a couple of loops with that and some fsrw commands as well.
I will have another go at the code you put up!
the pread command that just increments fsrw's file place would be a perfect drop-in for what I have, but the seeking sounds way better.
I hope this thread isn't too annoying... [noparse]:)[/noparse]
In an effort to speed up the updating of the display with a pre-populated display buffer, I chopped up a version of the out method in TV_TextHH, outspec:
PUB outspec(dataptr,d) | i, k
repeat d
i := byte[noparse][[/noparse]dataptr++]
case i
$00: screen[noparse][[/noparse]row * cols + col] := (color << 1 + 32 & 1) << 10 + $200 + 32 & $FE
other: screen[noparse][[/noparse]row * cols + col] := (color << 1 + i & 1) << 10 + $200 + i & $FE
col++
This method allows you to send it the address of the begining of an array of characters, and the length of that array.
I use 0s in my array to note an area that has no characters, hence the substitution of a Spc char instead (32) for display.
I'm happy to say that this has cut my speed cost by more than half!
However I can see there is definately more that can be done. Like getting rid of the multiplication.
I only have to call this once per-line from PrEditor, but it still executes a loop per character to do the shifts..etc and place that result into the screen[noparse]/noparse buffer.
Does anyone know of a faster way to fill up the screen[noparse]/noparse buffer knowing that:
I will never be using this method to update any colors
I want to do d many characters
I'm thinking of keeping a version to display a page, and to display a line.
Yeah, I have made some progress:
This is for updating the whole editing section with one line from the main program:
PUB outwa(dataptr,d, l) | i, k
screenpos := row * cols
k := (color << 11) + 544
repeat l
repeat d
i := byte[noparse][[/noparse]dataptr++]
if i == 00
screen[noparse][[/noparse]screenpos + col] := k
else
screen[noparse][[/noparse]screenpos + col] := (color << 1 + i & 1) << 10 + $200 + i & $FE
col++
screenpos := screenpos+col+1
col := 1
Its faster, but I'm dumping the screen[noparse]/noparse array now to see how I can go further, without all the "(color << 1 + i & 1) << 10 + $200 + i & $FE"
I think you want to go to a model where you are *scrolling* the screen array (usually) and then just
repaint the new line (on the top or bottom). You cna scroll the screen array using one of the memory
copy routines, and they will run at assembly speed and give you all the lines but one for "free".
Of course you'll need to figure out when you can do this and when you can't, but that's not tough code.
With this you should get a massive improvement and be to the point where the seek routine is the
dominant speed issue.
Hey rokicki,
Can you give me a hand with the seek?
LongCounter:=0
repeat until LongCounter == FilePlaceCount
if FilePlaceCount > (LongCounter+1000)
sdcard.pread(@Garbage[noparse][[/noparse]0],1000)
LongCounter:=LongCounter+1000
else
sdcard.pread(@Garbage[noparse][[/noparse]0],(FilePlaceCount-LongCounter))
LongCounter:=LongCounter+(FilePlaceCount-LongCounter)
Above is what I currently am using,·FilePlaceCount is the position I want to jump too, it positions me to grab the next byte in the file that I'm using.
sdcard.seek(FilePlaceCount)
And above (error checking aside) is what I'm trying to replace it with.
The·pread after always returns the same data wether I seek(0) or seek(500), the data at the very begining of the file.
Just incase I didn't copy it correct, here is the·seek I am using:
pub seek(pos) | delta
if (direntry or pos < 0 or pos > filesize)
return -1
delta := (floc - frem) & ((SECTORSIZE << clustershift)-1)
if (pos < delta or pos > delta + (SECTORSIZE << clustershift))
return -1
if (pos < floc - bufend or pos => floc - bufend + SECTORSIZE)
' must change buffer
delta := floc + frem
floc := pos & (SECTORSIZE - 1)
frem := delta - floc
pfillbuf
else
bufat := pos & (SECTORSIZE - 1)
return 0
Its setup to scroll 300 Lines, and at first its faster than anyone could possibly read...but as we get deeper my·CornySeek routine really starts to get tired [noparse]:)[/noparse]
Hey rokicki,
That seek (when within cluster) just decimates the methods I have been using:
You can clearly see when the tell reports <> 0 and hit hops back to the CornySeek.
Again, the file I'm testing with is a real beast....each line has 300~350 characters before a Carriage Return. When testing with some Spin Files it never even goes out of cluster!
So thanks again Tom, I look forward to the release.
I have some other directions to go in with this·for a bit.
@Baggers - Hey did you see that your TVTextHH is what I'm working off of!
Right, I figure 32K is large enough to make it an "editor" while I get real seek working in parallel.
Real seek should be plenty fast because most of the time you're going to be scrolling up and
down within a small region of the file so within a cluster.
Comments
version of fsrw to give you seeks (although no guarantees that they will be super fast).
Also of course fsrw should support r+ and w+ (so you can read and write from the same
file).
Also you want two files open . . . some work to go there still.
1. If the file fits in memory, in total, then automatically it becomes a totally
in-memory editing system. Indeed, the first step to realizing this
system is to ignore the file prefix and suffix and just make the in-memory
part work.
2. Note that I can open a 1MB file, insert 314 chars at the top, and not
have to write any data until the file is closed (or the user scrolls down
a bunch) because my file "hole" can be smaller than my memory
"hole".
3. The amount of I/O done is proportional to the cursor movement;
during most inserts, you're just sticking characters into the in-memory
hole so no file I/O.
4. No need for a progress bar; things will be plenty fast.
5. The hole approach also pretty much automatically takes care of the
alignment requirements; you can make the file prefix and suffix
*always* be on sector boundaries, and the in-memory buffer
can take care of the odd sizes.
If you go this way let me know and we can talk about making the
needed changes to fsrw to support what you need without waiting for
the full next release.
Here is an incomplete version which I'm trying to get up to just a scrolling file Viewer (With no FileBuffer) before I get into the editing part, I could use some help on speed, I have attached the textfile that its set to open upon startup.
Here is whats up:
No FileBuffer to speak of (1 byte)
Uses Baggers TV_Text_Half_Height Driver (Slightly Modified)
FSRW of course
KeyBoard Only for now
DropDownMenu works but most of the Commands generated as a result do nothing
PageDown Works to Scroll Down one page
DownArrow Key Works to scroll Down 1 line
File ->Exit Command Reboots
Help ->About Command throws a pop-up window up
File-> Open lets you scroll through files on SDCard, but has not been resized for new display driver and is very slow, and does not really open a file anymore
Its just been changed to not use a file buffer so wordwrap is permanently off right now
I have no idea how to make it scroll sideways when wordwrap is off for very dense documents like the one I included.
I also have all the object/file names very short so it can compile under sphinx later without too much hassle.
There is probably some dead code in there for things that did at one time work..but since further changes have broken,,,please excuse that.
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Prop Forum Search (Via Google)
I dont know if you've had a chance to take a look at what was posted but I wanted to run some ideas on improving speed by you:
I think this document is an extreme example in terms of filesize and line length, but then if this can be done quick almost any file can.
Currently, I keep track of the location within the file on SD with a Long, and when you scroll, the program puts one character at a time into a 1 byte throw away var, this is to increment the internal counter that fsrw uses until I get to where I need to in the file. I'd like to pread the entire amount I dont want away...is there a way to pread into thin air, as a 1 command way to jump to the file location I'd like to be at before I start putting data onto the display? Is this what you mean by file seek?
Or another way I was thinking to speed things up would be to pread by 10 or 20 at a time until I got close, then do single characters.
Another question, what is a good way to move through the file to detect the end of a line other than the single char read&check that I have going on. I'm heading into the office to do some more on this.
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Prop Forum Search (Via Google)
Instead of moving ahead by 1 byte until I reached the point I need to be at (RED LINE), I did a bunch of tests to see where the best options where:
The best result ended up using a 1000byte garbage buffer to either move through the file 1000Bytes at a time...or if my place is less than that, jump immediately to that (BLACK LINE).
The test was to open the file and scroll down 1 line at a time, 50 times
I have more work to do but progress is being made.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Prop Forum Search (Via Google)
Let me look at your code, and then I'll hack together a seek command
that should be (almost) instantaneous.
If that's all that's holding things up, that's the least I can do.
current cluster. Otherwise they return -1. Make sure to test the return code and bail if it's -1.
Since the cluster size on SD cards for the prop *should* be 32K, this should enable you to keep
moving forward as I add a more comprehensive seek.
pub seek(pos) | delta if (direntry or pos < 0 or pos > filesize) return -1 delta := (floc - frem) & ((SECTORSIZE << clustershift)-1) if (pos < delta or pos > delta + (SECTORSIZE << clustershift)) return -1 if (pos < floc - bufend or pos => floc - bufend + SECTORSIZE) ' must change buffer delta := floc + frem floc := pos & (SECTORSIZE - 1) frem := delta - floc pfillbuf else bufat := pos & (SECTORSIZE - 1) return 0 pub tell return floc + bufat - bufendI had a mini-breakthrough and am having some small alignment issues, as soon as thats done I will hit the fsrw with those changes!
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Prop Forum Search (Via Google)
I can add more to fsrw if you need; in particular, adding full seek would
not be *terribly* difficult (I'd have to add a firstcluster variable though).
Now its time to Apply those fsrw changes and see what that brings!
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Prop Forum Search (Via Google)
Thanks for that code, I'm not having too much luck with it atm, however I wouldn't take time away from your scheduled release cycle becuase I just took a peek and my garbage buffer seek system only takes up .5 mil clocks out of a 30mil+ routine. [noparse]:)[/noparse] It does grow the farther into the file you are, but still its nothing compared to what else needs to be done.
Is it possible to have another cog prepping stuff for me in the background? I'm thinking thats one way to go, but who knows.
Thanks ppl, I'll keep you posted.
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Prop Forum Search (Via Google)
No sweat. I think you just want to keep it as simple as possible. The SD routines should be plenty fast for what you need, and if they are not
right now they will be when you need them.
I can give you a "pread" that, if you pass it a 0 for a buffer, it doesn't "use" the buffer. Would that be better?
Have you thought about my "hole" system? It's really sweet a simple, just a few integers giving the size of each portion.
Couple of questions..thanks about the mounting everytime! Duh, its the simple things we miss [noparse]:)[/noparse]
I have been opening/closing the file everytime I pull data from it, in some respects so that I may "reset" fsrws counter into the file because I cannot read backward (which would be really sweet).
But your saying that I can leave the file open.. and seek to whatever location within the file I want, without preading to that known location from the begining?
That sounds extremely handy, I can omit a couple of loops with that and some fsrw commands as well.
I will have another go at the code you put up!
the pread command that just increments fsrw's file place would be a perfect drop-in for what I have, but the seeking sounds way better.
thanks for all your help,
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Prop Forum Search (Via Google)
I hope this thread isn't too annoying... [noparse]:)[/noparse]
In an effort to speed up the updating of the display with a pre-populated display buffer, I chopped up a version of the out method in TV_TextHH, outspec:
PUB outspec(dataptr,d) | i, k repeat d i := byte[noparse][[/noparse]dataptr++] case i $00: screen[noparse][[/noparse]row * cols + col] := (color << 1 + 32 & 1) << 10 + $200 + 32 & $FE other: screen[noparse][[/noparse]row * cols + col] := (color << 1 + i & 1) << 10 + $200 + i & $FE col++This method allows you to send it the address of the begining of an array of characters, and the length of that array.
I use 0s in my array to note an area that has no characters, hence the substitution of a Spc char instead (32) for display.
I'm happy to say that this has cut my speed cost by more than half!
However I can see there is definately more that can be done. Like getting rid of the multiplication.
I only have to call this once per-line from PrEditor, but it still executes a loop per character to do the shifts..etc and place that result into the screen[noparse]/noparse buffer.
Does anyone know of a faster way to fill up the screen[noparse]/noparse buffer knowing that:
I will never be using this method to update any colors
I want to do d many characters
I'm thinking of keeping a version to display a page, and to display a line.
Thanks,
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Prop Forum Search (Via Google)
PUB outspec(dataptr,d) | i, k, z z := row * cols + col repeat d i := byte[noparse][[/noparse]dataptr++] case i $00: screen[noparse][[/noparse]z++] := (color << 1 + 32 & 1) << 10 + $200 + 32 & $FE other: screen[noparse][[/noparse]z++] := (color << 1 + i & 1) << 10 + $200 + i & $FE col++This is for updating the whole editing section with one line from the main program:
PUB outwa(dataptr,d, l) | i, k screenpos := row * cols k := (color << 11) + 544 repeat l repeat d i := byte[noparse][[/noparse]dataptr++] if i == 00 screen[noparse][[/noparse]screenpos + col] := k else screen[noparse][[/noparse]screenpos + col] := (color << 1 + i & 1) << 10 + $200 + i & $FE col++ screenpos := screenpos+col+1 col := 1Its faster, but I'm dumping the screen[noparse]/noparse array now to see how I can go further, without all the "(color << 1 + i & 1) << 10 + $200 + i & $FE"
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Prop Forum Search (Via Google)
repaint the new line (on the top or bottom). You cna scroll the screen array using one of the memory
copy routines, and they will run at assembly speed and give you all the lines but one for "free".
Of course you'll need to figure out when you can do this and when you can't, but that's not tough code.
With this you should get a massive improvement and be to the point where the seek routine is the
dominant speed issue.
Can you give me a hand with the seek?
LongCounter:=0 repeat until LongCounter == FilePlaceCount if FilePlaceCount > (LongCounter+1000) sdcard.pread(@Garbage[noparse][[/noparse]0],1000) LongCounter:=LongCounter+1000 else sdcard.pread(@Garbage[noparse][[/noparse]0],(FilePlaceCount-LongCounter)) LongCounter:=LongCounter+(FilePlaceCount-LongCounter)Above is what I currently am using,·FilePlaceCount is the position I want to jump too, it positions me to grab the next byte in the file that I'm using.
And above (error checking aside) is what I'm trying to replace it with.
The·pread after always returns the same data wether I seek(0) or seek(500), the data at the very begining of the file.
Just incase I didn't copy it correct, here is the·seek I am using:
pub seek(pos) | delta if (direntry or pos < 0 or pos > filesize) return -1 delta := (floc - frem) & ((SECTORSIZE << clustershift)-1) if (pos < delta or pos > delta + (SECTORSIZE << clustershift)) return -1 if (pos < floc - bufend or pos => floc - bufend + SECTORSIZE) ' must change buffer delta := floc + frem floc := pos & (SECTORSIZE - 1) frem := delta - floc pfillbuf else bufat := pos & (SECTORSIZE - 1) return 0Thanks,
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Prop Forum Search (Via Google)
fix it.
Sorry I was out all day . . .
-tom
Its setup to scroll 300 Lines, and at first its faster than anyone could possibly read...but as we get deeper my·CornySeek routine really starts to get tired [noparse]:)[/noparse]
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Prop Forum Search (Via Google)
Here's a new seek. With it your code screams!
Note however that there's still a problem; you are seeking past the end of the file
(at least in my test) at the end.
if (direntry or pos < 0 or pos > filesize) return -1 delta := (floc - bufend) & - (SECTORSIZE << clustershift) if (pos < delta or pos > delta + (SECTORSIZE << clustershift)) return -1 if (pos < floc - bufend or pos => floc - bufend + SECTORSIZE) ' must change buffer delta := floc + frem floc := pos & - SECTORSIZE frem := delta - floc pfillbuf bufat := pos & (SECTORSIZE - 1) return 0Oh yeah, ..is there a filesize method?
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Prop Forum Search (Via Google)
PUB GetFileSize
return filesize
That's off the top of my head, without opening the source.
but IIRC that's all you need. [noparse]:)[/noparse]
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
http://www.propgfx.co.uk/forum/·home of the PropGFX Lite
·
But today I have to do some housework. Maybe tonight.
That seek (when within cluster) just decimates the methods I have been using:
You can clearly see when the tell reports <> 0 and hit hops back to the CornySeek.
Again, the file I'm testing with is a real beast....each line has 300~350 characters before a Carriage Return. When testing with some Spin Files it never even goes out of cluster!
So thanks again Tom, I look forward to the release.
I have some other directions to go in with this·for a bit.
@Baggers - Hey did you see that your TVTextHH is what I'm working off of!
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Prop Forum Search (Via Google)
Real seek should be plenty fast because most of the time you're going to be scrolling up and
down within a small region of the file so within a cluster.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
http://www.propgfx.co.uk/forum/·home of the PropGFX Lite
·