Virtual Memory
MagIO2
Posts: 2,243
Outcome of the night:
I added some functions to FSRW. It uses the SD-card as virtual memory. Currently it's only for testing the principle. So, only long is currently supported, but if you can't wait for the finished version, feel free to add any datatype you need. In the end·my goal would be to add it directly to SDSPIPASM.
Preparation:
You need a file called swap.sys on your SD-card. It's very essential that the file is not fragmented. So, you should format the SD-card, copy swap.sys, then copy anything else you need on the card.
swap.sys can have any size you need. I started with 128MB.
FSRW.SPIN - changes:
Some minor changes in the existing code. Mount has to initialize the variables used. Popen has to return the sektor where the file swap.sys starts.
First parameter of the new functions is a virtual memory adress. It's adress space is given by the size of the file. Please note that currently there is no check for the adress. So, if the adress exceeds the size of the file, then this version will overwrite whatever is on the SD card.
vrMemLong will copy a long from the virtual memory to the HUB-RAM adress given as second parameter.
vwMemLong will copy a long from HUB-RAM to virtual memory.
vMemFlush is only needed, if you want to make sure that changes are written ... for example before someone switches off the prop - but who should do that ? ;o)
Have fun!
PS: No FSRW relevant things are touched, so FSRW will function in parallel. That means that you can have a file open while using virtual memory. Read directorys ... whatever
PPS: should have called the functions peek and poke ;o)
Post Edited (MagIO2) : 11/14/2009 11:40:05 PM GMT
I added some functions to FSRW. It uses the SD-card as virtual memory. Currently it's only for testing the principle. So, only long is currently supported, but if you can't wait for the finished version, feel free to add any datatype you need. In the end·my goal would be to add it directly to SDSPIPASM.
Preparation:
You need a file called swap.sys on your SD-card. It's very essential that the file is not fragmented. So, you should format the SD-card, copy swap.sys, then copy anything else you need on the card.
swap.sys can have any size you need. I started with 128MB.
FSRW.SPIN - changes:
Some minor changes in the existing code. Mount has to initialize the variables used. Popen has to return the sektor where the file swap.sys starts.
var
long swapsektor ' this is the first sektor of file swap.sys
long swapact ' this is the actual sektor read into swapspace
long swapactadr ' this is the virtual memory adress at which the actual sector starts
long swapmod ' flag to show if the memory has been changed
long swapspace[noparse][[/noparse]SECTORSIZE>>2]
pub vrMemLong( adr, buffer )
if (adr&$fffffe00) <> swapactadr
if swapmod
sdspi.writeblock( swapact, @swapspace )
swapactadr:=(adr&$fffffe00)
swapact:=swapactadr>>9+swapsektor
sdspi.readblock( swapact, @swapspace )
swapmod:=0
long[noparse][[/noparse] buffer ]:=swapspace[noparse][[/noparse] (adr&$1fd)>>2 ]
pub vwMemLong( adr, buffer )
if (adr&$fffffe00) <> swapactadr
if swapmod
sdspi.writeblock( swapact, @swapspace )
swapactadr:=(adr&$fffffe00)
swapact:=swapactadr>>9+swapsektor
sdspi.readblock( swapact, @swapspace )
swapspace[noparse][[/noparse] (adr&$1fd)>>2 ] := long[noparse][[/noparse] buffer ]
swapmod:=1
pub vMemFlush
if swapmod
sdspi.writeblock( swapact, @swapspace )
pub mount(basepin) | start, sectorspercluster, reserved, rootentries, sectors
{{
' Mount a volume. The address passed in is passed along to the block
' layer; see the currently used block layer for documentation. If the
' volume mounts, a 0 is returned, else abort is called.
......... here comes the mount-code .....
}}
[color=red]swapsektor := popen( string("swap.sys"), "r")
swapactadr := -1[/color]
return 0
pub popen(s, mode) | i, sentinel, dirptr, freeentry
' ........
if (mode == "r")
frem := SECTORSIZE << clustershift
if (frem > filesize)
frem := filesize
[color=red]return datablock[/color]
First parameter of the new functions is a virtual memory adress. It's adress space is given by the size of the file. Please note that currently there is no check for the adress. So, if the adress exceeds the size of the file, then this version will overwrite whatever is on the SD card.
vrMemLong will copy a long from the virtual memory to the HUB-RAM adress given as second parameter.
vwMemLong will copy a long from HUB-RAM to virtual memory.
vMemFlush is only needed, if you want to make sure that changes are written ... for example before someone switches off the prop - but who should do that ? ;o)
Have fun!
PS: No FSRW relevant things are touched, so FSRW will function in parallel. That means that you can have a file open while using virtual memory. Read directorys ... whatever
PPS: should have called the functions peek and poke ;o)
Post Edited (MagIO2) : 11/14/2009 11:40:05 PM GMT
Comments
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Computers are microcontrolled.
Robots are microcontrolled.
I am microcontrolled.
But you·can·call me micro.
Want to·experiment with the SX or just put together a cool project?
SX Spinning light display·
Want cheap wholesale electronic parts?
Transistor parts wholesale
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Links to other interesting threads:
· Home of the MultiBladeProps: TriBlade,·RamBlade,·SixBlade, website
· Single Board Computer:·3 Propeller ICs·and a·TriBladeProp board (ZiCog Z80 Emulator)
· Prop Tools under Development or Completed (Index)
· Emulators: CPUs Z80 etc; Micros Altair etc;· Terminals·VT100 etc; (Index) ZiCog (Z80) , MoCog (6809)
· Search the Propeller forums·(uses advanced Google search)
My cruising website is: ·www.bluemagic.biz·· MultiBladeProp is: www.bluemagic.biz/cluso.htm
Low bandwidth example:
320 x 240 x 2bpp x 60fps needs a bandwidth of 1.1Mb/seconds.
Random access writing to a SD card using FSRW isn't very fast.
I think that rewriting the same memory segments on a SD card 60 times per second isn't a very good idea.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
24 bit LCD Breakout Board coming soon. $21.99 has backlight driver and touch sensitive decoder.
This and the excellent driver obex.parallax.com/objects/475/ Mike has written would be a better solution.
I have been including this driver on my prototypes and it works fine for both high-speed RAM and Flash RAM needs.
Swap space and code storage would be a simple application for this driver. RAM uses a couple of the 23K256 devices,
and code & data use a Winbond W25X16AVDAIZ 2MB (or more). Works for me
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
JMH
Post Edited (James Michael Huselton) : 11/15/2009 4:32:06 PM GMT
I was nearly sure that someone already did something similar.
There is no doubt that it works excellent. But then I read about files agin .... that's exactly the overhead that I want to get rid of. Of course I did not check yet what the winbond driver overhead is here, but in the SD card driver the overhead was 25%. Just for reading a file sequentially. Writing will be even more overhead.
So, my intention is more a memory based access. The programmer should know what to find in which memory location. In the end I'd like to store bitmaps and menu-structures in this memory, which most likely will not be changed anymore.
On the other hand a set of functions can be created that's completely independend from the hardware. And you could even map different kind of hardware in different memory sections...
Let's see where it ends ;o)
I don't even use SD cards anymore.
Anyway, good luck with your Prop tinkering - remember to have fun!
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
JMH
PS: Little explaination what you find inside of the attachment:
pub getSwap
Returns the adress of the buffer used for swapping ... with this adress you can directly use the 512 bytes
pub checkSektor( adr )
Checks if the given adress is available in the currently loaded sektor. If not it's checked whether the current sektor has been modified and saves it on SD card if needed. Then the sektor which contains the adr is loaded. This is usually needed internally by the other functions, but it also makes sense in combination with getSwap.
pub vPeek( adr )
return the long found at the given adress
pub vPoke( adr, val )
set the long at the fiven adress to the value
pub vwMemCopy( adr, buffer, len )
copies a buffer from anywhere in HUB-RAM to the sektor. It's not allowed to exceed the 512 byte boundary. For example adr=500 then you can't copy more than 12 bytes. This restriction is not checked in the function because that would increase runtime.
pub vwMemCmpCopy( adr, buffer, len )
this also copies a buffer into the sektor-data, but it first checks that the data is really different. If not the sektor is not marked as 'being changed'.
pub vrMemCopy( adr, buffer, len )
This copies content of adr into the buffer.
pub vrMemLong( adr, buffer )
pub vwMemLong( adr, buffer )
Those are the functions of the proof of concept phase ... Guess these will turn out to be useless.
pub vMemFlush
If you want to make sure that the data is written to SD card, call this ... In between several vwXXXX-calls it's not needed, as a sektor is written automatically in case it has been changed and is replaced by an other sector.
pub vMemRollback
If you don't want to save the change you did in ONE sektor.
Post Edited (MagIO2) : 11/23/2009 7:04:47 PM GMT
Main changes:
1. Moved forward to FSRW 2.5 as I got it running on my setup ... 1.689 kB/s ;o)))
2. Added slots (4) to vMem.
A slot is so to say a window to the memory. With 4 slots it's possible to have a look at 4 different memory locations. This is usefull to avoid reload/writeback if you need to work in different memory areas at the same time. Main reason to implement that is that CassLan is currently trying out vMem for his editor project (preditor).
But be aware, if 2 slots are used for memory areas closer than 512 bytes, things might go wrong, as currently each slot would work on it's own copy.
Per default all slots work on the same file swap.sys, but you can also assign slots to other files.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Links to other interesting threads:
· Home of the MultiBladeProps: TriBlade,·RamBlade,·SixBlade, website
· Single Board Computer:·3 Propeller ICs·and a·TriBladeProp board (ZiCog Z80 Emulator)
· Prop Tools under Development or Completed (Index)
· Emulators: CPUs Z80 etc; Micros Altair etc;· Terminals·VT100 etc; (Index) ZiCog (Z80) , MoCog (6809)
· Search the Propeller forums·(uses advanced Google search)
My cruising website is: ·www.bluemagic.biz·· MultiBladeProp is: www.bluemagic.biz/cluso.htm
When I run the latest slot test, I get "Yest4" printed 4 times.
I have re-formatted my card and created a new swap just incase it was fragmented.
Any ideas?
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
NYC Area Prop Club
Prop Forum Search (Via Google)
·
I'm sorry that you seem to be a beta-tester for the vMem ;o) Fixed the problem you mentioned. I tried to save some calculations by using a variable in variable space accessed by several functions. But in some cases and for some functions the code which calculates the value has not been executed.
But I added 2 new functions to compensate your inconvenience. It's vw/rMemCopyNL. The NL (for no limit [noparse]:o[/noparse])version will work even if you cross the sector boundaries and allows to copy any size of buffers. For now you should rather switch to that. Later on I'll also provide a CmpNL version, as this will minimize write access to SD card. But comparing any size of data is to slow in SPIN. So, I'll do that when porting vMem to PASM.
Post Edited (MagIO2) : 12/6/2009 9:13:45 PM GMT
It isn't really virtual memory unless it's the executable's memory map you are virtualising. A virtual machine, ie: Both the Spin engine and the LMM engine, could present the right environment but, I assume, with a performance penalty to every instruction.
Memory in 'big processors' is also organized in pages where the OS if free to swap on disk and back as needed. That's why I've chosen the name virtual memory.
Also, paging is not virtual memory. Paging is a means of copying and tracking data between multiple memories. Virtual memory is a means of transparently remapping a program's memory accesses - hiding things like paging and framentation from the executable.
In a single tasking processor, eg: a Cog, there is no need for VM. The level of sharing doesn't warrant it. And what sharing there is can be satified with libraries, a filesystem and some conventions. The Hub is a little unique in that multiple programs really are sharing it but, as of yet, it's always a single application and therefore can be managed as I've listed.
There is potential for the LMM to grow into something that can manage multiple large applications at once. Maybe there should be some thought put into the PropII for facilitating both virtual memory and memory protection under the LMM.
So I switched to the latest posting of the VM version on fsrw.
The test runs great, however when I switched all the commands in my program to the NL versions, the program hung and the SD card would no longer mount, I had to re-format on the PC.
Would using the NL version with a len value of 1 do this?
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
NYC Area Prop Club
Prop Forum Search (Via Google)
·
But if your len it 1 you are always safe, even if you use the old non NL functions.
lenInSector seems to be the problem. I limited it to the max length of the rest of the buffer. But it needs an additional limit max to len.
·should help.
Same for the vrMemCopyNL. Could the prop 2 maybe be build with a little more tolerance against programming errors??? ;o)
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
NYC Area Prop Club
Prop Forum Search (Via Google)
·
I just built a new board with a uSD card socket, and for some reason PrEditor can't mount it. When I run the fsrw2.6 tests, I can choose any block mode and they all work, I tried copying the block file and switching with the vMem version of fsrw, still no good. The uSD card is good, if I put in inside an SD Adapter·I can use it on my uther board.
Any chance you could update vmem for 2.6? Maybe this would solve my issues, or if you have any ideas.
Thanks,
Rick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
NYC Area Prop Club
Prop Forum Search (Via Google)
·
as long as nothing has changed in the interface, it's easy to put vMem into a new FSRW version. You simply add the stuff to initialize vMem into the mountExplicit. As this will open the swap.sys to find out the first sector of the file popen has to return that sector instead of returning 0. So, that's the second change.
Then copy all the v.... functions and variables and you're done.