Standard for SD Driver interface... We need one!
Cluso99
Posts: 18,069
I have been doing some more work on my Propeller OS, expanding upon the work done by others.
The one thing that bugs me is that there are a number of nice drivers out there, but they all have different interfaces. I know Ross (Catalina) has advocated for standards too, as have others. The PASM interface is the part that I am referring to.
When I wrote the SD section for heater's ZiCog, I initially used the fsrw routines. Later, I updated to using the newer mb-rawb-spi26. This required quite a bit of reworking. Then I progressed to using KyeDos as the basis for my Prop OS. Kye uses a different interface variant to the previous two fswrw's.
I would like to be able to substitute all these different drivers, so that I can find the best one for the job. Kye has done a great job with the upper FAT16/32 interface in spin. But some times, not all these routines are required. I am looking at the OS keeping a cog running the PASM driver running, and be usable for ZiCog, the Prop OS, etc.
What these interfaces have in common are these...
Obviously the pins need to be initialised.
Kye's driver has interaction for CSD & CID Registers from the cards and handles SD, MMC, plus SDHC & SDXC support. I am not quite sure at this time why the upper drivers need to know this.
Other parameters are preset by spin in Kye's code, but I think these could be set by the pasm code at initialisation time.
Read Sector...
Also known as ReadBlock, gives a command such as "R" or "r", and passes an address (pointer) to where the 512 byte buffer is located in hub. Upon completion, the command is cleared and aresult is returned.
Write Sector...
Also known as WriteBlock, gives a command such as "W" or "w", and asses an address (pointer) to where the 512 byte buffer is located in hub. Upon completion, the command is cleared and a result is returned.
Additional commands...
Kye also uses "M" for mount and "O" for operation. I cannot actually see what "O" really does.
In addition to this, Kye's pasm driver keeps 2 bytes updated with CD (card detect status) and WP (card write protect status).
Ross (Catalina) uses a 2 tiered single long and double long for communicating. Since both are always implemented, I believe a 4 long block (per cog) would be a better and simpler use, but that is only IMHO.
But we do need to decide if it will be "R" or "r", "W" or "w", and how we pass the command and return the status.
Oh, and don't forget, Kye's Driver and Spin FAT code is part of a Parallax App Note
What I would like is that the PASM section becomes a standalone driver, and can be invoked by any language, not just spin. In my OS, it can remain resident in a cog, to be used by any program that is subsequently loaded, including ZiCog/CPM2.2, etc. I guess I need to see what Catalina does as it uses a modified Kye driver.
So, what are your thoughts, or do I do my own thing???
The one thing that bugs me is that there are a number of nice drivers out there, but they all have different interfaces. I know Ross (Catalina) has advocated for standards too, as have others. The PASM interface is the part that I am referring to.
When I wrote the SD section for heater's ZiCog, I initially used the fsrw routines. Later, I updated to using the newer mb-rawb-spi26. This required quite a bit of reworking. Then I progressed to using KyeDos as the basis for my Prop OS. Kye uses a different interface variant to the previous two fswrw's.
I would like to be able to substitute all these different drivers, so that I can find the best one for the job. Kye has done a great job with the upper FAT16/32 interface in spin. But some times, not all these routines are required. I am looking at the OS keeping a cog running the PASM driver running, and be usable for ZiCog, the Prop OS, etc.
What these interfaces have in common are these...
- Initialisation command
- Read Sector command
- Write Sector command
- optional RTC command(s)
Obviously the pins need to be initialised.
Kye's driver has interaction for CSD & CID Registers from the cards and handles SD, MMC, plus SDHC & SDXC support. I am not quite sure at this time why the upper drivers need to know this.
Other parameters are preset by spin in Kye's code, but I think these could be set by the pasm code at initialisation time.
Read Sector...
Also known as ReadBlock, gives a command such as "R" or "r", and passes an address (pointer) to where the 512 byte buffer is located in hub. Upon completion, the command is cleared and aresult is returned.
Write Sector...
Also known as WriteBlock, gives a command such as "W" or "w", and asses an address (pointer) to where the 512 byte buffer is located in hub. Upon completion, the command is cleared and a result is returned.
Additional commands...
Kye also uses "M" for mount and "O" for operation. I cannot actually see what "O" really does.
In addition to this, Kye's pasm driver keeps 2 bytes updated with CD (card detect status) and WP (card write protect status).
Ross (Catalina) uses a 2 tiered single long and double long for communicating. Since both are always implemented, I believe a 4 long block (per cog) would be a better and simpler use, but that is only IMHO.
But we do need to decide if it will be "R" or "r", "W" or "w", and how we pass the command and return the status.
Oh, and don't forget, Kye's Driver and Spin FAT code is part of a Parallax App Note
What I would like is that the PASM section becomes a standalone driver, and can be invoked by any language, not just spin. In my OS, it can remain resident in a cog, to be used by any program that is subsequently loaded, including ZiCog/CPM2.2, etc. I guess I need to see what Catalina does as it uses a modified Kye driver.
So, what are your thoughts, or do I do my own thing???
Comments
There are three layers of fat driver
First, PASM code, doing a physical job - transfers bits to and from SD card, all these command bytes, etc,
Second, low level interface. It is what you described. Init, sector read, sector write
Third, file system interface, to make file open, close, read and write with all this file system maintenance. Boot sector, FATs, directories.
So if you want only sector operations , you can strip Kye's driver from all this filesystem stuff.
OS environment is needed for the Propeller. For example, some system variables at upper bytes of hub ram, then the frame buffer and defined command set. Some cogs running i/o system: vga, kbd, mouse, filesystem. Then, the main application, written in Spin, C, whatever, can communicate with this OS layer via these system variables.
con sys_command=$00007FFC sys_var_block=$00007FF0 (...) pub something (...) s:=string("This is a test string") long[sys_var block]:=10 'x long[sys_var_block+4]:=5 'y long[sys_var_block+8]:=s 'pointer to string long[sys_command]:=$00020001 ' group 02 - vga commands, command 01 - output a string repeat until long[sys_command]==$FFFFFFFF ' wait for OS to do its job (...)I don't know if there is any such defined system environment for the Propeller. If not... it has to be defined and written.
I am looking (I will be) separating the PASM code from the SPIN code in Kye's driver. But I am looking for a consistent way for this to be done. I am not sure if the MB-RAWB code is any faster.
Johnny Mac: I am always after the fastest which is why I keep looking. If someone comes up with a new way, then I want it to be compatible so I don't have to change lots of code as I have in the past.
Oh, and I omitted the "B" boot (cold and warm) command from the drivers - this is a requirement but only takes a small amount of code.
Several people have made hard disk and floppy disk device drivers, and they could use FAT filesystem code, too, as long as it wasn't written assuming it would be used only for SD cards.
The FAT code could call general device init, read, write sector and device status functions independently of the underlying device.
I'd suggest removing RTC code. While it may be handy to have around, it really doesn't belong in an SD card block device driver.
I'd also suggest allowing for general purpose control commands with status response in the interface. For SD cards, it's nice to be able to query the SD card CSD register for the size of the card, rather than relying on what the FAT configuration says. I added that functionality to a version of Kye's driver for one project I had.
Other kinds of devices have their own control needs that must pass through the interface, like spinning up or down hard or floppy drive motors and querying for device readiness before attempting to do data access, so a general status interface probably would be useful.
Lastly, I'll second Jon's request that the underlying device block driver run as fast as possible, for those applications that need the raw speed.
If it happens to make someone else happy too then that's great.
Starting a project just to make everyone happy is "adventurous" at best.
Good luck.
Bean
I want to decouple the FAT driver software from the PASM driver software, and have a standard interface agreed. Then, we can substitue various version of the FAT with various versions of the PASM driver, choosing the best of each side for the particular purpose. Then we all (spin, basic, catalina, forth, gcc, zicog/cpm, etc) will benefit from a standard interface.
Currently, everyone who works on an SD driver is making a different interface, so it is not possible without work (considerable in some instances) to substitute drivers, nor even test the benefits of each.
I suspect that the mb-rawb-spi26 outperforms Kye's sd-mmc-fatengine in the pasm section. Neither is compatible, so its hard to actually decide if this is true.
Kye's FAT (spin) code is bloated. No offence to Kye, because that is precisely his stated intention - to provide all the services required and support all cards and FAT16 & FAT32. Kye has done a magnificent job too. And if we use bst with the optimiser, unused routines will be discarded.
Once I get done with my next release of the CMUcam4 over at SparkFun I will be done developing SPIN code. I'll just have to provide maintenance and that's it.
It would be nice to have a dead code eliminator is SPIN...
Thanks,
However, that does not change my thread.
I believe it is necessary to have a consistent interface from any language to the pasm driver.
It doesn't matter how good or mainstream it becomes, there will always be some who don't want it.
Take for example: "I dislike C with a vengence."
So, do what makes you happy. If others enjoy it too, then that's great.
I'm not sure how moving to C helps solve the problem presented by Cluso in this thread.
We have a bunch of drivers that provide access to block devices but they are not interchangable.
This is a similar problem to the often discussed general problem of creating PASM code drivers that can be used from any language rather than relying on Spin for the set up and runtime interface.
The same issues apply to C when you get there.
As an aside, for the same funtionality, LMM code compiled from C is somewhat bigger than Spin byte codes. So as much as I love to have C for the Prop it is not a cure all.
@Cluso99 - My block driver is really designed for use with just my file system driver. It has a lot of stuff going on in it to prevent bad usage cases. For example: My file system will detect if you try to switch an SD card out for another SD card while the driver has the SD card mounted. In the single threaded case this is not a problem. But imagine that you have two copies of my driver both mounting the SD card. Then it would be possible with two threads (cogs) using the SD card not to see that the card was swapped unless you implement CID checking. My block driver binds the mounted file system to the CID (128-bit unique value for every card). This prevents any weirdness from going on.
I have the pasm code staying resident, but I next need to implement the "standard" interface between the spin and pasm programs. I do not propose to have more than 1 program at a time using the pasm driver. With my hardware, it is not possible to change the SD card while power is applied because I share pins with other devices (often SRAM) and this is quite likely to kill the SRAM chip, but at a minimum, if the SRAM is being accessed it will corrupt the data.
Without anything better being discussed, I will implement a 2 long interface, where 1 is the command/status register, and the other long is a pointer (usually to the buffer). I may break the first long into 4 bytes.
Here are my proposed commands...
"I" = initialise (i.e. start - passes pins, etc)
"R" = read sector
"W" = write sector
"B" = warm boot (keeps resident cogs active)
"X" = cold boot (full boot)
I am not sure if "M" mount is required or not.
I came across this a long while back and have recycled the concept over and over in my own code such that little remains of the original Cluso99 memory access code except the acknowlegement text. But it is essentially 4 longs - a command, a source, a destination and a count.
I am amazed at how many things can be done with such little data. I think it is because if you want to send more data (eg from the Source) then just point the Source at the location of the data list. Ditto the Destination. I use this for external memory, for graphics commands and even for sending commands to a pasm routine to decode a .bmp file.
In keeping with your last post "I" is always reserved as "Initialise". But other letters of the alphabet can be used as you like. At least that is what I have been doing using the "Cluso99" standard. It works for me!
Neither the C standards or POSIX say anything about how to write device drivers. They specify the API between applications and the operating system.
Operating systems are free to have whatever device driver and other internal interfaces they like.
So what Cluso is asking could be seen as like asking for a Linux device driver module to be usable in BSD, Minix, Windows or whatever environment.
But one could also look at as: loading a driver to COG is like those binary firmware blobs that wifi or other devices need to be loaded with before they will work. The same binary blob is used from Linux or Windows or whatever.
Looked at that way it seems like a good idea to standardize things. I just get the feeling that is akin to herding cats though.
As you say, it works for a lot of things, saves allocating 1 long, and then 2 longs, and maybe more (as in Catalina). If I allocated a fixed 4 long per cog, then whatever is running can use that 4 long interface.
Here is an excert of the code from my TriBlade/RamBlade Driver
VAR ' communication params(5) between cog driver code - only "command" and "errx" are modified by the driver long command, hubaddrs, ramaddrs, blocklen, errx, cog ' rendezvous between spin and assembly (can be used cog to cog) ' command = R, W, S, M (read/write/set/move a block of sram) =0 when operation completed by cog ' hubaddrs = hub address for data buffer ' ramaddrs = ram address for data ($0_0000...$7_FFFF for 512KB total) ' blocklen = ram buffer length for data transfer ' errx = returns =0 (false=good), else <>0 (true & error code) ' cog = cog no of driver (set by spin start routine) PUB start : err_ ' Initialise the RamBlade driver command := "I" cog := 1 + cognew(@rb_start, @command) if cog == 0 err_ := $FF ' error = no cog else repeat while command ' driver cog sets =0 when done err_ := errx ' driver cog sets =0 if no error, else xx = error code PUB stop if cog cogstop(cog~ - 1) PUB idle : err_ ' Place the bus in the idle state (for SD driver use) [not reqd for RamBlade] err_ := 0 PUB active : err_ ' Activate the bus (from SD driver use) [not reqd for RamBlade] err_ := 0 PUB DoCmd(command_, hub_address, ram_address, block_length) : err_ ' Do the command: R, W, S, M (read/write/set/move a block of sram) hubaddrs := hub_address ' hub address start ramaddrs := ram_address ' ram address start blocklen := block_length ' block length command := command_ ' must be last !! ' Wait for command to complete and get status repeat while command ' driver cog sets =0 when done err_ := errx ' driver cog sets =0 if no error, else xx = error code DAT '' +--------------------------------------------------------------------------+ '' | Cluso's RamBlade3 Cog Driver | '' +--------------------------------------------------------------------------+ org 0 rb_start ' setup the pointers to the hub command interface (saves execution time later) ' +-- These instructions are overwritten as variables after start comptr mov comptr, par ' -| hub pointer to command hubptr mov hubptr, par ' | hub pointer to hub address ramptr add hubptr, #4 ' | hub pointer to ram address lenptr mov ramptr, par ' | hub pointer to length errptr add ramptr, #8 ' | hub pointer to error status cmd mov lenptr, par ' | command R/W/S/i/a hubaddr add lenptr, #12 ' | hub address ramaddr mov errptr, par ' | ram address len add errptr, #16 ' | length err nop ' -+ error status returned (=0=false=good) ' Initialise hardware [not reqd for RamBlade or RamBlade3???] init mov err, #0 ' reset err=false=good mov dira, #0 ' all inputs ' set done & status done wrlong err, errptr ' status =0=false=good, else error x wrlong zero, comptr ' command =0 (done) ' wait for a command pause rdlong cmd, comptr wz ' command ? if_z jmp #pause ' not yet ' decode command cmp cmd, #"R" wz ' R = read block if_z jmp #rdblock cmp cmd, #"W" wz ' W = write block if_z jmp #wrblock cmp cmd, #"S" wz ' S = set (uses first hub byte as value to set) if_z jmp #setblock ' cmp cmd, #"M" wz ' M = move (a ram block) ' if_z jmp #??? mov err, cmd ' error = cmd (unknown command) jmp #done '---------------------------------------------------------------------------------------------------------I agree with hindsight that all this can now be done in 4 longs as follows...
Long+0: perhaps divided into 4 bytes...
+0: The command
+1: Status returned (if required)
+2: Some flags
+3: Set by the cog to identify the type loaded (akin to Ross's plugin type see catalina/target/catalina_common.spin)
Long+1:
First parameter or pointer (typically hub address/pointer)
Long+2:
Second parameter or pointer (optional)
Long+3:
Third parameter or pointer (typically length)
I propose the commands be as follows, with other to be added as required...
"I": Initialise (initialise first time after loading)
"Q": Quick reinitialise (typically for a new program re-accessing a resident driver)
"R": read a char/block/length
"W": write a char/block/length
"S": status enquiry
"B": warm (soft) boot (leaves driver cogs loaded)
"X": cold boot (reboot cog 0 from eeprom)
$00: driver ready and waiting
Others as required by drivers.
I propose the status be as follows...
$00: status good
$80-FF: (negative) = errors
Flags defined by driver type
Driver type as per Catalina 3.6 definition (copy inserted below) plus...
_STDIN = tbd (standard input character device)
_STDOUT = tbd (standard output character device)
_STDIO = tbd (standard input and output character device)
_AUXIN = tbd (auxiliary input character device)
_AUXOUT = tbd (auxiliary output character device)
_AUXIO = tbd (auxiliary input and output character device)
I defined standard and auxiliary because I would like to be able to substitute the drivers on the fly for these. Also, I would like to be able to input and/or output to more than one device without changing the user code.
One thought - the "command" long typically is one letter which really means 3/4 of the long is wasted. You could use the rest of the long to pass and return binary flags or other data if you wanted.
I see this working by loading a tiny stub pasm code - it would be a little slower but drivers are not loaded frequently enough to be a problem, and it does not preclude the standard hub way anyway. SO I am trying to have the maximum hub free while getting the benefits of cogs remaining operational between program execution.
I already can go from my OS variant to ZiCog/CPM and back to the OS. Next is to use the std in & out drivers instead of the drivers we currently have in ZICog, followed by the FAT SD drivers. This will mean that we can release a single ZiCog binary for various devices. Once we are here, we can use all the CPM programs such as editors, spreadsheets, etc and transfer the files between CPM file structure and the FAT structure. So we can edit a propeller program in CPM using say wordstar or vedit, and then transfer the file back to FAT, return to the prop os and compile the prop program. Voila... no PC
Hi Cluso,
As I have said elsewhere ... good luck on getting this particular herd of cats to ever agree on anything. Mostly, I think they just like to argue
But (as I have also said elsewhere) ... if you do manage to get a consensus on a suitable mechanism that does everything Catalina needs, then Catalina will adopt it. I don't think there is anything particularly "special" about the existing mechanism Catalina uses - other than that it is the only one that has ever been fully implemented.
However, I should clarify a few things ...
Catalina uses Spin to launch objects only because Spin is the native high-level language of the Propeller v1. But Spin is not "inherent" in the proposal for standardizing communications between cogs. A registry can just as easily be set up and manipulated from any language, and on the Propeller v2 it is quite likely to be set up and manipulated without using Spin at all - yet I also expect Catalina on the Propeller v2 will maintain exactly the same registry structure as Catalina on the Propeller v1 - because this structure makes sense given the Propeller's fundamental hardware architecture and model of shared memory interaction.
Also, I wasn't aware I was "avoiding" anything ... as far as I was aware there is nothing that can't be easily addressed by the approach (if not the precise detail) of the solution implemented in Catalina. For instance, I saw a post earlier in this thread from JohnyMac about needing "raw speed" - but there is nothing preventing this. All the proposed solution asks (via Layers 0 and 1) is that you register your plugins (so that others have a chance of actually finding them if they need to, and do not simply assume they can overwrite the cogs they are running in) and also that you use a common mechanism for allocating Hub RAM (so that others will not tromp all over your memory buffers). Pretty much all plugins need to do these very basic things if they want to be able to be used in a "plug-and-play" fashion - but other than these bare necessities there is nothing to prevent plugins from otherwise operating (or co-operating) at full speed. Catalina's mechanism does not dictate a particular interaction model, and in fact Catalina uses several different ones itself. It certainly proposes a standard mechanism for cogs interacting with other cogs that provide well-defined services (this is layers 2 and 3) - but it also explicitly allows other models of interaction.
As for standardizing access to the SD Card - once you have the above infrastructure in place, this is trivial - all you really need is are simple "initialize", "read sector" and "write sector" services. Yes, you can get fancier if you want to - but beyond a certain point all you end up doing is making your solution less portable and less useful, not more.
Ross.
BTW I looked at 3.6 and saw you seem to be using an older version of Kye's driver and you have made some fixes to bugs. Do you know if Kye has fixed these in his latest code?
Hopefully when I have completed this task I can give you the new SD code to improve Catalina too.
I agree that I don't want to make the interface too restrictive. But I would like to offer some proposals for the extra services and commands to be preferred.
I'm not sure what version Kye's driver is up to. I posted him my bug fixes on the version I use a long time ago, but I don't know if he ever incorporated them. Most likely he found and fixed the same bugs himself in other ways.
When I get time, I'll do a "refresh" of the version of his code I use. But there is really no hurry - Catalina only uses Kye's drivers as part of the Catalyst SD card loader. Catalina itself uses DOSFS, which is a completely different beast.
Ross.
I keep a version history in the file... Compare the dates of last edit.
They were not barbs. They were observations. If you produce something we can use, we will include it in part or in whole with credit. As of now we don't need it.
Propeller-GCC also uses DOSFS for run-time SD card file-system support. It might be interesting if there was a single PASM COG module that could unburden some the intermediate-level read/write I/O, but there is already a COG that does the physical layer access.
Added: I noticed in another thread that you mentioned making a nibble mode SD card driver. Propeller GCC could easily adopt the PASM driver for it if you make it. The Spinneret board has SD compatible connections if you want a quick test platform.
There have been so many negative comments from the GCC team in various threads that personally I am convinced they perceive themselves above others. I am not the only one with this perception. And I am sorry, but I perceive you as heading it. (venting now off my chest)
You say GCC will use something if it works, but no info on what the GCC is doing. It may be a better solution, but you are silent.
I don't know anything about DOSFS so I will look it up. I am trying to strip the pasm drivers from the coupled spin callers. Whether the spin or GCC is 1 or 2 tiers is immaterial to my immediate task. I am trying tosplit the bottom layer - just the raw sector driver to spi code, and make a semi-standard interface for it.
As for nibble mode, thanks. I have plenty of small modules here to do the work. The interesting thing to note is that the semi standard way we hook up the SD pins to the prop now (IIRC DI,CLK,DO,-CS) is all wrong for nibble mode which must be (D0,D1,D2,D3,CMD,CLK where CMD & CLK can be reversed).
The Spinneret SD card has all the proper connections, and many people have one already, so many can benefit. Think you can squeeze all the SD commands into a COG to deliver a generic access interface?
Hi Kye,
Thanks for that. Once I fixed those bugs in the version I had, I never found a need to move to a later version (Catalyst doesn't need much except to be able to read sectors) - but I'll check out the latest version for the next release of Catalina.
Ross.
DOSFS info here. Catalina used it, and GCC then also adopted it. It is free, widely used, written in C, and makes only a trivial demands on an underlying SD card driver - i.e. to be able to read & write single sectors. These functions are simple enough to implement in a single cog, but DOSFS is not - it takes quite a few kilobytes. Just be aware the released version has a few problems - there are various fixes available in various places on the internet. As far as I know, Catalina currently implements them all.
Ross.