Upcoming changes to FlexProp file systems
ersmith
Posts: 6,068
I've slowly over the past few months been re-working how file systems work in FlexC/FlexSpin, to try to separate the file system logic from the drive I/O logic. It's starting to come to life, so I hope to be able to post a beta version "soon". The major changes are:
(1) File systems will now do their I/O via a file handle (struct vfs_file_t *) rather than directly to a hard-coded device. This opens up a lot of interesting possibilities, such as being able to mount file images (loopback file systems) or use file systems on devices they weren't originally designed for (e.g. FatFS on flash, or littlefs on SD).
(2) As a prerequisite for this, the off_t data type is now 64 bits, so we can seek to 64 bit offsets within files, i.e. FlexC is not longer restricted to a maximum file size of 2GB. In practice this isn't a big deal, but eventually might be handy for video players and such.
(3) Reading raw data from disks should be easier, since there will be a way to open the disk as a file (this is still TBD).
(4) The existing file system calls should still work, they'll open the default device for the file system (SD card, flash, or whatever) and then use that as the base for the file system.
@evanh I hope this won't cause you too much grief with your SD card work. I wish I had done this years ago so that you could have implemented the new SD driver as just a plain device. It's certainly overdue 
Comments
Cool...I've had the same thought for awhile - an SD card with, say, a bunch of floppy images that could be used in an emulator, etc.
Will the suggested way of using it in spin still be to use the libc object? It sounds like from #4 yes; just curious - I just tried it for the first time a couple weeks ago and it was pretty painless to get it all working.
The obvious concern when making things more complicated: Did you make sure the RAM usage and performance didn't regress?
Sounds useful. If this gets documented maybe we can put filesystems in PSRAM too using my mailbox driver and share this memory with other COGs (I vaguely recall you wanting to do that but ran into problems).
Sounds good to me. I just blandly copied the whole fatfs and only edited what I had to to add my changes. I've barely learnt a thing about how it currently all works and I'm not in any rush to look any further than I have to.
Yes, I don't plan to port all of this to spin, so using the libc object will be the best choice.
I haven't done timings yet, I'm still working on getting Evan's SD card code ported over (the initial FatFs target was a host file disk image
). I don't expect performance to regress too much -- there's an extra layer of code, but it's thin and the I/O will usually be the bottleneck. Similarly for RAM usage: in the default case the code won't have to allocate buffers, otherwise it might need an extra 512 bytes on the stack. But there is a bit more code, so we'll have to see how much binaries grow.
Eric,
A feature that is available with Roger's 8-pin SD socket is it can detect both card insertion and removal independently of operation. The power control pin, aside from the power switch control function, has a weak 22k pull-up that will go high when no card present. And when a card is inserted the same pin has a 1800 ohm pull-down, turning on the card power, that clearly identifies when a card has been inserted. Software can override the resistors by driving the pin.
This card detect ability could provide for auto mounting in the future. And one option is an event could be setup to latch the state change in hardware without needing to employ interrupts ... Although, this event setup could easily be trashed by other uses of the user program. But that would be true of any configured IRQs too. I've had to be careful to ensure other uses can't mess up the internal events in the new driver.
One question I have for Eric is will this new filesystem driver structure have the ability to read/write data atomically between COGs so an SD card could potentially be a shared storage device for multiple COGs running SPIN2 or C routines? Perhaps via the use of a lock? Ideally the taking/releasing of such locks might need to be co-ordinated with the filesystem design itself to maintain its integrity with the possibility of multiple writers being active, but in its simplest form could at least help prevent one COG from trashing the pin state of another COG's current pin state when an individual sector is being read or written. Or maybe some of this was already being done before now, I don't know.
I'm not sure auto-mounting is something we'd really want, although auto-detection definitely is. But that would be an ioctl on the SD card device, so it's kind of orthogonal to the file system design.
'
Both the file system and the underlying device driver would have to support multiple COGs. I don't think the current file systems are re-entrant, so this probably isn't feasible in the short term.
Pity. I wonder if we could one day get the filesystem & driver to point where a disk could be fully shared by all COGs, or, if not, then enable a multiple readers + single writer model, or even just one COG at a time, if that's simpler? I do realize this is the sort of thing an OS would normally provide and we don't exactly have one on the P2, but there may be some ways and means something could be done with some sort of co-operative yielding. If the design allowed an optional plug in hook to intercept these filesystem calls then maybe some sort of sharing/access scheduler layer could be developed later to try to support this kind of multi-COG support. It would track requests from each COG and divide up the resource. I imagine it'd get pretty complicated quickly though.
Should be relatively easy to use the filesystem stack as if it were a driver. Implement you own multi-cog shared buffer management scheme on top with a single cog managing files on the SD card.
With this SD mode driver, one thing I am doing on the compile line is setting the Fcache size to 256 registers. This is needed because I copy each completed data block into cogRAM to do the CRC processing while the streamer loads the subsequent data block to hubRAM.
I currently copy the data to cogRAM starting at address 126. The size of my Fcache'd block read handling code is currently 72 registers. So the 256 could be reduced to say 208 ish. I note this is still above the default though. So changing the default Fcache size might be in order to make it less obtuse for others.
In addition to larger Fcache needs, I'm also allocating 8 registers (
extern register uint32_t *) for parameter presetting. This is a speed thing. It makes it simple and fast to copy them in one hit. And it solved a compile warning too, where specifying register numbers numerically wasn't ideal.>
Ah. That's going to be a problem for the bytecode modes, where there just isn't that much COG memory available
. I guess we could have two SD drivers, one for each mode.
Is lutRAM free when bytecoding? I could copy the data block there. It'd be a smidgen slower processing the CRC but not significant.
No, the interpreter uses the LUT (not all of it, possibly, but it depends on the program -- the bytecode is dynamically generated based on the instructions used). The documentation for the available memory definitely needs improving, it's very much centered on the default (ASM) backend. Having said that, it probably is reasonable for you to just go ahead and write with the ASM backend in mind, and we can always fall back to the old driver for bytecode. Everyone should expect a performance hit when using bytecode anyway.
I'm good with that.