On-Board Flash File System
cgracey
Posts: 14,133
Finally at about v1.0.
This uses the on-board W25Q128 flash chip that is (almost) always connected to the P2. It turns the unused 15.5MB (after the initial 512KB boot area) into a solid-state drive for up to almost 4,000 files.
NEW V1.1 with 2.5x faster SPI and CS-high-on-done:
Comments
Nice. Glad to see it has wear leveling.
The clocking looks odd. It'll need a touch-up, for idle high, to share the pins with an SD Card.
Here's a version modified (slightly) to work in flexspin -- I had to initialize a variable, and change another variable from local to global to avoid a conflict with inline assembly. To compile this with flexspin you'll need the absolute latest flexspin (i.e. built from github) that has the GETCRC() builtin implemented.
Neat! I'd been working on a FAT flash file system, but had not yet figured out a way to solve the problem of wear on the FAT. If I understand it correctly, this solution avoids that problem altogether (no FAT!). I may revisit what I've got so far and see if I can incorporate it.
I wrote a really simplistic shell type program to interactively work with the driver. I am having trouble with format -- it seems not to clear away the files with latest driver (even running Chip's demo). Maybe I've done something wrong with the way I wrote my extra example files.
Updated: 15 SEP with version 1.3.0. See post #63
Jon, I made a mistake on Format() when I was refining the handling of the %vvv lifecycle bits. Should be like this:
PUB Format() | i, first '' Format file system and (re)mount it repeat i from 0 to BLOCKS - 1 'cancel all active blocks ReadBlock(i, @first, $000, $000) if lookdown(first.[7..5] : %011, %101, %110) CancelBlock(i) Mount() '(re)mount flashif first.[7]
...needed to be...
if lookdown(first.[7..5] : %011, %101, %110)
I will update the file in the top post.
This is really great. Although I haven't got the need of a real filesystem so far I can use it for my servo project to store configuration data in a secure (make before break) way. So far, I have written the data to a fixed location since wear is not so much of an issue as configuration doesn't change very often. But now I can also implement error logging and maybe even a safe firmware loader/updater without much trouble.
As we discussed yesterday in the live forum ("quasi"-) subdirectories could be implemented by simply adding slashes to the filenames. This would mean that the full pathnames would be limited to 60 characters which is enough for most cases.
To allow for future updates without braking existing data stored in the flash I'd like to reserve at least two bits in a file header block:
(Sub-) directories could be implemented in two ways:
1. Storing directories as special files which have the DIR bit set. They contain a list of files as array of IDs of the file headers.
2. Storing directories as empty file headers with only the name. Each file header contains a pointer (ID) of the parent directory.
1 requires the directory to be re-written each time a file is added or removed, 2 doesn't. But 2 requires more space in each file header.
Seems simple and efficient.
However this:
PRI FlashReceive(BuffAddress, ByteCount) | Bits, Data org wrfast #0,BuffAddress 'start fast write .byte mov Bits,#8 'ready for a byte .bit drvl #SPI_CK '2! CK low waitx #6 '2+6 delay drvh #SPI_CK '2! CK high testp #SPI_DO wc '2 sample DO input rcl Data,#1 '2 save received bit djnz Bits,#.bit '4 loop if more bits to receive wfbyte Data 'write byte djnz ByteCount,#.byte 'loop if another byte to receive drvl #SPI_CK '2! CK low drvl #SPI_DI '2! DI low drvh #SPI_CS '2! CS high terminate command endseems to end with the flash clock signal set low. This will harm the SD card inserted, sooner or later. The driver needs to keep the clock high when idle.
Chip,
I've just tried to LOC (Pasm instruction) a VAR. It doesn't work but could Spin2 be made to work with this? eg:
movbyts Command,#%%1230 '2 reverse the three address bytes for sending loc pb, #@CommandBuf '2 wrlong Command, pb '9+ CS deselect is 50ns at 360 MHz (50ns / 2.777ns = 18 ticks)@cgracey Great job! Thank you for this!
--Terry
Here's the minimal clocking changes to support idle-high (CPOL=1) for SD Card sharing
And this is what I'd normally do at the low level
This should be very handy Chip. Nice work.
A future path to gain support for sub-directories will be good too, if that is planned for down the track. Listing ~4000 files in a single folder will get old quickly once the storage really builds up, and having folders will allow different P2 run time applications to share this common resource better, if their filenames overlapped for example.
I initially wondered if this could be useful for a filesystem on the HyperFlash board going through my memory driver at the lowest level, but the native sector size for that device is much larger at 256k which is not ideal and would probably need some custom erase changes which probably doesn't make sense. Also all P2 systems running with SPI boot flash already have this resource present and available.
I'm not a big fan of SD cards or FAT filesystems, and so this is a great feature for the EC module, IMO!
I'll also chime in on the prospect of sub-directories. Personally, I'd rather modify this to allow "tagging" support. If you want to group files together logically (as in a sub-directory), then simply give them the same tag value, such as "#configuration" for the equivalent of a directory of config files. It's rather easy to write/maintain all the tagging information within a single meta-data file. IIRC, something like this was even discussed back when @cgracey began this effort.
Yes, tags are a departure from convention, but they really open things up.
I certainly agree. I implemented a tagging system for asset management in a game engine about 15 years ago. At the time, I got a lot of push-back from other developers on the project, but we found (in time) that the vastly more flexible grouping of files possible was worth the overhead of maintaining the metadata.
However, nowadays, since social media is chock full of hash tag this and that, (maybe??) it should be viewed as "mainstream".
You would have to pre-place that @CommandBuf value into a local variable so that it could then be used in your in-line PASM code. The compiler doesn't know where that address will be at runtime.
I don't really understand how a tagging system is anything but vastly inferior to nested directories in utility and efficiency.
I can live with a single "directory" for the time being. What Chip's FFS needs right away is open_append() and seek() -- which likely share the seek() mechanism so it's a 2-for-1.
It may not be, but in my own mind it's a grass-is-greener panacea.
Thinking about how to do this and, in the process, generalize block-replacement.
Cool. I don't know if you tried my simple interactive demo, but at the top is a list of thoughts and suggestions. Others will have different ideas, and it's worth discussing them now.
{ Demo program using Chip Gracey's initial flash file system. Suggested additions/mods to CG FFS API (* = exists) * format() * mount() * modify open(p_filename, mode) mode = "r" | "w" | "a" new seek(handle, position) new read(handle, p_buf, count) * change name read_byte(handle) new read_word(handle) new read_long(handle) new read_str(handle, p_dest, maxlen) new write(handle, p_buf, count) * change name write_byte(handle, value) new write_word(handle, value) new write_long(handle, value) new write_block(handle, p_src) new write_str(handle, p_src) * close(handle) * stats() * directory() * exists(p_filename) new is_open(p_filename) new file_handle(p_filename) * change name get_filesize(p_filename) new get_filestatus(handle) new get_filecount() * rename(p_oldname, p_newname) * delete(p_filename) Thoughts -- should mount() set a flag that lets other code do file operations? -- abandon abort traps for return values; use negative values for error codes }I guess write_block will need a corresponding read_block.
Another thing would be modify, so not appending to a file but seek a position and (over-)write something, while not changing the rest of the file.
And YES, please change the flash access as @evanh suggested to Mode 3(?) to allow SD access when using the flash file system.
Mike
The thing about folders/sub-directories, is that it is already a well established concept in computing. A hell of a lot of code is already written to support/expect that. So if you are working with some C code or Python or whatever, files and paths are commonly used. If we ever wanted to port that code to run on the P2 it would be simplest to retain the capability of using filepaths. Now whether or not and how that gets translated internally to tags at a lower storage level is a different story, but IMO it would be good to preserve the concept of nested folders from some top level root level. Of course people can start out with just a single level to begin with but it would be good to have a path (pun intended) to allow sub-directories too in the future.
Tbf I would consider directories an optional feature for the flash filesystem as is (limited to 15.5MB and all). Just files is fine.
If you really want folders and not doing a lot of writing (mostly just reading), then it is possible to put a FAT volume on Flash.
But, this is probably better for many applications.
It was kind of a waste to have that 15.5 MB and not being able to easily use it...
Might take some work to make this as robust as the LittleFS from FlexProp.
I haven't tried that yet either, but has a good pedigree.
Would be great if this and LittleFS were compatible, but I'm guessing that is not the case.
The LOC can be relative encoded though. Isn't everything in the object fixed relatively?
Doh! VARs are not in the object are they. They're instantiated.
PS: No biggie, I had already worked around it by throwing away the hubRAM storage entirely.
From my minor messing around it doesn't seem to be particularly good. Adds huge bloat to your application (much larger than the default FAT driver! (without LFN/exfat/etc enabled))
In a limited FS like this directories may be simply tagged as a number in a file header/name, and then described in a directories description file