Shop OBEX P1 Docs P2 Docs Learn Events
On-Board Flash File System — Parallax Forums

On-Board Flash File System

cgraceycgracey Posts: 14,151
edited 2023-08-18 21:00 in Propeller 2

Finally at about v1.0.

This uses the on-board W25Q128 flash chip that is (almost) always connected to the P2. It turns the unused 15.5MB (after the initial 512KB boot area) into a solid-state drive for up to almost 4,000 files.

NEW V1.1 with 2.5x faster SPI and CS-high-on-done:

«134

Comments

  • RaymanRayman Posts: 14,632

    Nice. Glad to see it has wear leveling.

  • evanhevanh Posts: 15,910

    The clocking looks odd. It'll need a touch-up, for idle high, to share the pins with an SD Card.

  • Here's a version modified (slightly) to work in flexspin -- I had to initialize a variable, and change another variable from local to global to avoid a conflict with inline assembly. To compile this with flexspin you'll need the absolute latest flexspin (i.e. built from github) that has the GETCRC() builtin implemented.

  • RossHRossH Posts: 5,461

    Neat! I'd been working on a FAT flash file system, but had not yet figured out a way to solve the problem of wear on the FAT. If I understand it correctly, this solution avoids that problem altogether (no FAT!). I may revisit what I've got so far and see if I can incorporate it.

  • JonnyMacJonnyMac Posts: 9,102
    edited 2023-09-15 14:19

    I wrote a really simplistic shell type program to interactively work with the driver. I am having trouble with format -- it seems not to clear away the files with latest driver (even running Chip's demo). Maybe I've done something wrong with the way I wrote my extra example files.

    Updated: 15 SEP with version 1.3.0. See post #63

  • cgraceycgracey Posts: 14,151
    edited 2023-08-17 03:34

    @JonnyMac said:
    I wrote a really simplistic shell type program to interactively work with the driver. I am having trouble with format -- it seems not to clear away the files with latest driver (even running Chip's demo). Maybe I've done something wrong with the way I wrote my extra example files.

    Jon, I made a mistake on Format() when I was refining the handling of the %vvv lifecycle bits. Should be like this:

    PUB Format() | i, first
    
    '' Format file system and (re)mount it
    
      repeat i from 0 to BLOCKS - 1             'cancel all active blocks
        ReadBlock(i, @first, $000, $000)
        if lookdown(first.[7..5] : %011, %101, %110)
          CancelBlock(i)
    
      Mount()                       '(re)mount flash
    

    if first.[7]

    ...needed to be...

    if lookdown(first.[7..5] : %011, %101, %110)

    I will update the file in the top post.

  • @cgracey said:
    This uses the on-board W25Q128 flash chip that is (almost) always connected to the P2. It turns the unused 15.5MB (after the initial 512KB boot area) into a solid-state drive for up to almost 4,000 files.

    This is really great. Although I haven't got the need of a real filesystem so far I can use it for my servo project to store configuration data in a secure (make before break) way. So far, I have written the data to a fixed location since wear is not so much of an issue as configuration doesn't change very often. But now I can also implement error logging and maybe even a safe firmware loader/updater without much trouble.

  • As we discussed yesterday in the live forum ("quasi"-) subdirectories could be implemented by simply adding slashes to the filenames. This would mean that the full pathnames would be limited to 60 characters which is enough for most cases.

    To allow for future updates without braking existing data stored in the flash I'd like to reserve at least two bits in a file header block:

    • 00 means use the simple format as it's defined now
    • 01 means this is a directory, not a file
    • 10 means use an extended format, say longer filenames
    • 11 is reserved for future extensions

    (Sub-) directories could be implemented in two ways:
    1. Storing directories as special files which have the DIR bit set. They contain a list of files as array of IDs of the file headers.
    2. Storing directories as empty file headers with only the name. Each file header contains a pointer (ID) of the parent directory.

    1 requires the directory to be re-written each time a file is added or removed, 2 doesn't. But 2 requires more space in each file header.

  • pik33pik33 Posts: 2,366

    Seems simple and efficient.

    However this:

    PRI FlashReceive(BuffAddress, ByteCount) | Bits, Data
    
        org
    
        wrfast  #0,BuffAddress      'start fast write
    
    .byte   mov Bits,#8         'ready for a byte
    
    .bit    drvl    #SPI_CK         '2! CK low
        waitx   #6          '2+6    delay
        drvh    #SPI_CK         '2! CK high
        testp   #SPI_DO     wc  '2  sample DO input
        rcl Data,#1         '2  save received bit
        djnz    Bits,#.bit      '4  loop if more bits to receive
    
        wfbyte  Data            'write byte
        djnz    ByteCount,#.byte    'loop if another byte to receive
    
        drvl    #SPI_CK         '2! CK low
        drvl    #SPI_DI         '2! DI low
        drvh    #SPI_CS         '2! CS high     terminate command
    
        end
    

    seems to end with the flash clock signal set low. This will harm the SD card inserted, sooner or later. The driver needs to keep the clock high when idle.

  • evanhevanh Posts: 15,910
    edited 2023-08-17 09:00

    Chip,
    I've just tried to LOC (Pasm instruction) a VAR. It doesn't work but could Spin2 be made to work with this? eg:

        movbyts Command,#%%1230     '2  reverse the three address bytes for sending
        loc     pb, #@CommandBuf    '2
        wrlong  Command, pb     '9+ CS deselect is 50ns at 360 MHz (50ns / 2.777ns = 18 ticks)
    
  • @cgracey Great job! Thank you for this!

    --Terry

  • evanhevanh Posts: 15,910

    Here's the minimal clocking changes to support idle-high (CPOL=1) for SD Card sharing

  • evanhevanh Posts: 15,910

    And this is what I'd normally do at the low level

  • roglohrogloh Posts: 5,786
    edited 2023-08-18 07:22

    This should be very handy Chip. Nice work.

    A future path to gain support for sub-directories will be good too, if that is planned for down the track. Listing ~4000 files in a single folder will get old quickly once the storage really builds up, and having folders will allow different P2 run time applications to share this common resource better, if their filenames overlapped for example.

    I initially wondered if this could be useful for a filesystem on the HyperFlash board going through my memory driver at the lowest level, but the native sector size for that device is much larger at 256k which is not ideal and would probably need some custom erase changes which probably doesn't make sense. Also all P2 systems running with SPI boot flash already have this resource present and available.

  • brianhbrianh Posts: 22
    edited 2023-08-17 16:59

    I'm not a big fan of SD cards or FAT filesystems, and so this is a great feature for the EC module, IMO!

    I'll also chime in on the prospect of sub-directories. Personally, I'd rather modify this to allow "tagging" support. If you want to group files together logically (as in a sub-directory), then simply give them the same tag value, such as "#configuration" for the equivalent of a directory of config files. It's rather easy to write/maintain all the tagging information within a single meta-data file. IIRC, something like this was even discussed back when @cgracey began this effort.

  • cgraceycgracey Posts: 14,151

    @brianh said:
    I'm not a big fan of SD cards or FAT filesystems, and so this is a great feature for the EC module, IMO!

    I'll also chime in on the prospect of sub-directories. Personally, I'd rather modify this to allow "tagging" support. If you want to group files together logically (as in a sub-directory), then simply give them the same tag value, such as "@configuration" for the equivalent of a directory of config files. It's rather easy to write/maintain all the tagging information within a single meta-data file. IIRC, something like this was even discussed back when @cgracey began this effort.

    Yes, tags are a departure from convention, but they really open things up.

  • @cgracey said:

    Yes, tags are a departure from convention, but they really open things up.

    I certainly agree. I implemented a tagging system for asset management in a game engine about 15 years ago. At the time, I got a lot of push-back from other developers on the project, but we found (in time) that the vastly more flexible grouping of files possible was worth the overhead of maintaining the metadata.

    However, nowadays, since social media is chock full of hash tag this and that, (maybe??) it should be viewed as "mainstream".

  • cgraceycgracey Posts: 14,151

    @evanh said:
    Chip,
    I've just tried to LOC (Pasm instruction) a VAR. It doesn't work but could Spin2 be made to work with this? eg:

      movbyts Command,#%%1230     '2  reverse the three address bytes for sending
      loc     pb, #@CommandBuf    '2
      wrlong  Command, pb     '9+ CS deselect is 50ns at 360 MHz (50ns / 2.777ns = 18 ticks)
    

    You would have to pre-place that @CommandBuf value into a local variable so that it could then be used in your in-line PASM code. The compiler doesn't know where that address will be at runtime.

  • @cgracey said:
    Yes, tags are a departure from convention, but they really open things up.

    I don't really understand how a tagging system is anything but vastly inferior to nested directories in utility and efficiency.

  • I can live with a single "directory" for the time being. What Chip's FFS needs right away is open_append() and seek() -- which likely share the seek() mechanism so it's a 2-for-1.

  • cgraceycgracey Posts: 14,151
    edited 2023-08-17 17:57

    @Wuerfel_21 said:

    @cgracey said:
    Yes, tags are a departure from convention, but they really open things up.

    I don't really understand how a tagging system is anything but vastly inferior to nested directories in utility and efficiency.

    It may not be, but in my own mind it's a grass-is-greener panacea.

  • cgraceycgracey Posts: 14,151

    @JonnyMac said:
    I can live with a single "directory" for the time being. What Chip's FFS needs right away is open_append() and seek() -- which likely share the seek() mechanism so it's a 2-for-1.

    Thinking about how to do this and, in the process, generalize block-replacement.

  • Cool. I don't know if you tried my simple interactive demo, but at the top is a list of thoughts and suggestions. Others will have different ideas, and it's worth discussing them now.

    {
       Demo program using Chip Gracey's initial flash file system.
    
       Suggested additions/mods to CG FFS API (* = exists)
    
       *               format()
       *               mount()
       * modify        open(p_filename, mode)  mode = "r" | "w" | "a"
       new             seek(handle, position)
       new             read(handle, p_buf, count)
       * change name   read_byte(handle)
       new             read_word(handle)
       new             read_long(handle)
       new             read_str(handle, p_dest, maxlen)
       new             write(handle, p_buf, count)
       * change name   write_byte(handle, value)
       new             write_word(handle, value)
       new             write_long(handle, value)
       new             write_block(handle, p_src)
       new             write_str(handle, p_src)
       *               close(handle)
       *               stats()
       *               directory()
       *               exists(p_filename)
       new             is_open(p_filename)
       new             file_handle(p_filename)
       * change name   get_filesize(p_filename)
       new             get_filestatus(handle)
       new             get_filecount()
       *               rename(p_oldname, p_newname)
       *               delete(p_filename)
    
    
       Thoughts
       -- should mount() set a flag that lets other code do file operations?
       -- abandon abort traps for return values; use negative values for error codes
    }
    
  • I guess write_block will need a corresponding read_block.

    Another thing would be modify, so not appending to a file but seek a position and (over-)write something, while not changing the rest of the file.

    And YES, please change the flash access as @evanh suggested to Mode 3(?) to allow SD access when using the flash file system.

    Mike

  • roglohrogloh Posts: 5,786
    edited 2023-08-18 01:12

    The thing about folders/sub-directories, is that it is already a well established concept in computing. A hell of a lot of code is already written to support/expect that. So if you are working with some C code or Python or whatever, files and paths are commonly used. If we ever wanted to port that code to run on the P2 it would be simplest to retain the capability of using filepaths. Now whether or not and how that gets translated internally to tags at a lower storage level is a different story, but IMO it would be good to preserve the concept of nested folders from some top level root level. Of course people can start out with just a single level to begin with but it would be good to have a path (pun intended) to allow sub-directories too in the future.

  • Tbf I would consider directories an optional feature for the flash filesystem as is (limited to 15.5MB and all). Just files is fine.

  • RaymanRayman Posts: 14,632
    edited 2023-08-17 23:15

    If you really want folders and not doing a lot of writing (mostly just reading), then it is possible to put a FAT volume on Flash.
    But, this is probably better for many applications.

    It was kind of a waste to have that 15.5 MB and not being able to easily use it...
    Might take some work to make this as robust as the LittleFS from FlexProp.
    I haven't tried that yet either, but has a good pedigree.

    Would be great if this and LittleFS were compatible, but I'm guessing that is not the case.

  • evanhevanh Posts: 15,910
    edited 2023-08-17 23:24

    @cgracey said:

    @evanh said:
    Chip,
    I've just tried to LOC (Pasm instruction) a VAR. It doesn't work but could Spin2 be made to work with this? eg:

        movbyts Command,#%%1230     '2  reverse the three address bytes for sending
        loc     pb, #@CommandBuf    '2
        wrlong  Command, pb     '9+ CS deselect is 50ns at 360 MHz (50ns / 2.777ns = 18 ticks)
    

    ... The compiler doesn't know where that address will be at runtime.

    The LOC can be relative encoded though. Isn't everything in the object fixed relatively?
    Doh! VARs are not in the object are they. They're instantiated.

    PS: No biggie, I had already worked around it by throwing away the hubRAM storage entirely.

  • @Rayman said:
    Might take some work to make this as robust as the LittleFS from FlexProp.
    I haven't tried that yet either, but has a good pedigree.

    From my minor messing around it doesn't seem to be particularly good. Adds huge bloat to your application (much larger than the default FAT driver! (without LFN/exfat/etc enabled))

  • pik33pik33 Posts: 2,366

    In a limited FS like this directories may be simply tagged as a number in a file header/name, and then described in a directories description file

Sign In or Register to comment.