Shop OBEX P1 Docs P2 Docs Learn Events
On-Board Flash File System - Page 3 — Parallax Forums

On-Board Flash File System

135

Comments

  • NEWS

    Just released a minor update. P2-FLASH-FS GitHub repository:

    v1.2.1 - seek() change and updated demo from Jon
    Features

    • Updated seek() in support of flexspin. Now supports seek +/- from current position as well as seek relative to file start.
    • Updated with latest interactive demo from Jon
    • Updated seek regression test to exercise the new position-relative seek
    • Updated other regression tests to the latest seek() signature where used

    Installation instructions are found on the top page of the Repo.

    This version should now work with uSD without interfering. We need users to certify that this is now in good shape.

    All methods are backed by a regression test suite. At each release, this one, and all upcoming releases, the test suite will be run before the release is posted so that we can be sure future updates will work when they are released. When a release is posted the updated regression test code along with the logs of the passing runs are also versioned.

    If you find something not working as you expect, then please file an issue at my GitHub repository issues page

    What's next?
    I'm currently working on validating the flash filesystem in a multi-cog environment. Once this testing is complete, I'll notify us all here in this thread.
    I'm also planning on full directory functionality to follow in an upcoming release.

    Enjoy!

    Stephen

  • JonnyMacJonnyMac Posts: 9,156
    edited 2023-10-20 00:44

    I have updated my demo to use the latest version (1.3.0) of Chip's and Stephen's flash filesystem driver. The demo is a simple shell program that understands basic DOS and LINUX commands (no wildcards), and a few others specific to the flash and demo.

      help       display this help                        
      version    display P2FFS version                         
      cls        clear terminal window                          
      clear      clear terminal window                           
      mount      mount flash filesystem                          
      format     format flash                                    
      stats      display P2FFS statistics                        
      dir        list files in P2FFS                             
      ls         list files in P2FFS                             
      type       display contests of file (type filename)        
      cat        display contests of file (cat filename)         
      ren        change file name (ren oldname newname)          
      mv         change file name (mv oldname newname)           
      copy       copy from src to dest (copy srcfile destfile)   
      cp         copy from src to dest (cp srcfile destfile)     
      del        delete file (del filename)                      
      rm         delete file (rm filename)                       
      demo       create demo files in FFS                        
      append     append demo string to text file                 
      snum       display flash serial number                     
      uid        display flash serial number 
    

    Tested with:

    • Propeller Tool
    • Spin Tools IDE
    • FlexProp 6.4.0 (check Options\Use internal PST terminal)

    Another Update: Since I need a command parser for a work project I continued fussing with this demo. After looking at Eric's shell demo (in C) made some changes:

    • command line and arguments are parsed in one go
    • renamed some methods for clarity
    • added details to stats, dir, ls
    • argument count from command line is enforced for everything
    • re-organized and made things prettier (in my eyes, of course)

    Yet Another Update!

    • extracted and improved parsing code to separate object
    • Minor bug-fixes and updates (e.g., you cannot copy a file to itself).

    New file (23 SEP 2023) tested with Propeller Tool, Spin Tools (0.31.0) and FlexProp (6.5.0)
    -- thanks to Eric Smith for finding a subtle requirement of Chip's code that caused me a snag with FlexProp

    25 SEP 2023 -- Fixed a couple small "gotchas" in the help text.

    18 OCT 2023 -- Added an append demo and removed the buffered read which caused confusion.

  • RaymanRayman Posts: 14,744
    edited 2023-09-15 18:57

    Anybody figured out how to reboot with a binary stored on flash file system yet?
    Is that even possible? I imagine it is but maybe one needs to reset all smartpins to have a clean start?
    Probably have to handle setting the P2 clock manually?

  • pik33pik33 Posts: 2,383
    edited 2023-09-15 20:31

    I have a binary loader in the Basic interpreter. It is easy with a PSRAM system. Eventually I will add the flash FS to the interpreter but maybe.. let it stabilize. I have enough problems with the interpreter itself.

    It is easy with a PSRAM because you can load all the binary to the PSRAM while the loader can still access all high level filesystem stuff

    In FlexBasic,

      open fullfilename for input as #9
      r=geterr() : if r then print "System error ";r;": ";strerror$(r) :close #9 : return
      let pos=1: let r=0 : let psramptr=0
      do
        get #9,pos,block(0),1024,r : pos+=r 
        psram.write(varptr(block(0)),psramptr,1024) 
        psramptr+=r                                     ' move the buffer to the RAM and update RAM position. Todo: this can be done all at once
      loop until r<>1024  orelse psramptr>=$7C000                           ' do until eof or memory full
      cpustop(audiocog)                                 ' stop all driver cogs except PSRAM
      cpustop(videocog)
      cpustop(usbcog)
      cpustop(housekeeper_cog)
      let loadingcog=cpu(@loadcog,@pslock)                          ' start loading cog
      cpustop(cpuid())
    

    The loadcog assembly looks like this:

    '' ------------------------------- the loader cog for BRUN
    
            asm shared
    
                    org
    loadcog         cogid   t11                     ' get a cogid
                    mul     t11, #12                        ' compute the offset to PSRAM mailbox 
                    add     mailbox, t11                     ' add offset to find this COG's mailbox
    
                    mov     psramaddr,#0
    
    p101            mov     buf1,psramaddr          ' psramaddr=hubaddr
                    mov     buf2,##16384            ' loading size
                    mov     cmd,psramaddr                   ' set the address for reading
                    setnib  cmd, #%1011, #7                 ' attach the command - read burst
                    setq    #2              ' write 3 longs to the mailbox
                    wrlong  cmd, mailbox            ' read the PSRAM
    p102            rdlong  cmd, mailbox                    ' poll mailbox for result
                    tjs     cmd, #p102                  ' retry until valid 
    
                    add     psramaddr,##16384
            cmp     psramaddr,##$7C000 wcz
        if_lt   jmp     #p101               ' loop until full hub loaded
    
    
                    cogstop #7              ' stop psram driver
    
                    cogid   t11             ' get id
                    coginit #0,#0               ' start the new program
                    cogstop t11             ' stop the loader
    
    t11         long    0
    mailbox     long    $7FF00
    psramaddr   long    0
    pslockval   long    0
    cmd             long    0
    buf1            long    0
    buf2            long    16384
    
            end asm
    
    '----- The end ---------------------------------------------------------------------------------------------------
    

    The PSRAM driver mailboxes has to be moved to the end of the hub ram to make this possible.


    Without the PSRAM, but with a flash filesystem.. i think it can be done too. The problem may be a fragmented file and non-file data in sectors. The loader program should be as short as possible, to fit at the very top of the RAM. It should determine what bytes and from where to load while the high level file system stuff is still running, then do the job and restart the P2.

    I don't care about the clock. It is already set for the coginit #0,#0 to run, and then the new binary will take the control and set the clock as it wants. I don't know if the stil running smartpins may collide with the new program. The loader code can however unprogram them before restarting. Things that I started with the loader: a multimedia player and *yume, worked without any problems with this loader.

  • FlexBASIC already has a CHAIN command which should run just fine with the Parallax Flash file system. To run the file "boot.run" from flash, do something like:

    mount "/f", _vfs_open_parallaxfs()
    chain "/f/boot.run"
    print "Chain failed, we should never get here. Last error was ";  strerror$(geterr())
    
  • pik33pik33 Posts: 2,383
    edited 2023-09-15 20:49

    @ersmith said:
    FlexBASIC already has a CHAIN command which should run just fine with the Parallax Flash file system. To run the file "boot.run" from flash, do something like:

    mount "/f", _vfs_open_parallaxfs()
    chain "/f/boot.run"
    print "Chain failed, we should never get here. Last error was ";  strerror$(geterr())
    

    Only if the new loaded binary is not too big. That's why I use PSRAM. The workaround can be simple: CHAIN a smallest possible intermediate loader, that will CHAIN a new big binary.

  • JonnyMacJonnyMac Posts: 9,156
    edited 2023-09-15 22:16

    [deleted]

  • Stephen MoracoStephen Moraco Posts: 321
    edited 2023-09-16 21:16

    NEWS

    More compatibility adjustments for FlexSpin. P2-FLASH-FS GitHub repository:

    v1.3.1 - Continued compatibility updates for FlexSpin and add new method
    Features

    • Adds new method: PUB file_size_for_handle(handle) : size_in_bytes
    • Seek() now correctly returns file length for seek(handle, POSX, SK_FILE_START)
    • Updated regression tests for testing new methods and verifying seek(), read(), write() returned values

    Installation instructions are found on the top page of the Repo. Full details for this release can be found at V1.3.1 Release Page

    This version should now work with uSD without interfering. We need users to certify that this is now in good shape.

    All methods are backed by a regression test suite. At each release, this one, and all upcoming releases, the test suite will be run before the release is posted so that we can be sure future updates will work when they are released. When a release is posted the updated regression test code along with the logs of the passing runs are also versioned.

    If you find something not working as you expect, then please file an issue at my GitHub repository issues page

    What's next?
    I'm currently working on validating the flash filesystem in a multi-cog environment. Once this testing is complete, I'll notify us all here in this thread.
    I'm also planning on full directory functionality to follow in an upcoming release.

    Enjoy!

    Stephen

  • Stephen MoracoStephen Moraco Posts: 321
    edited 2023-09-19 23:27

    NEWS

    Releasing multi-cog support. P2-FLASH-FS GitHub repository:

    v1.4.0 - Finish implementation needed to support multi-cog access
    Features

    • Driver now supports multi-cog access, using locks, return code per cog, and adjusted FlashChip signaling to leave signals in the proper state for each Cog use.

    NOTE: this driver does not use a Cog. It is called from any cog needing to access the onboard flash chip and runs in the calling cog.

    Installation instructions are found on the top page of the Repo. Full details for this release can be found at V1.4.0 Release Page

    This version should now work with uSD without interfering. We need users to certify that this is now in good shape.

    All methods are backed by a regression test suite. At each release, this one, and all upcoming releases, the test suite will be run before the release is posted so that we can be sure future updates will work when they are released. When a release is posted the updated regression test code along with the logs of the passing runs are also versioned.

    If you find something not working as you expect, then please file an issue at my GitHub repository issues page

    This release is done with the same compilers as earlier releases.
    However, It looks like there may be some issues with our latest compilers. I'll let you know what I find out.

    What's next?
    I'm also planning on full directory functionality to follow in an upcoming release.

    Enjoy!

    Stephen

  • evanhevanh Posts: 16,023

    Is this a fork or continuation or alternative of what Chip provided in the opening post?

  • JonnyMacJonnyMac Posts: 9,156
    edited 2023-09-20 14:41

    Best described as a continuation. Chip turned the basic working system over to Stephen for clean-up, documentation, and update of API (I asked for a couple of small features). I think that Chip is still working on new features in his original code and passing them to Stephon for incorporation to the "nice" version.

  • evanhevanh Posts: 16,023

    Thanks. I've been hamstering elsewhere and, checking in, got a little confused with the increased size of Stephen's code.

  • Is the following assumption valid? If I overwrite a file that is shorter than one block (4k-headersize = 3k+something bytes) and a power failure happens I can be sure that the file contains either the old data or the new (but is never empty or contains garbage or incomplete data).

  • AribaAriba Posts: 2,690

    @ManAtWork said:
    Is the following assumption valid? If I overwrite a file that is shorter than one block (4k-headersize = 3k+something bytes) and a power failure happens I can be sure that the file contains either the old data or the new (but is never empty or contains garbage or incomplete data).

    Yes, the old data was not deleted or overwritten. If the new written block is not valid, then the old block is activated again at next mount(). So you will get the old data. That's not only true for short files with one block, also bigger files will behave like that.

    Andy

  • @Ariba said:

    @ManAtWork said:
    Is the following assumption valid? If I overwrite a file that is shorter than one block (4k-headersize = 3k+something bytes) and a power failure happens I can be sure that the file contains either the old data or the new (but is never empty or contains garbage or incomplete data).

    Yes, the old data was not deleted or overwritten. If the new written block is not valid, then the old block is activated again at next mount(). So you will get the old data.

    Very good! This simplifies the storage of user settings and configuration data. No need for an extra backup.

    That's not only true for short files with one block, also bigger files will behave like that.

    Hmm, I doubt that. AFAIK, only one block is buffered. So if I write multiple blocks and the power failure happens between writing the 1st and the 2nd block then I assume the old file is already deleteted and only the first block of the new file is valid. Or is the old file really preserved until I close the new file?

  • Hmm, I doubt that. AFAIK, only one block is buffered. So if I write multiple blocks and the power failure happens between writing the 1st and the 2nd block then I assume the old file is already deleteted and only the first block of the new file is valid. Or is the old file really preserved until I close the new file?

    Since writing the new first block will point to the old second block, I think you'll be okay -- other than having old data in the second block. No?

  • Ok, this means the file is overwritten block by block but the length stays the same unless I write past the old end. This is possibly the best the filesystem can do to avoid loss of data. But we still need to be careful. For example if text data is written and it is not always equally formatted (line length, number of digits...) lines passing block boundaries could be garbled. And even if the data is properly formatted surprises can happen. For example if I overwrite "PID_gain = 0.999" with "PID_gain = 1.000" and the block boundary just happens to be at the "." the gain could end up being set to 1.999. But thats totally OK, I don't see any solution to this other than to backup the whole file until the new one is written completely.

    But for files that fit into a single block we should be safe in all cases. That's all I ask for. :)

  • AribaAriba Posts: 2,690

    As I see it in the code, the whole new file is written into new blocks. These blocks are set as temporary. On Close() the new file is activated and the blocks of the old file are marked as free.
    This means the file exists two times when you overwrite it, which can be a problem for really big files. I think you can not write a 8MB big file and then overwrite it again with 8MB. This will result in a Drive-Full error.

  • I don't see a problem with this as long as it's known and documented. I prefer safety over space-efficiency. If anyone needs to overwrite big files without power failure safety he can always choose to delete the old file, first.

  • JonnyMacJonnyMac Posts: 9,156
    edited 2023-09-23 16:56

    I updated my FFS demo in post #63.
    Oi. I found some gotchas while re-testing with Spin Tools and Flex Prop.

  • cgraceycgracey Posts: 14,206
    edited 2023-09-23 06:20

    @Ariba said:
    As I see it in the code, the whole new file is written into new blocks. These blocks are set as temporary. On Close() the new file is activated and the blocks of the old file are marked as free.
    This means the file exists two times when you overwrite it, which can be a problem for really big files. I think you can not write a 8MB big file and then overwrite it again with 8MB. This will result in a Drive-Full error.

    This is CORRECT!!!

    The new head block has the same ID as the old head block, but it remains inactive until all the new body blocks are written. So, in one atomic bit write, from 1 to 0, the NEW head block becomes superior to the OLD head block, and all its trailing body blocks (with NEW ID's) are suddenly attached to a head and a NEW file exists. At that point, the OLD/inferior head block and all its trailing body blocks are freed.

    If power goes down at any point and this process is disrupted, either the OLD file will still exist and the headless body blocks of the NEW file will be freed, or the NEW file will exist and any unfreed blocks from the OLD file will be freed. It all pivots on one of three lifecycle bits being programmed in the NEW head block, which make it superior to the OLD head block.

  • New shell demo for Chip's/Stephen's FFS uploaded to post #63.
    -- works with Propeller Tool, Spin Tools, and FlexProp

  • Today, I tried to run the first tests with the flash filesystem. I'm using the latest version of FlexProp (6.5.0) and a custom board with a ZD25W016 flash chip. It only has 2MB, bnut to be absolutely sure that it works with all of my boards I've set LAST_BLOCK = $0FF. So it should work with all chips of at least 1MB and small files. However, I don't get very far.

    struct __using("flash_fs.spin2") fs;
    
    void MountTest ()
    {
      printf ("start test, ");
      _waitms (1000);
      fs.mount ();
      printf ("mounted");
    }
    

    crashes immediately and doesn't output anything. If I comment out fs.mount (); it correctly prints "start test," and after a second "mounted". The file "flash_fs.spin2" is from "/flexprop/include/filesys/parallax" and looks like the latest version from september 19th. I think I did something fundamentally wrong....

  • JonnyMacJonnyMac Posts: 9,156
    edited 2023-09-25 17:13

    Since you're using C you might want to look at the shell demo in the FlexProp folder. I just ran it, mounted the flash (mount /pfs), changed to it (cd /pfs) and then did a directory listing of the files that are created by my Spin demo.

    I did test the driver with my demo in FlexProp, but only in Spin2.

  • That snippet works for me, but then my flash is already formatted with the Parallax file system. I'm not sure what would happen if it isn't formatted -- fs.mount() should return an error.

    That said, you might find it easier to use the flexspin C mount wrapper function, which would then allow you to use all the standard C I/O functions with the file system:

    #include <stdio.h>
    #include <sys/vfs.h>
    
    void MountTest ()
    {
      int r;
      printf ("start test, ");
      _waitms (1000);
      r = mount ("/pfs", _vfs_open_parallaxfs());
      if (r == 0) {
          printf ("mounted");
      } else {
          perror ("mount failed");
      }
    }
    
    void main() {
        MountTest ();
    }
    
  • I downloaded Jon's demo from post #63. I can compile and run it. It has some problems not outputting line-feeds to the terminal so I only see one line. But at least it doesn't crash. I can mount the it and stats displays something like ...(0.5MB) so I think my board and the flash is not the problem.

  • JonnyMacJonnyMac Posts: 9,156
    edited 2023-09-25 17:26

    Make sure you have Options/Use Internal PST Terminal selected. This is what I get on Windows.

  • Ok, I patched some of the CRs to LFs. I seem to haven't catched them all but now I can see at least some of the output. I did a mount, demo, vers, ls and stats. It shows:

    Mountedmount
       0 file(s)        0 bytes
                        506368 bytes free
    Writing 13 bytes to file1
    Writing 14 bytes to file2
    Writing 15 bytes to file3
    Writing 25 bytes to P2
    Writing 36 bytes to USA
    Writing 22 bytes to Fruits
       0 file(s)        0 bytes
                        482632 bytes free
    P2FFS 1.4.0ion
    P2FFS size... 524288 (0.5MB)
    Files........ 0
    Blocks....... 128
    - size....... 4096
    - used....... 6
    - free....... 122
    P2FFS>
    

    So 6 blocks are used by the 6 files written but ls says "0 files, 0 bytes". So something is broken but not all.

  • JonnyMacJonnyMac Posts: 9,156
    edited 2023-09-25 18:14

    It should look like this.

  • Ok, I've tested it now on 3 different boards. My CNC controller has a ZD25Q16C flash chip from Zetta. As mentioned above this doesn't work with the filesystem. One block per file is reported as used but the files don't show up. I guess the flash is somehow incompatible with what the filesystem expects. Maybe it has multi-level cells that don't support single bit writes when the previous content is different from $FF. If implemented in a clever way it should but sometimes the Chinese engineers have their own sort of logic...

    No big problem. I've bought 4 different flash chips for the purpose of trying out which one works best. I should find at least one that works.

    Then I tried the KISS board. It's one of the older ones with an AT25SF041 from Adesto with only 0.5MB. This time it looks much better. I can mount the filesystem, run "demo" and do a "dir". The files show up nicely. Of course, the 0.5MB is not enough to support both the boot image and the filesystem. But at least it shows that the filesystem works if the right chip is there and the problem is not somewhere else (compiler...).

    The third board is a newer version of the KISS board with an AT25SF321 with 4MB. It also works (almost) without problems. However, on both KISS boards the format command seems to hang. I've waited for more than one minute but it doesn't return. How long is a format expected to take? The data sheet says a full chip erase is <15s, a 4kB block erase <50ms. So writing 128 blocks should take only a few seconds.

Sign In or Register to comment.