@cgracey Chip, no matter if you implement subdirectories or tagging or whatever now or later or never at all...
Please reserve at least one flag bit in the block definitions to allow for future extensions. This makes it possible that the software including the filesystem will be updated while the data stored in the flash remains valid
Comment says "filenames are up to 59 characters, plus a zero terminator"
Doesn't say any characters are not allowed.
Seems you could just include a "\" in the filename to implement one's own directory structure as long as names are kept relatively short.
@Rayman said:
Comment says "filenames are up to 59 characters, plus a zero terminator"
Doesn't say any characters are not allowed.
Seems you could just include a "\" in the filename to implement one's own directory structure as long as names are kept relatively short.
I think a 59 char filename is too short if implementing directories as you suggest, perhaps much too short. 255 would be ample but I suggest 247 chars plus zero terminator so that 15 whole pages of data could fit in header blocks. Also, when reading filename stop when zero is read.
@Rayman said:
Comment says "filenames are up to 59 characters, plus a zero terminator"
Doesn't say any characters are not allowed.
Seems you could just include a "\" in the filename to implement one's own directory structure as long as names are kept relatively short.
I think a 59 char filename is too short if implementing directories as you suggest, perhaps much too short. 255 would be ample but I suggest 247 chars plus zero terminator so that 15 whole pages of data could fit in header blocks. Also, when reading filename stop when zero is read.
I thought the same this morning when looking at this. More of a gut feel than anything. Also some of that spare 247 byte space might be possible to re-use for other things later such as file creation timestamps or any other metadata associated with a file people might dream up....sorry the SW engineer in me is already trying to future proof someone else's thing, lol. The 256 bytes per block could burn 6% of the flash space if everything is a small file, which while not particularly excessive is still not an insignificant overhead.
I don't think this needs future proofing at all. Infact, it should definitely not have it if it comes at the expense of code complexity. The storage is so small that you probably end up rewriting it a lot. And if it gets used in an actual P2-based product, it's not like anyone's ever going to upgrade-in-place the filesystem on that, that's a silly case to waste even a single byte on.
@Rayman said:
Comment says "filenames are up to 59 characters, plus a zero terminator"
Doesn't say any characters are not allowed.
Seems you could just include a "\" in the filename to implement one's own directory structure as long as names are kept relatively short.
I think a 59 char filename is too short if implementing directories as you suggest, perhaps much too short. 255 would be ample but I suggest 247 chars plus zero terminator so that 15 whole pages of data could fit in header blocks. Also, when reading filename stop when zero is read.
I thought the same this morning when looking at this. More of a gut feel than anything. Also some of that spare 247 byte space might be possible to re-use for other things later such as file creation timestamps or any other metadata associated with a file people might dream up....sorry the SW engineer in me is already trying to future proof someone else's thing, lol. The 256 bytes per block could burn 6% of the flash space if everything is a small file, which while not particularly excessive is still not an insignificant overhead.
It's not "future proofing" or "premature optimization" to try and make the P2 the better, something which I've tried to do since day one on this forum.
I still think the 59 char limit for filenames is not enough. However if 247 is regarded as excessive then 127 would be a compromise.
@TonyB_ said:
It's not "future proofing" or "premature optimization" to try and make the P2 the better, something which I've tried to do since day one on this forum.
I still think the 59 char limit for filenames is not enough. However if 247 is regarded as excessive then 127 would be a compromise.
I wasn't disagreeing, just noting the tendency I have to start optimizing really early.
Yes, 59 characters is small for paths or tags. It will get expanded.
I optimized the SPI code to send and receive 1 bit per 8 sysclocks. So, at 320MHz, the flash's CK pin cycles at 40MHz. The speed limit for a normal read ($03 command) is 50MHz. This is a one-size-fits-all timing solution that is hardcoded to work from DC to 350MHz+. It's 2.5x faster than the first version.
PRI FlashCommand(Command, ByteCount)
org
fltl #SPI_DO '2! DO input
fltl #SPI_CK '2! reset CK smart pin
wrpin #%01_00101_0,#SPI_CK '2 set CK for transition output, starts out low
wxpin #4,#SPI_CK '2 set timebase to 4 clocks per transition
drvh #SPI_CS '2! CS high
waitx #14 '2+14 CS deselect is 50ns at 350 MHz (50 / 2.777ns = 18)
drvl #SPI_CS '2! CS low
movbyts Command,#%%1230 'reverse order of post-command address bytes for sending
end
FlashSend(@Command, ByteCount) 'send command
PRI FlashSend(BuffAddress, ByteCount) | Data
org
rdfast #0,BuffAddress 'start fast read
.byte rfbyte Data '2 read byte
rep @.r,#1 '2 protect from interrupts
fltl #SPI_CK '2! reset smart pin CK
drvl #SPI_CK '2! start smart pin CK, restarts base period
shl Data,#24 + 1 wc '2 get D7
drvc #SPI_DI '2! output D7
wypin #16,#SPI_CK '2* begin 16 clock transitions
rep @.r,#7 '2 repeat to output D6..D0
shl Data,#1 wc '2 get data bit
drvc #SPI_DI '2! output data bit
waitx #2 '2+2 delay makes 8 clocks/bit
.r
djnz ByteCount,#.byte '4|2 loop if another byte to send
drvl #SPI_DI '2! DI low
end
PRI FlashReceive(BuffAddress, ByteCount) | Data
org
wrfast #0,BuffAddress 'start fast write
.byte rep @.r,#1 '2 protect from interrupts
fltl #SPI_CK '2! reset smart pin CK
drvl #SPI_CK '2! start smart pin CK, restarts base period
wypin #16,#SPI_CK '2* begin 16 clock transitions
waitx #3 '2+3 align TESTP's to before clock fall (DC..350MHz+)
rep @.r,#8 '2 repeat to input D7..D0
waitx #2 '2+2 delay makes 8 clocks/bit
testp #SPI_DO wc '2 input data bit
rcl Data,#1 '2 save data bit
.r
wfbyte Data '2 write byte
djnz ByteCount,#.byte '4|2 loop if another byte to receive
drvh #SPI_CS '2! CS high, terminates command
wypin #1,#SPI_CK '2* CK high, leave SD card CS high
end
@TonyB_ said:
It's not "future proofing" or "premature optimization" to try and make the P2 the better, something which I've tried to do since day one on this forum.
I still think the 59 char limit for filenames is not enough. However if 247 is regarded as excessive then 127 would be a compromise.
I wasn't disagreeing, just noting the tendency I have to start optimizing really early.
Yes, 59 characters is small for paths or tags. It will get expanded.
Thanks Chip. If all files are single-block, filename length of 59/127/247 would use 1.44/3.1/6.0% of total space, so 127 probably best. I think FileCheck() should read up to the first zero byte in the filename and what comes after that is user-defined.
Re randomness, use getrnd() once only at the start to get an initial non-zero seed, then use the XORO32 ?? operator thereafter for random blocks. A lot of time was spent on XORO32 and its consecutive 16-bit PRN values will be more random than sampling GETRND every so often. Don't skip any XORO32 PRN words.
@cgracey said:
I updated the files in the top post.
I recommend using pulse mode smartpin and inverting the clock pin - fully CPOL=1. Doing this ensures least extraneous clocking of SD Card in SPI mode.
Oh, and SD interface mode doesn't depend on CS at all. It is contingent on CMD low (start-bit) when clocked. So, to prevent issuing an extraneous command, setting MOSI high upon EEPROM routine init and exit is advised.
Yeah it's a always a balance to avoid that one (which can lock in restrictions quite early) and also avoid it's evil brother called feature creep, which can also cause an endless misery.
@TonyB_ said:
It's not "future proofing" or "premature optimization" to try and make the P2 the better, something which I've tried to do since day one on this forum.
I still think the 59 char limit for filenames is not enough. However if 247 is regarded as excessive then 127 would be a compromise.
I wasn't disagreeing, just noting the tendency I have to start optimizing really early.
Yes, 59 characters is small for paths or tags. It will get expanded.
Thanks Chip. If all files are single-block, filename length of 59/127/247 would use 1.44/3.1/6.0% of total space, so 127 probably best. I think FileCheck() should read up to the first zero byte in the filename and what comes after that is user-defined.
Re randomness, use getrnd() once only at the start to get an initial non-zero seed, then use the XORO32 ?? operator thereafter for random blocks. A lot of time was spent on XORO32 and its consecutive 16-bit PRN values will be more random than sampling GETRND every so often. Don't skip any XORO32 PRN words.
EDIT:
Changed "iterations" to "16-bit PRN values"
I tried initially seeding a variable and then doing ??variable whenever I needed the next XORO32 value, instead of just doing a GETRND(). The results are about the same:
I am glad I tried this, just to see, but I think it's maybe a toss-up. I think the quality differences don't show up in something this coarse.
Also, if we have high-quality random numbers, which we do, thanks to you guys, it doesn't even matter how or when we pick them because they are just.... random.
One thing I didn't see in this 1.0 rev code Chip, was the ability to seek into a flash file at some file position. It's probably a complicated thing for writes and may be more questionable to support for that, but certainly for reads this is a very useful feature to have in order to parse or index structured data for example in larger files without needing to read complete 4kB blocks first just to skip over some of them. With the much slower speed of SPI flash vs HUB the ability to randomly seek into a file could become useful for improving read performance, especially when dealing with file sizes greater than HUB RAM can store directly. Of course you may still need to read the block headers to figure out sizes and next block IDs depending on what state you do keep in HUB etc but can hopefully still skip transferring in a lot of the other data if it is not needed.
It may be worth considering if/how file seeks (for reads) could be implemented at some point.
@cgracey said:
Yes, tags are a departure from convention, but they really open things up.
I don't really understand how a tagging system is anything but vastly inferior to nested directories in utility and efficiency.
Can you imagine the web without a tagging system? As in you just navigate the hierarchical file system underpinning all those web pages?
@Wuerfel_21 said:
Tbf I would consider directories an optional feature for the flash filesystem as is (limited to 15.5MB and all). Just files is fine.
I think it's not even needed. Just extend the code as needed for applications that want to "query" the filesystem using semantics, which is what my tagging mod will do for my application.
@cgracey said:
Yes, tags are a departure from convention, but they really open things up.
I don't really understand how a tagging system is anything but vastly inferior to nested directories in utility and efficiency.
Can you imagine the web without a tagging system? As in you just navigate the hierarchical file system underpinning all those web pages?
Yes? Many websites map into a directory hierarchy pretty directly. Otherwise you just use search and hyperlinks, right? Very few websites have a page tag system that's actually useful for finding something you want to read.
Anyways, the real issue is that you wouldn't solve any of the issues that directories solve.
@TonyB_ said:
It's not "future proofing" or "premature optimization" to try and make the P2 the better, something which I've tried to do since day one on this forum.
I still think the 59 char limit for filenames is not enough. However if 247 is regarded as excessive then 127 would be a compromise.
I wasn't disagreeing, just noting the tendency I have to start optimizing really early.
Yes, 59 characters is small for paths or tags. It will get expanded.
Thanks Chip. If all files are single-block, filename length of 59/127/247 would use 1.44/3.1/6.0% of total space, so 127 probably best. I think FileCheck() should read up to the first zero byte in the filename and what comes after that is user-defined.
Re randomness, use getrnd() once only at the start to get an initial non-zero seed, then use the XORO32 ?? operator thereafter for random blocks. A lot of time was spent on XORO32 and its consecutive 16-bit PRN values will be more random than sampling GETRND every so often. Don't skip any XORO32 PRN words.
EDIT:
Changed "iterations" to "16-bit PRN values"
I tried initially seeding a variable and then doing ??variable whenever I needed the next XORO32 value, instead of just doing a GETRND(). The results are about the same
Could you please post FlashFileSystem_16MB.spin2 that uses XORO32? It doesn't matter if non-random stuff is out-of-date.
@TonyB_ said:
It's not "future proofing" or "premature optimization" to try and make the P2 the better, something which I've tried to do since day one on this forum.
I still think the 59 char limit for filenames is not enough. However if 247 is regarded as excessive then 127 would be a compromise.
I wasn't disagreeing, just noting the tendency I have to start optimizing really early.
Yes, 59 characters is small for paths or tags. It will get expanded.
Thanks Chip. If all files are single-block, filename length of 59/127/247 would use 1.44/3.1/6.0% of total space, so 127 probably best. I think FileCheck() should read up to the first zero byte in the filename and what comes after that is user-defined.
Re randomness, use getrnd() once only at the start to get an initial non-zero seed, then use the XORO32 ?? operator thereafter for random blocks. A lot of time was spent on XORO32 and its consecutive 16-bit PRN values will be more random than sampling GETRND every so often. Don't skip any XORO32 PRN words.
EDIT:
Changed "iterations" to "16-bit PRN values"
I tried initially seeding a variable and then doing ??variable whenever I needed the next XORO32 value, instead of just doing a GETRND(). The results are about the same
Could you please post FlashFileSystem_16MB.spin2 that uses XORO32? It doesn't matter if non-random stuff is out-of-date.
I can do that tomorrow, but if you declare a VAR long 'Random' and seed it in Mount() by doing 'Random := getrnd()', and then replace the two getrnd() instances in NewBlock() with ??Random, you will have it.
@cgracey said:
Reminds me of this saying:
“Premature optimization is the root of all evil”
The temptation is overwhelming.
@rogloh said:
Yeah it's a always a balance to avoid that one (which can lock in restrictions quite early) and also avoid it's evil brother called feature creep, which can also cause an endless misery.
I totally agree. You can't please everyone, anyway. And I never asked for the actual implementation of any extension. I'm totally happy with the way it is, now.
But as somebody else in the live forum also suggested. It would be great if the software could identify IF there is a valid filesystem in the upper part of the flash ROM and hopefully WHAT version and format it is using. As there is no global FAT or master block it's hard to put an ID somehere to idetify the file system type and version.
If you think it's not worth to waste a single bit or byte for that then there are other possibilities. You could use a different CRC polynomial for a hypotetically (not yet existing) new format. An older software version doesn't know the new CRC and will simply ignore files using it. (The case of downgrading the software is rare and not very important, anyway)
But the opposite case of an updated software finding an older filesystem is totally totally realistic and useful, I think.
@TonyB_ said:
Re randomness, use getrnd() once only at the start to get an initial non-zero seed, then use the XORO32 ?? operator thereafter for random blocks. A lot of time was spent on XORO32 and its consecutive 16-bit PRN values will be more random than sampling GETRND every so often. Don't skip any XORO32 PRN words.
EDIT:
Changed "iterations" to "16-bit PRN values"
I tried initially seeding a variable and then doing ??variable whenever I needed the next XORO32 value, instead of just doing a GETRND(). The results are about the same
Could you please post FlashFileSystem_16MB.spin2 that uses XORO32? It doesn't matter if non-random stuff is out-of-date.
I can do that tomorrow, but if you declare a VAR long 'Random' and seed it in Mount() by doing 'Random := getrnd()', and then replace the two getrnd() instances in NewBlock() with ??Random, you will have it.
Chip, no need to post file now. What you've described is not the best way to do it. xoroshiro32++ generates a 16-bit PRN and as you know because it was your idea XORO32 does a double iteration. Max block number = $FFF therefore a 16-bit PRN is more than enough each time you need a new random block. However, using a separate ?? every time means that you're skipping half the PRNs which reduces the randomness. Instead you should use every 16-bit PRN value that ?? produces.
You'll need a new 32-bit PRN variable, say PRN32, and a new flag that toggles between true and false each time you want a new random block. If flag is true, then do PRN32 := ??Random and use low word of PRN32 for block. If flag is false, use high word of PRN32 for block. You have a choice of the low or high 12-bits of the word for the block, or any other 12-bit group. I'm not sure whether Evan tested low and high 12 bits but quality should be much the same as every bit is thought to be equally good. Randomness will be improved compared to your current method.
@cgracey said:
Reminds me of this saying:
“Premature optimization is the root of all evil”
The temptation is overwhelming.
@rogloh said:
Yeah it's a always a balance to avoid that one (which can lock in restrictions quite early) and also avoid it's evil brother called feature creep, which can also cause an endless misery.
One more thing that may be useful for a future revision of this software is the append operation.
It doesn't appear that we can do this right now with the 1.0 version code. When a file already exists and gets opened for writing this flash filesystem code will rewrite this file by starting the new writes at the beginning of the new file and deleting the old version of the file once closed. That's useful too of course, however for an application writing a log file to flash we may desire the ability to continue writing at the end of an existing file when we open it.
So maybe a future API to select this such as OpenAppend(pFilename) would be handy. With this capability you can just start your new writes at the end of an existing file by figuring out its length when it is opened and setting up the block pointer accordingly within the last block it uses. If this file doesn't already exist, it would just default to the current behaviour. Seems like it might be possible without a lot of work with any luck.
@rogloh said:
One more thing that may be useful for a future revision of this software is the append operation.
It doesn't appear that we can do this right now with the 1.0 version code. When a file already exists and gets opened for writing this flash filesystem code will rewrite this file by starting the new writes at the beginning of the new file and deleting the old version of the file once closed. That's useful too of course, however for an application writing a log file to flash we may desire the ability to continue writing at the end of an existing file when we open it.
So maybe a future API to select this such as OpenAppend(pFilename) would be handy. With this capability you can just start your new writes at the end of an existing file by figuring out its length when it is opened and setting up the block pointer accordingly within the last block it uses. If this file doesn't already exist, it would just default to the current behaviour. Seems like it might be possible without a lot of work with any luck.
Yes, I am working on this now. Also, we will have a circular file type, too.
@cgracey said:
Yes, I am working on this now. Also, we will have a circular file type, too.
Cool. A circular file will useful for other endless log files that need to be sized constrained, rather than continue consuming blocks indefinitely until we run out of space otherwise we need to manage that process manually to avoid that.
I imagine the wrap around position in the file itself would just have to be some multiple of the block size minus overheads in this case to avoid it becoming too complicated to manage, right?
@cgracey said:
Yes, I am working on this now. Also, we will have a circular file type, too.
Cool. A circular file will useful for other endless log files that need to be sized constrained, rather than continue consuming blocks indefinitely until we run out of space otherwise we need to manage that process manually to avoid that.
I imagine the wrap around position in the file itself would just have to be some multiple of the block size minus overheads in this case to avoid it becoming too complicated to manage, right?
I think the user will specify a size in bytes and the file system will take care of the blocks needed.
In our Live Forum you heard about the latest updates to Chip's new Flash Filesystem.
These updates can now be found at the P2-FLASH-FS GitHub repository:
v1.2.0 Features
Wear-leveling write mechanism
Writes are structured to facilitate recovery of file structure after unplanned power failure
Block identification is independent of a block's physical location
Filenames are 127+1 characters (plus zero terminator)
Seeks supported
File Append supported
Circular file reads/writes supported
Fully regression tested before this release (See RegresssionTests page for status and results)
Installation instructions are found on the top page of the Repo.
This version should now work with uSD without interfering. We need users to certify that this is now in good shape.
All methods are backed by a regression test suite. At each release, this one, and all upcoming releases, the test suite will be run before the release is posted so that we can be sure future updates will work when they are released.
What's next?
I'm currently working on validating the flash filesystem in a multi-cog environment. Once this testing is complete, I'll notify us all here in this thread.
I'm also planning on full directory functionality to follow in an upcoming release.
I've updated the latest flexspin github code to use the new version of the Parallax flash file system (thanks to both Chip and Stephen for the improvements). The shell demo allows you to mount the parallax file system with mount /pfs, after which you can do the usual commands like ls /pfs or cd /pfs. Before mounting it for the first time you may want to do mkfs /pfs to re-initialize your flash, as the new format is not compatible with the old.
Comments
@cgracey Chip, no matter if you implement subdirectories or tagging or whatever now or later or never at all...
Please reserve at least one flag bit in the block definitions to allow for future extensions. This makes it possible that the software including the filesystem will be updated while the data stored in the flash remains valid
Comment says "filenames are up to 59 characters, plus a zero terminator"
Doesn't say any characters are not allowed.
Seems you could just include a "\" in the filename to implement one's own directory structure as long as names are kept relatively short.
I think a 59 char filename is too short if implementing directories as you suggest, perhaps much too short. 255 would be ample but I suggest 247 chars plus zero terminator so that 15 whole pages of data could fit in header blocks. Also, when reading filename stop when zero is read.
I thought the same this morning when looking at this. More of a gut feel than anything. Also some of that spare 247 byte space might be possible to re-use for other things later such as file creation timestamps or any other metadata associated with a file people might dream up....sorry the SW engineer in me is already trying to future proof someone else's thing, lol.
The 256 bytes per block could burn 6% of the flash space if everything is a small file, which while not particularly excessive is still not an insignificant overhead.
Variable size structures and linked lists. :P
I don't think this needs future proofing at all. Infact, it should definitely not have it if it comes at the expense of code complexity. The storage is so small that you probably end up rewriting it a lot. And if it gets used in an actual P2-based product, it's not like anyone's ever going to upgrade-in-place the filesystem on that, that's a silly case to waste even a single byte on.
Reminds me of this saying:
“Premature optimization is the root of all evil”
The temptation is overwhelming.
It's not "future proofing" or "premature optimization" to try and make the P2 the better, something which I've tried to do since day one on this forum.
I still think the 59 char limit for filenames is not enough. However if 247 is regarded as excessive then 127 would be a compromise.
I wasn't disagreeing, just noting the tendency I have to start optimizing really early.
Yes, 59 characters is small for paths or tags. It will get expanded.
I optimized the SPI code to send and receive 1 bit per 8 sysclocks. So, at 320MHz, the flash's CK pin cycles at 40MHz. The speed limit for a normal read ($03 command) is 50MHz. This is a one-size-fits-all timing solution that is hardcoded to work from DC to 350MHz+. It's 2.5x faster than the first version.
PRI FlashCommand(Command, ByteCount) org fltl #SPI_DO '2! DO input fltl #SPI_CK '2! reset CK smart pin wrpin #%01_00101_0,#SPI_CK '2 set CK for transition output, starts out low wxpin #4,#SPI_CK '2 set timebase to 4 clocks per transition drvh #SPI_CS '2! CS high waitx #14 '2+14 CS deselect is 50ns at 350 MHz (50 / 2.777ns = 18) drvl #SPI_CS '2! CS low movbyts Command,#%%1230 'reverse order of post-command address bytes for sending end FlashSend(@Command, ByteCount) 'send command PRI FlashSend(BuffAddress, ByteCount) | Data org rdfast #0,BuffAddress 'start fast read .byte rfbyte Data '2 read byte rep @.r,#1 '2 protect from interrupts fltl #SPI_CK '2! reset smart pin CK drvl #SPI_CK '2! start smart pin CK, restarts base period shl Data,#24 + 1 wc '2 get D7 drvc #SPI_DI '2! output D7 wypin #16,#SPI_CK '2* begin 16 clock transitions rep @.r,#7 '2 repeat to output D6..D0 shl Data,#1 wc '2 get data bit drvc #SPI_DI '2! output data bit waitx #2 '2+2 delay makes 8 clocks/bit .r djnz ByteCount,#.byte '4|2 loop if another byte to send drvl #SPI_DI '2! DI low end PRI FlashReceive(BuffAddress, ByteCount) | Data org wrfast #0,BuffAddress 'start fast write .byte rep @.r,#1 '2 protect from interrupts fltl #SPI_CK '2! reset smart pin CK drvl #SPI_CK '2! start smart pin CK, restarts base period wypin #16,#SPI_CK '2* begin 16 clock transitions waitx #3 '2+3 align TESTP's to before clock fall (DC..350MHz+) rep @.r,#8 '2 repeat to input D7..D0 waitx #2 '2+2 delay makes 8 clocks/bit testp #SPI_DO wc '2 input data bit rcl Data,#1 '2 save data bit .r wfbyte Data '2 write byte djnz ByteCount,#.byte '4|2 loop if another byte to receive drvh #SPI_CS '2! CS high, terminates command wypin #1,#SPI_CK '2* CK high, leave SD card CS high endI updated the files in the top post.
Thanks Chip. If all files are single-block, filename length of 59/127/247 would use 1.44/3.1/6.0% of total space, so 127 probably best. I think
FileCheck()should read up to the first zero byte in the filename and what comes after that is user-defined.Re randomness, use
getrnd()once only at the start to get an initial non-zero seed, then use the XORO32??operator thereafter for random blocks. A lot of time was spent on XORO32 and its consecutive 16-bit PRN values will be more random than sampling GETRND every so often. Don't skip any XORO32 PRN words.EDIT:
Changed "iterations" to "16-bit PRN values"
I recommend using pulse mode smartpin and inverting the clock pin - fully CPOL=1. Doing this ensures least extraneous clocking of SD Card in SPI mode.
Oh, and SD interface mode doesn't depend on CS at all. It is contingent on CMD low (start-bit) when clocked. So, to prevent issuing an extraneous command, setting MOSI high upon EEPROM routine init and exit is advised.
Yeah it's a always a balance to avoid that one (which can lock in restrictions quite early) and also avoid it's evil brother called feature creep, which can also cause an endless misery.
I tried initially seeding a variable and then doing ??variable whenever I needed the next XORO32 value, instead of just doing a GETRND(). The results are about the same:
I am glad I tried this, just to see, but I think it's maybe a toss-up. I think the quality differences don't show up in something this coarse.
Also, if we have high-quality random numbers, which we do, thanks to you guys, it doesn't even matter how or when we pick them because they are just.... random.
One thing I didn't see in this 1.0 rev code Chip, was the ability to seek into a flash file at some file position. It's probably a complicated thing for writes and may be more questionable to support for that, but certainly for reads this is a very useful feature to have in order to parse or index structured data for example in larger files without needing to read complete 4kB blocks first just to skip over some of them. With the much slower speed of SPI flash vs HUB the ability to randomly seek into a file could become useful for improving read performance, especially when dealing with file sizes greater than HUB RAM can store directly. Of course you may still need to read the block headers to figure out sizes and next block IDs depending on what state you do keep in HUB etc but can hopefully still skip transferring in a lot of the other data if it is not needed.
It may be worth considering if/how file seeks (for reads) could be implemented at some point.
Can you imagine the web without a tagging system? As in you just navigate the hierarchical file system underpinning all those web pages?
I think it's not even needed. Just extend the code as needed for applications that want to "query" the filesystem using semantics, which is what my tagging mod will do for my application.
Yes? Many websites map into a directory hierarchy pretty directly. Otherwise you just use search and hyperlinks, right? Very few websites have a page tag system that's actually useful for finding something you want to read.
Anyways, the real issue is that you wouldn't solve any of the issues that directories solve.
Could you please post FlashFileSystem_16MB.spin2 that uses XORO32? It doesn't matter if non-random stuff is out-of-date.
I can do that tomorrow, but if you declare a VAR long 'Random' and seed it in Mount() by doing 'Random := getrnd()', and then replace the two getrnd() instances in NewBlock() with ??Random, you will have it.
I totally agree. You can't please everyone, anyway. And I never asked for the actual implementation of any extension. I'm totally happy with the way it is, now.
But as somebody else in the live forum also suggested. It would be great if the software could identify IF there is a valid filesystem in the upper part of the flash ROM and hopefully WHAT version and format it is using. As there is no global FAT or master block it's hard to put an ID somehere to idetify the file system type and version.
If you think it's not worth to waste a single bit or byte for that then there are other possibilities. You could use a different CRC polynomial for a hypotetically (not yet existing) new format. An older software version doesn't know the new CRC and will simply ignore files using it. (The case of downgrading the software is rare and not very important, anyway)
But the opposite case of an updated software finding an older filesystem is totally totally realistic and useful, I think.
Chip, no need to post file now. What you've described is not the best way to do it. xoroshiro32++ generates a 16-bit PRN and as you know because it was your idea XORO32 does a double iteration. Max block number = $FFF therefore a 16-bit PRN is more than enough each time you need a new random block. However, using a separate
??every time means that you're skipping half the PRNs which reduces the randomness. Instead you should use every 16-bit PRN value that??produces.You'll need a new 32-bit PRN variable, say PRN32, and a new flag that toggles between true and false each time you want a new random block. If flag is true, then do
PRN32 := ??Randomand use low word of PRN32 for block. If flag is false, use high word of PRN32 for block. You have a choice of the low or high 12-bits of the word for the block, or any other 12-bit group. I'm not sure whether Evan tested low and high 12 bits but quality should be much the same as every bit is thought to be equally good. Randomness will be improved compared to your current method.The story of my past 18 months, right there
Craig
One more thing that may be useful for a future revision of this software is the append operation.
It doesn't appear that we can do this right now with the 1.0 version code. When a file already exists and gets opened for writing this flash filesystem code will rewrite this file by starting the new writes at the beginning of the new file and deleting the old version of the file once closed. That's useful too of course, however for an application writing a log file to flash we may desire the ability to continue writing at the end of an existing file when we open it.
So maybe a future API to select this such as
OpenAppend(pFilename)would be handy. With this capability you can just start your new writes at the end of an existing file by figuring out its length when it is opened and setting up the block pointer accordingly within the last block it uses. If this file doesn't already exist, it would just default to the current behaviour. Seems like it might be possible without a lot of work with any luck.Yes, I am working on this now. Also, we will have a circular file type, too.
Cool. A circular file will useful for other endless log files that need to be sized constrained, rather than continue consuming blocks indefinitely until we run out of space otherwise we need to manage that process manually to avoid that.
I imagine the wrap around position in the file itself would just have to be some multiple of the block size minus overheads in this case to avoid it becoming too complicated to manage, right?
I think the user will specify a size in bytes and the file system will take care of the blocks needed.
NEWS
In our Live Forum you heard about the latest updates to Chip's new Flash Filesystem.
These updates can now be found at the P2-FLASH-FS GitHub repository:
Installation instructions are found on the top page of the Repo.
This version should now work with uSD without interfering. We need users to certify that this is now in good shape.
All methods are backed by a regression test suite. At each release, this one, and all upcoming releases, the test suite will be run before the release is posted so that we can be sure future updates will work when they are released.
If you find something not working as you expect, then please file an issue at my github repository issues page
What's next?
I'm currently working on validating the flash filesystem in a multi-cog environment. Once this testing is complete, I'll notify us all here in this thread.
I'm also planning on full directory functionality to follow in an upcoming release.
Enjoy!
Stephen
Github requires a login to file issues.
I've updated the latest flexspin github code to use the new version of the Parallax flash file system (thanks to both Chip and Stephen for the improvements). The shell demo allows you to mount the parallax file system with
mount /pfs, after which you can do the usual commands likels /pfsorcd /pfs. Before mounting it for the first time you may want to domkfs /pfsto re-initialize your flash, as the new format is not compatible with the old.No worries. You are always welcome to email me directly with details of the issue you wish to file if you are not a GitHub user.
or DM me for help.