PHAT Engine Beta - Full File System Support For The Prop - Looking for testers
Kye
Posts: 2,200
Thanks for reading,
·
I’ve finished basic beta testing for a new file system driver designed exclusively for the propeller chip. The driver is designed to allow the user to access a FAT32 or FAT16 file system on an SD card attached to the propeller chip.
·
The driver itself contains three software parts – A block driver and a FAT32/16 file system driver and a RTC driver.
·
The block driver gives the file system driver access to a piece of physical media that can be read and written in 512 bytes chunks. In this case that piece of physical media is a SD card.
·
The file system driver is the heart of the system and uses the block driver to read and write to a FAT16/32 file system on the physical media. The file system driver is what you the user communicates to.
·
The RTC driver gives the file system driver access to time and date information so that it can properly apply time stamps on files and folders in the file system.
·
So, the features the file system has included contain the ability to:
·
-- Mount any one of the 4 logical partitions on a standard computer file system.
-- Mount a FAT16/32 file system that is located inside of any of the 4 logical partitions.
·
After the file system is mounted additional commands may be executed. If the file system has not been mounted yet all other commands are disabled.
·
The block driver will prevent the user from mounting the physical media if the block driver itself cannot initialize the physical media which must occur before mounting the physical media.· Additionally, if a block read or write error occurs then the block driver will automatically un-mount the physical media in order to protect the file system from improper access. Any files that were open at that time are also closed.
·
If an error occurs then the file system must be remounted. This is done to prevent the file system driver from improperly accessing any file system using possibly outdated environmental variables.
·
If the file system was successfully mounted then the mount function will return true. If the file system failed to mount the mount function will return false.
·
All other use functions that have the possibility of failing behave in the same manner. True for success and false for failure. If a function returns false then the file system must be remounted.
·
However, higher level functions will use the length of a string to report success, failure, or other error formats.
·
For example, when opening a file the function will return a string of length if the file is found, a string of no length if the file was not found, and false if an error occurred. Because the string pointer is an address other than 0 it is effectively a logical true. Note that the string pointer returned will always be to the same address.
·
To access files in the file system the driver uses listing functions which are given to the user to find files and folders. The most important of these include “list name”, “list size”, list search”, and “list reset”. Through using these four functions the file system driver is able to find all files and folders in the current directory and properly display them for listing purposes.
The full set of listing functions give the user the ability to:
·
-- List the next file or directory 8.3 name.
After the next file or directory 8.3 file name is listed then the user can retrieve additional information which includes:
·
-- The creation date and time.
-- The last access date.
-- The last modification date and time.
-- The file or folder size. (Folders have no size.)
-- The read only, hidden, system, directory, and archive attributes for that file or folder.
·
All of this functionality is controlled through the “list name” function which lists the next file or folder in the current directory. That said, to start listing from the top of the current directory the “list reset” function allows the user reset the “list name” function so that it will start from the top of the current directory.
·
Finally, the “list search” function allows the use to search through the current directory for a specific file or folder by comparing 8.3 file names.
·
So, an 8.3 file name is a file or folder name that has 8 leading characters and 3 trailing characters. The names are stores in all capital letters and must be unique in the current directory. The file system driver will handle the conversion from natural 8.3 file names to standardize 8.3 file names for you.
·
So, you can enter names in any of these formats:
·
“name.txt” – maps to “NAME···· TXT”
“namedStuff.text” – maps to “NAMEDSTUTEX”
“namesThatAreToLong” – maps to “NAMESTHA·· “
·
The file system driver will grab the first 8 characters for the file or folder name and the next 3 characters only after a period for the extension. Any invalid characters being entering in the 8.3 file name are automatically turned in an underscore. The driver will allow you to enter completely invalid names; however it will always apply the proper rules converting them to proper names. That said, what you enter telling the driver what to do and what it does may differ if your 8.3 file name does not follow naming convention rules.
·
A good primer on this can be found at http://en.wikipedia.org/wiki/8.3_filename
·
A good primer on FAT16/32 can also be found at http://en.wikipedia.org/wiki/File_Allocation_Table
·
Long file names are not supported at this time due to their complexity and irrelevance since the FAT16/32 file system operates without them as they are an extension.
·
A good primer on Long file names can be found at http://en.wikipedia.org/wiki/Long_filename
·
Finally, the “list name” function always returns names in the 8.3 file name format in standard form.
·
Now, the more advanced features of the file system included the ability to:
·
-- Change the current working directory.
-- Change the attributes for a file or folder.
-- Rename files or folders.
-- Delete files or folders.
-- Create new files or folders.
·
By using these five functions the user can build their own file and folder structure from the ground up and edit already made files or folders for their use.
·
Next up are the file access functions. Note that when a file is not open these functions will do nothing and when a file is opened only these functions will do anything and all other functions are disabled.
·
These functions give the user the ability to:
·
-- Open a file for reading, writing, or appending (a mode of writing).
-- Close open files and write their last access, modification, and last size data out to the physical media.
-- Seek back and forth throughout a file in an optimized way.
-- Get the current position within a file.
-- Read a character at a time from a file.
-- Write a character at a time from a file.
-- Read a sector (512 bytes) at time from a file.
-- Write a sector (512 bytes) at time from a file.
·
By using these functions the user has full FAT32/16 file system access.
·
Last but not least the file system driver supports the ability for the user to query the free space on the card. This function returns then the number of free sectors on the partition dedicated to the FAT32/16 file system. Since this function performs a linear count on all clusters of the FAT it can take a long time to finish executing and may not return for many minutes.
·
So, the file system driver has the following limitations:
·
-- The partition must not be more than 1TB in size and must located in the first 1TB of sectors on the physical media. Because SPIN performs signed math the 2TB limit is not possible.
·
-- The maximum file size is (2GB – 512B). Any file greater than this size can still be access but the bytes above the limit will be ignored.· The file size query will also never return a number above this limit. Because SPIN performs signed math the 4GB limit is not possible.
·
-- A directory can contain 65536 entries and no more. This included the hidden long file names so the limit may be reached even before 65536 files are found. This is not actually a limit of the file system driver but mandated by Microsoft specifications.
·
Features that are not supported but may be supported in the future include:
·
-- FAT32 free space queries and next free cluster optimizations. (For faster FAT32 file system access.)
-- Accessing the volume ID.
-- Setting the file system error flags when an error occurs.
-- Formatting the partition.
·
Thanks again for reading,
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,
Post Edited (Kye) : 7/28/2009 3:20:44 PM GMT
·
I’ve finished basic beta testing for a new file system driver designed exclusively for the propeller chip. The driver is designed to allow the user to access a FAT32 or FAT16 file system on an SD card attached to the propeller chip.
·
The driver itself contains three software parts – A block driver and a FAT32/16 file system driver and a RTC driver.
·
The block driver gives the file system driver access to a piece of physical media that can be read and written in 512 bytes chunks. In this case that piece of physical media is a SD card.
·
The file system driver is the heart of the system and uses the block driver to read and write to a FAT16/32 file system on the physical media. The file system driver is what you the user communicates to.
·
The RTC driver gives the file system driver access to time and date information so that it can properly apply time stamps on files and folders in the file system.
·
So, the features the file system has included contain the ability to:
·
-- Mount any one of the 4 logical partitions on a standard computer file system.
-- Mount a FAT16/32 file system that is located inside of any of the 4 logical partitions.
·
After the file system is mounted additional commands may be executed. If the file system has not been mounted yet all other commands are disabled.
·
The block driver will prevent the user from mounting the physical media if the block driver itself cannot initialize the physical media which must occur before mounting the physical media.· Additionally, if a block read or write error occurs then the block driver will automatically un-mount the physical media in order to protect the file system from improper access. Any files that were open at that time are also closed.
·
If an error occurs then the file system must be remounted. This is done to prevent the file system driver from improperly accessing any file system using possibly outdated environmental variables.
·
If the file system was successfully mounted then the mount function will return true. If the file system failed to mount the mount function will return false.
·
All other use functions that have the possibility of failing behave in the same manner. True for success and false for failure. If a function returns false then the file system must be remounted.
·
However, higher level functions will use the length of a string to report success, failure, or other error formats.
·
For example, when opening a file the function will return a string of length if the file is found, a string of no length if the file was not found, and false if an error occurred. Because the string pointer is an address other than 0 it is effectively a logical true. Note that the string pointer returned will always be to the same address.
·
To access files in the file system the driver uses listing functions which are given to the user to find files and folders. The most important of these include “list name”, “list size”, list search”, and “list reset”. Through using these four functions the file system driver is able to find all files and folders in the current directory and properly display them for listing purposes.
The full set of listing functions give the user the ability to:
·
-- List the next file or directory 8.3 name.
After the next file or directory 8.3 file name is listed then the user can retrieve additional information which includes:
·
-- The creation date and time.
-- The last access date.
-- The last modification date and time.
-- The file or folder size. (Folders have no size.)
-- The read only, hidden, system, directory, and archive attributes for that file or folder.
·
All of this functionality is controlled through the “list name” function which lists the next file or folder in the current directory. That said, to start listing from the top of the current directory the “list reset” function allows the user reset the “list name” function so that it will start from the top of the current directory.
·
Finally, the “list search” function allows the use to search through the current directory for a specific file or folder by comparing 8.3 file names.
·
So, an 8.3 file name is a file or folder name that has 8 leading characters and 3 trailing characters. The names are stores in all capital letters and must be unique in the current directory. The file system driver will handle the conversion from natural 8.3 file names to standardize 8.3 file names for you.
·
So, you can enter names in any of these formats:
·
“name.txt” – maps to “NAME···· TXT”
“namedStuff.text” – maps to “NAMEDSTUTEX”
“namesThatAreToLong” – maps to “NAMESTHA·· “
·
The file system driver will grab the first 8 characters for the file or folder name and the next 3 characters only after a period for the extension. Any invalid characters being entering in the 8.3 file name are automatically turned in an underscore. The driver will allow you to enter completely invalid names; however it will always apply the proper rules converting them to proper names. That said, what you enter telling the driver what to do and what it does may differ if your 8.3 file name does not follow naming convention rules.
·
A good primer on this can be found at http://en.wikipedia.org/wiki/8.3_filename
·
A good primer on FAT16/32 can also be found at http://en.wikipedia.org/wiki/File_Allocation_Table
·
Long file names are not supported at this time due to their complexity and irrelevance since the FAT16/32 file system operates without them as they are an extension.
·
A good primer on Long file names can be found at http://en.wikipedia.org/wiki/Long_filename
·
Finally, the “list name” function always returns names in the 8.3 file name format in standard form.
·
Now, the more advanced features of the file system included the ability to:
·
-- Change the current working directory.
-- Change the attributes for a file or folder.
-- Rename files or folders.
-- Delete files or folders.
-- Create new files or folders.
·
By using these five functions the user can build their own file and folder structure from the ground up and edit already made files or folders for their use.
·
Next up are the file access functions. Note that when a file is not open these functions will do nothing and when a file is opened only these functions will do anything and all other functions are disabled.
·
These functions give the user the ability to:
·
-- Open a file for reading, writing, or appending (a mode of writing).
-- Close open files and write their last access, modification, and last size data out to the physical media.
-- Seek back and forth throughout a file in an optimized way.
-- Get the current position within a file.
-- Read a character at a time from a file.
-- Write a character at a time from a file.
-- Read a sector (512 bytes) at time from a file.
-- Write a sector (512 bytes) at time from a file.
·
By using these functions the user has full FAT32/16 file system access.
·
Last but not least the file system driver supports the ability for the user to query the free space on the card. This function returns then the number of free sectors on the partition dedicated to the FAT32/16 file system. Since this function performs a linear count on all clusters of the FAT it can take a long time to finish executing and may not return for many minutes.
·
So, the file system driver has the following limitations:
·
-- The partition must not be more than 1TB in size and must located in the first 1TB of sectors on the physical media. Because SPIN performs signed math the 2TB limit is not possible.
·
-- The maximum file size is (2GB – 512B). Any file greater than this size can still be access but the bytes above the limit will be ignored.· The file size query will also never return a number above this limit. Because SPIN performs signed math the 4GB limit is not possible.
·
-- A directory can contain 65536 entries and no more. This included the hidden long file names so the limit may be reached even before 65536 files are found. This is not actually a limit of the file system driver but mandated by Microsoft specifications.
·
Features that are not supported but may be supported in the future include:
·
-- FAT32 free space queries and next free cluster optimizations. (For faster FAT32 file system access.)
-- Accessing the volume ID.
-- Setting the file system error flags when an error occurs.
-- Formatting the partition.
·
Thanks again for reading,
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,
Post Edited (Kye) : 7/28/2009 3:20:44 PM GMT
Comments
Right now the driver uses a slow block driver than has an acess speed of about 3KB a seconds so the driver will not be blazing fast.
A better block driver will be released when the driver leaves beta testing.
However, until then please feel free to use the slower block driver and file system driver to check out all the functionality.
This demo uses the serial port and the prop terminal to give the user acess to a shell command line like enviorment to move arround on the SD card.
You will need a DS1307 RTC to use the timing functionality. If you do not have one then the driver will still work but it will use incorrect date and time stamps. Look in the clockEngine.spin file for wiring details.
The block driver uses the same pins as FSRW for block acess to the SD card. The same pins should be compatible. The wiring information can be found in the FATEngine.spin file.
The demo uses the propeller chip's serial lines to communicate back to the prop terminal on the computer and all wiring for any platform should be compatible.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,
Post Edited (Kye) : 7/29/2009 6:19:19 PM GMT
I'm taking suggestions on any other functions to add or change·and I'm looking for help from anyone who can provided a more powerful block driver.
Also, please post feedback on how the file system driver preforms and if you have found any problems with it. I already know the block driver has problems.
Thank for your help,
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,
Post Edited (Kye) : 7/27/2009 9:10:11 PM GMT
Kye - you are cranking out some awesome stuff. Keep up the good work.
Does this object take three cogs then? One for RTC, one for block and one for file system? If so, any chance to reduce the RTC into one of the other cogs? You said if the RTC IC isn't attached it gives bogus data/time - any chance to force a specific date and time? For instance a RTC chip might not be used in my system, but the date/time may come periodically over another medium like radio/wire. The Prop should be able to track date/time enough with periodic updates.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Timothy D. Swieter, E.I.
www.brilldea.com - Prop Blade, LED Painter, RGB LEDs, 3.0" LCD Composite video display, eProto for SunSPOT
www.tdswieter.com
Inside the file system driver object near the bottom of the code is the update time function which controls the time stamps used in the object. Its very straight forward to change the place where the time stamps are derived from. You can also just put constants in there.
The block driver is the only thing that uses a cog. The file system driver is just a library of rountines. Its pretty much like a FSRW++.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,
You do not need the DS1307 to test the code.
If your system has a 5Mhz crystal and your pins are configured as such:
Data_Out_Pin = 0 '' ─ Data Out - To SD Card. '' Clock_Pin = 1 '' ─ Clock - To SD Card. '' Data_In_Pin = 2 '' ─ Data In - To SD Card. '' Chip_Select_Pin = 3 '' ─ Chip Select - To SD Card.You can test my demo with no problems. It uses pin 31 for receiving and pin 30 for transmiting just like fullDuplexSerial.
Just load the demo program up and go.
If you do have a DS1307 please attach it to the I2C lines the propeller chip using for booting purposes.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,
Post Edited (Kye) : 7/28/2009 4:56:56 PM GMT
I don't have a DS1307 clock on this prop board, and don't really want to take the time to configure one for this test, so I replaced the clock driver delay routine with a simple prop delay function, and replaced the filesystem timestamp function with a dummy timestamp for now.
But what I've been working on is not an SD device, but a hard drive. So for my purposes, I got rid of all specific card access routines, and created just three interface routines - device init, block read and block write, which should be able to work with either an SD or HD device. I would think your routines could use rokiki/lonesock's fast SD routines by using a similar interface to their code - device init, sector read, sector write, with options to read or write multiple sectors.
I formatted the hard drive with two partitions, one FAT16 and one FAT32, and stuck a few text files in each for testing, haven't yet gotten the partition signature recognized, but I've had it working in the past so I don't anticipate it will take too long before I can test your filesystem routines on my hard drive.
David
Since block read and write are essentailly the same function I combined them into one. I also made my block driver have a bit more control over intialization so that the higher level driver could let the block driver do all the work. I'll port the FRSW rouintine over sooner or latter but I'm trying to get all the bugs out of the file system driver first.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,
Post Edited (Kye) : 7/28/2009 5:07:32 PM GMT
Between being buried in work and expo stuff I'm hunting for Propeller time,
but count me in! As soon as I get a chance to get back into the lab this will
be on my list of things to test. Looks like the perfect answer to upgrading
PropDOS.
OBC
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
New to the Propeller?
Visit the: The Propeller Pages @ Warranty Void.
Hanno
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Download a free trial of ViewPort- the premier visual debugger for the Propeller
Includes full debugger, simulated instruments, fuzzy logic, and OpenCV for computer vision. Now a Parallax Product!
Cards which mounted:
* SanDisk 256MB SD (FAT16)
* SanDisk 2GB uSD (FAT16)
* SanDisk 2GB "Ultra II" SD (both FAT16 and FAT32, I reformatted to test)
Cards which failed to mount:
* Kodak 2GB SD (FAT16)
* Panasonic 8MB SD (FAT16)
* PNY 4GB uSDHC (FAT32)
* Canon MMC+ 32MB (FAT16) - I know you never said this would work with MMC, but I thought I'd try [noparse][[/noparse]8^)
Things that worked:
* ls (mostly)
* cd (mostly)
* cat
I did not try writing to any of the cards, 50% isn't _that_ good of odds
Directory listing had an issue where the period was not displayed in the filenames. (e.g. "song_f~1mp3")
Changing the directory using "." or ".." was weird(for example, in the root dir do a "cd ..").
Other than that (and the speed) it's looking great.
Jonathan
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
lonesock
Piranha are people too.
Lonesock, do you have a good spin only driver that I can port over to this piece of code? I really need to get a better block driver.
There are a few things I have to add to the block driver to make it compatible with my code however. Like automatic umounting and such.
More developement on the way!
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,
This will definately add to the great collection of code being developed at the moment. Sadly, I do not have the time (or currently a board) to test it.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Links to other interesting threads:
· Home of the MultiBladeProps: TriBladeProp, RamBlade, TwinBlade,·SixBlade, website
· Single Board Computer:·3 Propeller ICs·and a·TriBladeProp board (ZiCog Z80 Emulator)
· Prop Tools under Development or Completed (Index)
· Emulators: Micros eg Altair, and Terminals eg VT100 (Index) ZiCog (Z80), MoCog (6809)
· Search the Propeller forums (via Google)
My cruising website is: ·www.bluemagic.biz·· MultiBladeProp is: www.bluemagic.biz/cluso.htm
You can download the block drivers from the SourceForge site:
fsrw.hg.sourceforge.net/hgweb/fsrw/
Just pick the revision you want, there will be a "files" line, pick sdspi.spin, for example, then hit "raw". Or you can use a Mercurial client. I use TortoiseHg on Windows.
Jonathan
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
lonesock
Piranha are people too.
Also, the way LS list the names of files is not an error. Its how the names are stored on the FAT. I'll need to add some string processing to the listing command to make that look better.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,
(All the guys who were interested before in my open discussion of how to implement this... Where are you?)
Thanks,
...
Yes, the code does work now. Try out all the features and see what you think.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,
I've got your mount method correctly pulling up the four partition addresses and sizes from the MBR of my HD, but I think I have an error in how my code translates the partition LBA address into the old cyl/head/sector fields of the hard disk registers.
I do get the correct partition signature from partition zero but not the other three, so I want to get that fixed before doing anything more.