Help traversing directory structure with FAT16/32 Full System Driver
pgbpsu
Posts: 460
I'm using Kye's Full System File driver to interact with an SD card for logging data. At start-up (and at other times) I need to traverse the SD card to figure out what's on there (and the number of files). Does anyone have code that does this which I could follow as an example. My SD card structure is pretty simple:
I can loop over the root directory to list contents by doing this (taken from an APP note):
But I can't figure out how to get into subdirectories and then back out. Any help would be greatly appreciated.
Thanks,
Peter
The following code lists the files in the root directory, then changes to the directory TO_SEND, and lists its contents. I'm including it here for others that may face this problem.
Here is the output
Thanks for all the input. As far as I'm concerned this is solved.
/TO_SEND/ ' one directory
/some_files_in_here ' flat files in here
/SENT/ ' another directory with subdirectories
/YYYYMMDD/HHMMSS.CSV
/YYYYMMDD/HHMMSS.CSV
/YYYYMMDD/HHMMSS.CSV
I can loop over the root directory to list contents by doing this (taken from an APP note):
repeat while(entryName := SD.LIST_ENTRIES("N"))
'make sure the entry is not a directory
ifnot(SD.LIST_IS_DIRECTORY)
UART.PUTC(DEBUG, CR)
SD.OPEN_FILE(entryName, "R")
UART.STR(DEBUG, entryName) ' print out the filename
SD.CLOSE_FILE
numFiles++
But I can't figure out how to get into subdirectories and then back out. Any help would be greatly appreciated.
Thanks,
Peter
The following code lists the files in the root directory, then changes to the directory TO_SEND, and lists its contents. I'm including it here for others that may face this problem.
UART.STR(DEBUG, string(13,"Files on disk: "))
SD.LIST_ENTRIES("W") ' wrap around
repeat while(entryName := SD.LIST_ENTRIES("N"))
if(SD.LIST_IS_DIRECTORY)
UART.PUTC(DEBUG, CR)
UART.PUTC(DEBUG, "\")
UART.STR(DEBUG, entryName) ' print out the filename
else
UART.PUTC(DEBUG, CR)
UART.PUTC(DEBUG, " ")
UART.STR(DEBUG, entryName) ' print out the filename
numFiles++
entryName := \SD.CHANGE_DIRECTORY(string("\TO_SEND"))
UART.STR(DEBUG, string(13,"Changing to directory: "))
UART.STR(DEBUG, entryName) ' print out the directory name
SD.LIST_ENTRIES("W") ' wrap around
repeat while(entryName := SD.LIST_ENTRIES("N"))
ifnot(SD.LIST_IS_DIRECTORY)
UART.PUTC(DEBUG, CR)
UART.PUTC(DEBUG, " ")
UART.STR(DEBUG, entryName) ' print out the filename
UART.PUTC(DEBUG, " ")
numFiles++
Here is the output
Files on disk: ~1.TRA \TO_SEND \TRASHE~1 15082616.30C 15082616.31C \SPOTLI~1 15082710.20C USDA.TXT \SENT Changing to directory: TO_SEND 15082616.32X 15082616.35X 15082616.40X 15082616.45X 15082616.50X 15082616.55X 15082617.00X 15082617.05X
Thanks for all the input. As far as I'm concerned this is solved.
Comments
It seems you desire something like a CD command, and to retain the current path.
You do understand that there is a Directory attribute included with the file name? You would have to identity all file names that have a Directory attribute. There are other attributes as well - hidden, system, read-only, archive.
I can't help but wonder if you want an easy option or a complete FAT file management utility set. I haven't looked at Kye's Complete File System in detail... but will do so.
So far, the link to a An006 - FAT16/32 Full File System Driver is giving me a 404-Not Found. That is a bit frustrating....forum changeover recently.
===========
For many, the easiest way to use the Propeller with an SDcard is to not use sub-directories.... less search code involved. If it isn't in the top directory, it is just considered an unrelated item. To really seek out sub-directories, you have a . and .. to direct searches up and in the current directory, and you have to buffer a lot of data about a directory tree.
If you desire another way to explore a whole file system tree in Propeller on an SDcard, you might need to do so interactively rather than via robot.
So you might consider Dave Hein's pfth Forth V1.03 which includes Spinix for SDcard file management. It also provides an ASCII text editor and a few other nice items.
http://forums.parallax.com/discussion/143472/pfth-an-ans-forth-interpreter-for-the-prop
Dave's work might also help you realize how time consuming complete searches can be.
I guess Kye's driver must have a similar method but it would help if you attached or had a link to the object you used.
It seems to me that you want a CD for change directory, a DIR to list directory contents, and maybe a few other specific DOS like commands.
I wrote a C driver that uses buttons and an LCD for navigating card files, and if I am not mistaken, it accesses sub-directories just fine. Perhaps there may be a clue in the code, to get you want you want.
http://forums.parallax.com/discussion/160923/c-simpleide-directory-listing-of-sd-card-files#latest
The number you have reached, AN006, is not in service at this time.
The new number is:
https://www.parallax.com/downloads?title_1=an0
Please make a note of it.
More or less. I was hoping there was an existing example out there but didn't do a very good job of explaining what I want to do. I'd like to keep 2 different directories and move things from one directory to the other after a certain amount of processing has been done to each file.
Using Kye's object I can create the directories and write files into them. The problem comes when I want to process the files found in one directory and move it to the 'Processed' directory. My problem is actually pretty simple. I simply need a list (one-by-one is fine) of the files in my 'To_Process' directory. Once I have the filename I can open that file and manipulate it as needed then move it to the new location.
Doing this without directories is possible (and in some way's prefered by me the coder) but the user would like to retain more info about the files than can fit in 8.3 filename- hence the directory structure. I'm more familiar with FSRW and have used it quite a lot. I like the speed and memory footprint but it doesn't allow for directories.
Hi Peter,
I can't say how Kye's driver does it but I suppose both approaches (his and yours) have to conform to the FAT standard so they must be similar at some level. His code (found here) has the necessary methods. I need to go back and re-create the trouble I was having. I've tried so many different things that didn't work I've lost track and started thinking about ways around this!
Unfortunately, Kye's code is too complicated for my simple brain. It's also quite large for what I (think) I need so I've been trying to remove methods that I don't use (such as all the write/read byte, word, long, string). I'm doing 512-byte buffer transfers to speed things up.
Here's the method for changing directories. Not sure it's very revealing. But I don't think changing directories is my problem. It's get a file listing in the new directory. The code I posted above (taken from the App Note) works fine for listing the root directory. It just doesn't seem to work for the new directory I've changed into.
PUB changeDirectory(directoryPathName) '' 38 Stack Longs '' //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// '' // Searches the file system for the specified directory and changes the current directory to be the specified directory. '' // '' // Returns the directory's name. '' // '' // If an error occurs this method will abort and return a pointer to a string describing that error. '' // '' // If a file is open when this method is called that file will be closed. '' // '' // DirectoryPathName - A file system path string specifying the path of the directory to search for. '' //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// closeFile if(lockFileSystem("R")) if(rootWithSpace(directoryPathName)) result := string("/") currentWorkingDirectory := 0 else result := evaluateName(evaluatePath(directoryPathName, "R")) ifnot(isDIRDirectory) abortError(Not_A_Directory) currentWorkingDirectory := readDIRCluster unlockFileSystemI must have had an old copy stashed on my computer and didn't realize the Parallax link was broken. I should have included a copy. Apologies. Publison, thanks for locating that.
But as far as I remember you can provide filename and path together as one string to open/close el al.
The driver is automagically switching into the right directory and opening, writing, closing or deleting the file.
Same goes for the directory.
So you do NOT need to do any CD commands, just provide the full path and filename/directory name to the methods.
Basically KYE's driver does support a current directory, but you do not need it at all.
Just use path and filename as string and you are fine.
Enjoy!
Mike
What if one doesn't know the filenames? I want to list the contents in a particular directory so I can work on each file in that directory. But I don't know the filenames ahead of time and therefore can't construct the full path.
I have to explain further. FAT_Engine has a concept of a current directory, but it changes every time you access a file. So you can CD into a directory and read the directorys/files in it.
But as soon as you access a file/directory outside of that scope you lose your current directory.
You basically need to track that for yourself.
In the FAT32 file system a directory is saved like a file, just with some other attribute.
Thus the test for: 'ifnot(SD.LIST_IS_DIRECTORY)' in your code in the first post.
If you do a 'if' instead of 'ifnot' you will get the directory names.
CDing into a directory (also possible with a full path) will set some sort of actual directory. But it will be gone at the next file access.
Because Fat_Engine has just one buffer.
But you can use multiple copies of the driver, each having its own VAR space, but sharing the code. The VAR space basically is unique to the object and the code and data and pasm driver get shared.
declare as
FAT[2] FatEngine... in the obj Section
Just mount one of them, the others are mounted also. (Just one start and one mount)
Now you have two instances to talk to.
One taking care of your 'current' directory to iterate thru the list, the other for file access.
And the file access will not clobber your listing of the directory.
Hope this helps you a bit.
Enjoy!
Mike
But Rumors told me that OpenSpin now can remove not used functions automagically.
I always used the PropTool and have at least 20 different Versions of Fat_Engine with commented out source to save space.
And it is quite painful to strip Fat_Engine of not used functions. And it is BIG. But also very powerful compared to FSRW.
But putting a array of Fat_Engine objects will just add some 600-700 bytes to your memory footprint per further instance. One sector buffer of 512 and some variables to hold the state. Not bad for having multiple files or directories open at the same time.
Enjoy!
Mike
I appreciate your help.
My problem is that I can't get the Kye's driver to list the files once I've changed directories. I think I've got a handle on how it should work. It just doesn't. I know I haven't posted enough code for anyone to diagnose my problem. I'm on the road and don't have access to my system to post actual code that's not working. I should be able to do that tomorrow. I was originally looking for examples because I knew I'd be on the road for a few days. To make things simpler and easier for others to help me, maybe I should have waited until I after I returned.
If you have an example of listing root directory contents followed by cd'ing into another directory and listing those contents, that would be ideal.
I promise I'll post actual code (and results) tomorrow.
Thanks
Just re-read your post. Maybe I'm not following very well. I'll re-create what I thought should work and post results tomorrow. If it's true that I'll need two instances of Kye's driver to manage this I'll have to find a work around. A bit frustrating- that driver advertises itself as quite capable, but so far I've not found it easy to use or simple to follow. Maybe that's what Rokiki and lonesock left directory support out of their driver?
The problem basically is the recursion you want to have. That does not really work with so limited memory.
But as far as I grooked your problem, you have to access one directory listing and then do some operations on the files therein, copying them then somewhere else.
To access multiple files at the same time requires a separate Fat_Engine object per file. Same with FSRW. And for Fat_Engine a directory is - basically - just a file.
If them file you need to work on fit in your remaining memory (I guess not) you can do something like - Read whole file into hub - process - save in different location. That could be done with one instance of FSRW or Fat_Engine.
But if you need access to multiple file at the same time you need multiple instances of the drivers. And your directory listing is also a file.
so declare say FAT[3] ... instead of FAT... in the object section. FAT is my standard name, you might have a different one.
start and mount just the first instance, say FAT[0].xxx
FAT[0].xxx instead of FAT.xxx
do not start or mount the other instances.
Then use FAT[0].xxx for directory access to iterate thru the files,
use FAT[1].xxx for reading your file and FAT[2].xxx for writing and everything will be smooth.
will cost you not much hub memory for the other two instances.
And the code gets easy.
Enjoy!
Mike
Kye's code is 'industrial rigorous'. He provided the low-level items for just about all and everything a commercial application might want to choose from. You are correct to weed out the unneeded.
The Propeller's limited 32K Hubram memory is soon overwhelmed by the vastness of the modern SDcard with up to 32Gbyte of file storage. A complete survey of such a card requires storing interim results and final reports on the card itself (if there is space) and will be slow going at best with lots of recursion.
I certainly can sympathize, but the problem is mostly that it is too low-level for a new user that has been entombed in DOS. One immediately is entangled with Master Boot Records, partitions, file pointers (called inodes in Unix/Linux) and so on.
So suddenly, you feel as if you have fallen down a rabbit hole and into the world of the actual detail required to support a file system. Without a real interest into what are the inner workings of FAT16/32, it all tends to repulse first-time users. (I have been nibbling away at FAT16/32 for years, but am no master.)
AN006 helps a little, but isn't trying to get the user to use a file system so much as it is trying to get key information to those already in-the-know.
Volunteerism and open-source are wonderful things, Kye's provided a lot. But Parallax should consider taking another step toward filing the gap with more useful educational documentation and code. They simply will get more sales if there is a less painless entry into SDcards.
In the meantime, idBruce has presented a C solution, and both Peter J and Dave Hein have offered up Forth solutions. Dave's work seems to focus mainly on FAT16.
For instance in my EASYNET server module if a webpage is not found then this bit of code runs: and then when it's time to send it it is simply: where FILE@ returns with the 32-bit starting address of the file in virtual memory, FSIZE@ returns with the size of the file as read from the directory and BLKSEND armed with the source and count sends that off in chunks down the current socket.
In fact we can interact with all the various components of the FS for instance:
MOUNT
ls FOPEN HTTP404.HTM...opened at 0001.D7DA ok
FILE@ FSIZE@ ok
.S SD DUMP cat HTTP404.HTM...opened at 0001.D7DA The interactive FOPEN <name> confirmed that the file was opened (reporting sector address) and promptly set file pointers etc for us. Next we ask for the virtual memory address of the file with FILE@ and the reported size with FSIZE@ and have a look at the stack with .S. After that I just decide to do a hex dump just as I would any memory but use the SD modifier so that it uses the SD virtual memory space instead and viola! Then list the file using the "cat" command. Simple, maybe that's why I call it "EASYFILE".
Oh, and one more little thing, how about we index in and read a byte or long from that file, say from offset 64:
FILE@ $40 + XC@ .BYTE 76 ok
FILE@ $40 + X@ .LONG 3A72.6576 ok
He has churned out a lot of functionality for the Propeller, give it a try if you don't desire all the twists and turns of low-level coding.
Hi Mike-
See my top post for the solution. I'm not sure where the recursion you mentioned came up but I think my problem was much simpler than that. In any case thanks for the input.
It sure looks like he's on to something. Unfortunately for this project, backing and starting again with either C (idBruce's suggestion) or Tachyon isn't an option but I will have to look at Tachyon for my next project.
Thanks