Long directory listing using fsrw

pgbpsu · 2009-10-29 15:41

I'm trying to use fsrw to read the contents of an SD card and return both file names and sizes. However what I thought would work doesn't. Here's what I'm doing.

uarts.str(DEBUG,string(13,"Listing Directory Contents: ", 13))
sdfat.opendir
repeat while 0 == sdfat.nextfile(@tbuf)
  uarts.str(DEBUG,@tbuf)
  uarts.str(DEBUG,string(" filesize:"))
  uarts.dec(DEBUG,sdfat.getfilesize)
  uarts.str(DEBUG,string(13))
uarts.str(DEBUG,string(13,"That's what you've got on disk!", 13))

Here's what I get:
Listing Directory Contents:
140709.CSV filesize:32768
160709.CSV filesize:32768
09091900.22M filesize:32768
That's what you've got on disk!

The file listing is correct but the sizes are not. It seems to get the correct file sizes if I open each file using popen and then request the size:

file_open := sdfat.popen(@tbuf, "r",2008,12,9,12,10,10)
uarts.str(DEBUG,@tbuf)
uarts.str(DEBUG,string(" filesize:"))
uarts.dec(DEBUG,sdfat.getfilesize)

09091900.22M filesize:1600752
This works.

So my question is how can I loop over each file on disk without creating an array in which I store ALL the file names. I don't know how many files will be on the disk but it should reach into the hundreds. I don't want to set aside that much memory in the Prop for something that's already saved on the SD card.

It seems like this should be straight forward, but I'm clearly missing something. Thanks in advance for any suggestions.
Regards,
Peter

lonesock · 2009-10-29 15:48

When you are reading a directory, the "file" that is open _is_ the directory listing. With the new FSRW you can declare multiple instances of the object, and use one to do the directory scan, and the other to open and get the filesize of each entry.

OBJ
  sdfat[noparse][[/noparse] 2 ]: "fsrw"

  ' make sure to mount sdfat[noparse][[/noparse]0] (you only need to mount one of the array)

PUB show_dir
    ' opening the dir is just like opening a file
    sdfat.opendir
    repeat while 0 == sdfat.nextfile(@tbuf)
      ' show the filename
      term.str( @tbuf )
      repeat 15 - strsize( @tbuf )
        term.tx( " " )
      ' so I need a second file to open and query filesize
      sdfat[noparse][[/noparse] 1 ].popen( @tbuf, "r" )
      term.dec( sdfat.get_filesize )
      sdfat[noparse][[/noparse] 1 ].pclose      
      term.str( string( " bytes", $0D ) )

Jonathan

edit: forgot the spaces inside the [noparse][[/noparse] ] brackets

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
lonesock
Piranha are people too.

Post Edited (lonesock) : 10/29/2009 3:56:35 PM GMT

pgbpsu · 2009-10-29 17:03

Jonathan-

Thanks for the reply (and all your work on the code). I'm using FSRW 1.7. I tried 2.3 but ran into problems with my cards so I backed up to what I already had working. I'll have another crack at 2.3 to see if that will solve it.

Thanks,
Peter

Post Edited (pgbpsu) : 10/29/2009 5:20:02 PM GMT

localroger · 2009-10-29 18:22

If you look at the PropCMD object in the obex, you'll see a way to do this with the older FSRW. I basically hacked FSRW to return the entire directory entry instead of just the filename so you can get any info that is in there, including the file length.

pgbpsu · 2009-10-29 18:27

I wasn't able to get fsrw2.3 working so I'm stuck using 1.7. Here's what I ended up doing which seems to work:

           uarts.str(DEBUG,string(13,"Listing Directory Contents...", 13))
           uarts.str(DEBUG,string(   "FILENAME       SIZE(BYTES)"))
           num_files := 0
           sdfat.opendir
           repeat while 0 == sdfat.nextfile(@tbuf)
             num_files++
           uarts.putc(DEBUG,13)
           repeat j from 1 to num_files 
             bytefill(@tbuf, 0, 20)  ' zero out tbuf
             sdfat.opendir
             repeat j
               sdfat.nextfile(@tbuf)
             file_open := sdfat.popen(@tbuf, "r",2008,12,9,12,10,10)
             uarts.str(DEBUG,@tbuf)
             repeat 15 - strsize(@tbuf)
               uarts.putc(DEBUG,32)
             uarts.str(DEBUG,Num.ToStr(sdfat.getfilesize, Num#SDEC11))
             sdfat.pclose
             uarts.str(DEBUG,string(13))
           uarts.str(DEBUG,string("Total number of files: "))
           uarts.dec(DEBUG,num_files)
           uarts.putc(DEBUG,13)

Here's what it produces:
Listing Directory Contents...
FILENAME SIZE(BYTES)
140709.CSV 324308
160709.CSV 9144
09091900.22M 67584
_~1.TRA 4096
09091906.08M 116736
09091915.00M 589824
09092000.03M 829440
09092100.03M 632832
SPEED.TST 64512
SPEED2.TST 64512
09102121.25S 8003760
200929~1.SU 1600752
200929~2.SU 1600752
200929~3.SU 1600752
200929~4.SU 1600752
200929~5.SU 1600752
Total number of files: 16

Which is correct; so I guess this is solved except that now I'd like to know how much of the disk is used and how much is free. Any suggestions on that?

Thanks,
Peter

lonesock · 2009-10-29 18:29

localroger said...
If you look at the PropCMD object in the obex, you'll see a way to do this with the older FSRW. I basically hacked FSRW to return the entire directory entry instead of just the filename so you can get any info that is in there, including the file length.

Great idea! If you need to stick with an older version of FSRW, this mod is probably easier than making sure you can do multiple FSRW objects.

Jonathan

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
lonesock
Piranha are people too.

lonesock · 2009-10-29 18:34

pgbpsu said...
I wasn't able to get fsrw2.3 working so I'm stuck using 1.7 ...

Sorry, I should have mentioned...can you try FSRW 2.4?

http://forums.parallax.com/showthread.php?p=844245

Jonathan

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
lonesock
Piranha are people too.

lonesock · 2009-10-29 18:38

pgbpsu said...
Which is correct; so I guess this is solved except that now I'd like to know how much of the disk is used and how much is free. Any suggestions on that?

Well, I can add a routine to the SD block layer to query the actual card's size. More importantly, though, to get the used/free info from the currently mounted partition, you'd have to walk through the whole FAT table. Rokicki would have a much better idea as to what that would involve than I.

Jonathan

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
lonesock
Piranha are people too.

pgbpsu · 2009-10-29 18:43

@localranger-

By the time I checked my mail and read your post, I've found a workable solution. Thanks for the suggestion, I'll have a look at PropCMD.

@lonesock-

A method to read the actual card size would be great (as long as it's not too much trouble). Since I'm reading each file's size I can keep track of the total and subtract it from the card's actual size. If that's within +/- 10% of the free space on the disk that would be fine. That would meet me needs.

Thanks,
Peter

lonesock · 2009-10-29 18:51

I'm a bit busy right now, but I'm pretty sure I remember Kye having code to query the card's size. You could hack that into FSRW. I also pinged Rokicki, so he may be able to respond after a while.

(BTW, if you pull the latest version of FSRW off the Mercurial server at SourceForge, there is a setdate function, which might be of use to you: fsrw.hg.sourceforge.net/hgweb/fsrw/fsrw/ )

Jonathan

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
lonesock
Piranha are people too.

rokicki · 2009-10-29 18:55

I'd be happy to write code to calculate the free space on disk. That would require a walk of the FAT, which can be large.
More problematic is the Spin code to check which entries are free; a typical FAT has 65K entries, each of which must be
individually checked. This would take perhaps a second or two in Spin (unless someone can come up with a super-fast
way to do it in Spin without executing an op for each entry).

If a slow one is sufficient, say the word and I'll write it.

File size during directory listings: that can be done.

Note that FAT32 includes an entry in the boot block that is the free space on the card (I think). fsrw does not currently
update this value (doing so would/could slow things down, and also increases the chances of trashing the card). It also
does not use this value. If the prop were a larger system with more memory, we would almost certainly maintain this
value and use it (and thus avoid the FAT scan).

FSRW *could* be modified to track the count of free FAT entries itself, so the free space scan could be done *once* and
then every subsequent time that value would just be updated based on fsrw's work. This way only the first free space
call would be slow and every subsequent one would be fast. This would be a tiny bit more work (mostly in the testing).
If this suffices where the slow-every-time does not, let me know.

Returning the "size of the card" is trivial (if you mean the size of the formatted volume on the card); returning the raw
size is a bit more work (but I've done it before) but is also probably less useful.

pgbpsu · 2009-10-29 19:04

@lonesock-

When I first searched the forums for help with filesize stuff I found a list of features Kye was working on. But I'm unable to locate that code and haven't heard anything from Kye (but it's not even been a full day yet and I have no idea what time zone he works in). The setdate function may be of use. This application has gps time and I'm actually using to to successfully time stamp my files. I have yet to find out if the fixed time/date I'm using in my popen when reading files sizes changes anything. I hope not; that's why i used the "r" option. I totally understand about being busy so please don't worry about this.

@rokicki-
I think we may have have gone round about this once before but at the block layer. After looking around a bit, I found some old code of mine that keeps reading the blocks until it craps out and claims that's the size of the disk. I'm not sure that was ever such a good idea and other than writing it up and playing with it, I never used it. I've moved up a layer to spin and my timing requirements are pretty relaxed. A routine that took a couple of seconds wouldn't be an issue for what I'm currently doing. I should point out that my working platform uses fsrw 1.7 and I'm only using disks formatted as FAT16.

Thanks,
Peter

lonesock · 2009-10-29 19:12

rokicki said...
Returning the "size of the card" is trivial (if you mean the size of the formatted volume on the card)

FSRW has the functions "getclustersize" and "getclustercount". (I had forgotten about that)

Jonathan

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
lonesock
Piranha are people too.

pgbpsu · 2009-10-29 19:18

@lonesock-
Thanks for pointing out rokicki's comment about 'size of card'. I somehow missed that.

Indeed, I just want to know what size the formatted volume card. Is it as simple as clustersize * clustercount?

Even I can do that!

p

lonesock · 2009-10-29 20:24

Well, clustersize is in bytes and you don't want to over-flow your result, so maybe use KB (assuming your clustersize is 1K or larger):

disk_KB := (sdfat.getclustersize >> 10) * sdfat.getclustercount

Jonathan

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
lonesock
Piranha are people too.

rokicki · 2009-10-29 20:29

Right, to be safe, the cluster size is always >= 512, so if you shift that by 9, and then multiply, and shift the result by 1,
you'll get kilobytes. That will suffice for a card size well past what SD cards are spec'ed to support.

pgbpsu · 2009-10-29 20:36

Gentleman thank you both. Got it working.
Regards,
Peter

rokicki · 2009-10-29 20:47

BTW, if I do code up a free space routine, it will probably be called "getfreeclustercount()" and return a cluster
count, so you'd have to do the multiplication/scaling yourself anyway.

Long directory listing using fsrw

Comments