Directory listing question using FSRW 2.6

Don M · 2013-04-29 18:11

Here is the code that I'm using:

pub SD_Directory | num_files, j, rslt

  rslt := \sd_mount
  if rslt < 0
    return
     
  num_files := 0
  sd.opendir

  slave.str(@sdfiledir1)
  
  repeat while 0 == sd.nextfile(@tbuf)
    num_files ++
  repeat j from 1 to num_files
    bytefill(@tbuf, 0, 20)
    sd.opendir
    repeat j
      sd.nextfile(@tbuf)
      sd[ 1 ].popen(@tbuf, "r")
    slave.str(@tbuf)
    repeat 15 - strsize(@tbuf)
      slave.tx(" ")
    slave.dec(sd[ 1 ].get_filesize)
    sd[ 1 ].pclose
    slave.tx(13)
    slave.tx(10)
  slave.str(string("Number of files: "))
  slave.dec(num_files)
  slave.tx(13)
  slave.tx(10)
  sd.pclose
  sd_unmount

I have 31 files on the sd card. When doing a directory listing the listing starts out fairly fast but then gradually slows down as the listing progresses. Anyone else ever notice this? what might be the bottle neck in this?

In case you're wondering- yes I have tried many different cards. It's not the cards. Is there a buffer somewhere that gets bogged down?

Mike G · 2013-04-29 18:16

It looks like the second (nested) repeat loop. Each iteration of j in the outer loop causes the inner loop to execute once, then twice... then 31 times. As written, the inner loop runs progressively slower with each outer loop iteration.

  repeat j from 1 to num_files
    bytefill(@tbuf, 0, 20)
    sd.opendir
    repeat j

Don M · 2013-04-29 18:24

Any suggestions for a fix? This was some code I found elsewhere on the forum.

Mike Green · 2013-04-29 18:33

You could look at the directory listing routine in FemtoBasic (for the FILES statement). It's a pretty straightforward directory listing routine as I remember it. I don't have access to it at the moment, but do a search for "FILES" in FemtoBasic.spin

Mike G · 2013-04-29 19:16

The following was copied form the FSRW source.

pub nextfile(fbuf) | i, t, at, lns
{{
'   Find the next file in the root directory and extract its
'   (8.3) name into fbuf.  Fbuf must be sized to hold at least
'   13 characters (8 + 1 + 3 + 1).  If there is no next file,
'   -1 will be returned.  If there is, 0 will be returned.
}}

It appears, invoking fsrw.nextfile(buffer) copies the 8.3 file name to the buffer argument. If no file is found then fsrw.nextfile(buffer) returns -1. The directory listing in the original posting can be simplified with a single repeat until. Instead of first finding the number of files keep incrementing a counting until fsrw.nextfile(buffer) returns -1. On each iteration that returns 0, either send the file name to the terminal, process the file name, or buffer the files name(s) for later processing.

Don M · 2013-04-30 05:20

After playing with this a bit I found that by not reporting the file size the directory listing is very fast. I commented out the 3 lines that involve the second instance of fsrw to read file size.

repeat while 0 == sd.nextfile(@tbuf)
    num_files ++
  repeat j from 1 to num_files
    bytefill(@tbuf, 0, 20)
    sd.opendir
    repeat j
      sd.nextfile(@tbuf)
      'sd[ 1 ].popen(@tbuf, "r")
    slave.str(@tbuf)
    repeat 15 - strsize(@tbuf)
      slave.tx(" ")
    'slave.dec(sd[ 1 ].get_filesize)
    'sd[ 1 ].pclose
    slave.tx(13)
    slave.tx(10)
  slave.str(string("Number of files: "))
  slave.dec(num_files)
  slave.tx(13)
  slave.tx(10)
  sd.pclose
  sd_unmount

Although it would be nice to have the filesize...

Mike Green · 2013-04-30 07:06

From FemtoBasic FILES statement:

if \fsrw.mount(spiDO,spiClk,spiDI,spiCS)
                  abort string("Can't mount SD card")
               fsrw.opendir
               b := 0
               d := false
               repeat while fsrw.nextfile(@f0) == 0
                  d := true
                  if b == 78
                     dsp.out(dsp#Cr)
                     b := 0
                  dsp.str(@f0)
                  ' put .popen / .get_filesize / .pclose here if you want
                  repeat 13 - strsize(@f0)
                     dsp.out(" ")
                  b := b + 13
               if d
                  dsp.out(dsp#Cr)
               \fsrw.unmount

Dave Hein · 2013-04-30 08:07

Can nextfile and popen be used in the same loop? I think FSRW uses the same sector buffer and state variables for both. You can try it to see if it works, but I think popen will mess up the state variables used by nextfiile. You could define two FSRW objects and use one for opendir/nextfile and the other for popen/get_filesize. Another possiblity is to modify the nextfile method, or add another method to FSRW that returns the filename and the filesize at the same time.

Don M · 2013-04-30 08:13

I was looking at that earlier this morning.

What are you doing with "b"? I see it starts out as "0" and you increment it by 13 through the repeat loop and if it equals 78 you do what? Send a CR? Are you counting characters?

Don M · 2013-04-30 08:15

Dave Hein wrote: »

Can nextfile and popen be used in the same loop? I think FSRW uses the same sector buffer and state variables for both. You can try it to see if it works, but I think popen will mess up the state variables used by nextfiile. You could define two FSRW objects and use one for opendir/nextfile and the other for popen/get_filesize. Another possiblity is to modify the nextfile method, or add another method to FSRW that returns the filename and the filesize at the same time.

That is what the original code I posted above does. It uses 2 iterations of FSRW. One to read the filenames and the other to get the filesize of the filename. It really slows down if you have a number of files to list.

Mike Green · 2013-04-30 08:26

Yes, "b" is used to keep track of the current display column number so a carriage return can be produced at the end of an 80 column display line. Each directory entry is displayed in 13 columns, so 6 entries = 78 columns. I didn't use a more meaningful name because this is part of a much larger section of code and "a", "b", "c", "d", and others are local temporary variables.

Dave Hein · 2013-04-30 08:30

I guess I was commenting more on Mike's code than yours. BTW. The "b" and the 78 that he uses are to just print 6 file names per line.

I think your code could be sped up a little bit by rewriting it as follows:

pub SD_Directory | num_files

  rslt := \sd_mount
  if rslt < 0
    return
     
  num_files := 0
  sd.opendir

  slave.str(@sdfiledir1)
  
  repeat while 0 == sd.nextfile(@tbuf)
    num_files ++
    sd[ 1 ].popen(@tbuf, "r")
    slave.str(@tbuf)
    repeat 15 - strsize(@tbuf)
      slave.tx(" ")
    slave.dec(sd[ 1 ].get_filesize)
    slave.tx(13)
    slave.tx(10)
  slave.str(string("Number of files: "))
  slave.dec(num_files)
  slave.tx(13)
  slave.tx(10)
  sd.pclose
  sd_unmount

This will still slow down as you proceed through the directory because popen will scan through the directory looking for the file each time. It takes longer to open a file that's located further in the directory. As I mentioned in my previous post, you could speed this up by modifying nextfile, or adding a new method to FSRW that returns both the filename and filesize.

JonnyMac · 2013-04-30 08:34

How many files do you need to support? I have a little music player app I'm working on for my column and was going to read the directory into RAM which allows me to list the files on an LCD and access them as I choose. Here's a demo of what I'm thinking that populates the files list and then displays the size.

pub directory | fcount, check, idx, p_fname, fsize

  ' populate filenames list first

  bytefill(@filenames, 0, 13 * 100)                                     ' clear list

  sd.opendir                                                            ' setup for .nextfile
  fcount := 0                                                           ' reset files count
 
  repeat
    check := \sd.nextfile(@filenames[13 * fcount])                      ' get next file in root
    if (check == 0)                                                     ' if found
      fcount += 1                                                       '  increment file count
    else                                                                ' else
      quit                                                              '  done

  ' iterate through list, show name and size

  term.tx(CLS)

  if (fcount > 0)
    term.str(@Banner)
    repeat idx from 0 to fcount-1                                       ' loop through files
      p_fname := @filenames[13 * idx]                                   ' point to name 
      fsize := open_file(p_fname, "r")                                  ' open, get size
      term.str(p_fname)                                                 ' print name
      term.tx(GOTOX)                                                    ' move cursor
      term.tx(16)
      if (fsize > 0)
        term.dec(fsize)                                                 ' print size
      else
        term.tx("?")
      term.tx(CR)

    term.tx(CR)
    term.dec(fcount)
    term.str(string(" file(s) found."))
      
  else
    term.str(string("No files found."))

  sd.pclose

Don M · 2013-04-30 08:36

Don M wrote: »

I was looking at that earlier this morning.

What are you doing with "b"? I see it starts out as "0" and you increment it by 13 through the repeat loop and if it equals 78 you do what? Send a CR? Are you counting characters?

I see now. You are listing them in a block sideways. When you get to 78 you start a new line.

Don M · 2013-04-30 08:42

Dave Hein wrote: »
I guess I was commenting more on Mike's code than yours. BTW. The "b" and the 78 that he uses are to just print 6 file names per line.

I think your code could be sped up a little bit by rewriting it as follows:
pub SD_Directory | num_files

  rslt := \sd_mount
  if rslt < 0
    return
     
  num_files := 0
  sd.opendir

  slave.str(@sdfiledir1)
  
  repeat while 0 == sd.nextfile(@tbuf)
    num_files ++
    sd[ 1 ].popen(@tbuf, "r")
    slave.str(@tbuf)
    repeat 15 - strsize(@tbuf)
      slave.tx(" ")
    slave.dec(sd[ 1 ].get_filesize)
    slave.tx(13)
    slave.tx(10)
  slave.str(string("Number of files: "))
  slave.dec(num_files)
  slave.tx(13)
  slave.tx(10)
  sd.pclose
  sd_unmount
This will still slow down as you proceed through the directory because popen will scan through the directory looking for the file each time. It takes longer to open a file that's located further in the directory. As I mentioned in my previous post, you could speed this up by modifying nextfile, or adding a new method to FSRW that returns both the filename and filesize.

Dave- That works 100% faster. Thanks for your help.

Don M · 2013-04-30 09:28

I'm stuck on trying to get the display formatting to work right. I took a bit from Mike's code and Dave's code to come up with the following:

pub SD_Directory2 | num_files, rslt, b

  rslt := \sd_mount
  if rslt < 0
    return
     
  num_files := 0
  b := 0 
  sd.opendir

  slave.str(@sdfiledir1)
  
  repeat while 0 == sd.nextfile(@tbuf)
    num_files ++
    sd[ 1 ].popen(@tbuf, "r")
    slave.str(@tbuf)
    repeat 12 - strsize(@tbuf)
      slave.tx(" ")
    slave.decn(sd[ 1 ].get_filesize, 8)
    slave.tx(" ") 
    if b == 42
      slave.tx(13)
      slave.tx(10)
      b := 0        
    b := b + 21
    
  slave.str(@sdfiledir2)
  slave.dec(num_files)
  slave.tx(13)
  slave.tx(10)
  sd.pclose
  sd_unmount

But for some reason I can't get the first line to work right. I tried changing the b := b + 21 and the if b == 42 statements but can't figure out why. Also tried initializing b to 1. The filename, space, 8 digit filesize and space equals 21

Here's what I get:

What am I doing wrong?

jurop · 2013-04-30 10:12

Have a look at how you built the loop - at a first glance, if you set b:= 21 before starting the loop - or moving the line b:=b+21 before the loop - it should work as desired.

Don M · 2013-04-30 10:37

I moved b := b +21 to before the if b == 42 statement and it works fine now. Thanks for your help.

Directory listing question using FSRW 2.6

Comments