fsrw isn't flushing all data

Circuitsoft · 2013-09-28 16:30

Hi,
I'm trying to write CSV to a log file on an SD card, and I'm finding that the file ends with several 512-byte blocks of nulls rather than the correct data.

I'm making a GPS data-logger based on the Skytraq Venus 6 chipset, configured to binary mode. My serial-handling cog calls proc_msg when it finds a complete navigation message from the GPS. 'Ser' is a FullDuplexSerial4port object with port 0 connected to the GPS and port 1 going to the upstream USB port.

I have a button attached to pin 22 with a simple hardware debounce circuit, and an LED on pin 23 that's handled by a 3rd cog.

PUB run_button | evt, last_state, last_time, failed
  evt := \sdfat.mount_explicit(24, 25, 26, 27) ' SD card on pins 24-27
  if evt
    led := LED_RED
    ser.str(1, string("Mount error "))
    Dec(evt, 0)
    ser.Tx(1, $0d)
  else
    led := LED_GREEN
    ser.str(1, string("Mount succeeded", $0d))
  
  repeat
    evt := 0
    if last_state and not ina[22]
      ' Button pressed
      
      last_time := cnt
      evt := 1
    if ina[22] and not last_state
      ' Button released
      last_time := cnt - last_time
      evt := 2
    last_state := ina[22]
    if evt == 2
      if last_time > (clkfreq << 1)
        ' Button held at least two seconds
        led := LED_GREEN_FAST_BLINK
        led := LED_OFF
        sdfat.unmount
      elseif last_time > (clkfreq >> 3)
        ' Button held at least an eigth of a second - real press
        if led =< LED_YELLOW
          ' Not in a state we can do anything about
        else
          if led == LED_GREEN
            failed := \sdfat.popen(string("log.csv"), "a")
            if failed
              led := LED_RED
            led := LED_GREEN_BLINK
            
          elseif led == LED_GREEN_BLINK
            led := LED_GREEN_FAST_BLINK
            sdfat.pclose
            led := LED_GREEN
    if good and (led == LED_GREEN_BLINK)
      Dec(week, 1)
      sdfat.pputc(",")
      Dec(centiseconds, 1)
      sdfat.pputc(",")
      Dec(lat, 1)
      sdfat.pputc(",")
      Dec(lon, 1)
      sdfat.pputc(",")
      Dec(alt, 1)
      sdfat.pputc($0d)
      sdfat.pputc($0a)
      good := 0

PRI proc_msg | i
  'ser.str(1, string("Got GPS: la"))

  ' Week is a 16 bit value, so need to 0-fill it
  ' Other values are 32 bits, so any existing data gets shifted out.
  week := 0
  repeat i from 7 to 8
    week <<= 8
    week |= gps_string[i]
  repeat i from 9 to 12
    centiseconds <<= 8
    centiseconds |= gps_string[i]
  repeat i from 13 to 16
    lat <<= 8
    lat |= gps_string[i] 
  repeat i from 17 to 20
    lon <<= 8
    lon |= gps_string[i] 
  repeat i from 25 to 28
    alt <<= 8
    alt |= gps_string[i]
  good := 1
  Dec(week, 0)
  Ser.Tx(1, ",")
  Dec(centiseconds, 0)
  Ser.Tx(1, ",")
  Dec(lat, 0)
  Ser.Tx(1, ",")
  Dec(lon, 0)
  Ser.Tx(1, ",")
  Dec(alt, 0)
  Ser.Tx(1, $0d)

So, I turn on the board, and my LED turns green. I wait until the GPS lock LED starts blinking and then I press the button such that my LED starts blinking green. After 10-15 seconds, I press the button for several seconds, and when I release, the LED turns off. I wait a few seconds in case the SD card needs to flush itself then depower the board, put the SD card into the computer, and get the attached file. The first 512 bytes of the file are correct, then it's nulls up until the proper length of the file, where it ends in $0d, $0a.

Am I doing something wrong that the file isn't getting flushed properly?

Don M · 2013-09-28 18:17

Isn't there a pflush method in the fsrw object? I'd try that before closing the file.

Circuitsoft · 2013-09-28 18:23

First line of pclose is pflush.

SRLM · 2013-09-28 18:40

I've experienced the same symptoms symptoms when I remove the SD card without making sure it's closed.* I don't know if that's the case, but just thought I'd throw it out there.

* This also includes the case of violent motion disrupting the contacts on the card.

Don M · 2013-09-28 18:42

The file you attached- looks like it did not finish writing the alt (altitude?) I see a bunch of nuls and no cr, lf at the end. Maybe you have to change your method to finish writing your data and when good := 0 then close your file?

evanh · 2013-09-28 18:43

Circuitsoft wrote: »

... where it ends in $0d, $0a.

I've not read any of the code but I'd say that is a bit of an indication that you are feeding nulls into the file.

Don M · 2013-09-28 18:55

Maybe change this part of the code:

elseif led == LED_GREEN_BLINK
  if good == 0
    led := LED_GREEN_FAST_BLINK
    sdfat.pclose
    led := LED_GREEN

Don M · 2013-09-28 19:00

How are you unmouting the card after you close the file?

Circuitsoft · 2013-09-28 19:11

Don M wrote: »

How are you unmouting the card after you close the file?

sdfat.unmount

Don M · 2013-09-28 19:15

No what I was refering to was how your logic flows to unmount the card after you have closed the file. So once you press your button I'm assuming you should:

1. finish any writing of data so you have a complete data element
2. close file (includes flushing of data)
3. unmount card

I don't follow your logic in your code as to how this happens. But maybe it's just me... :-)

Circuitsoft · 2013-09-28 19:17

evanh wrote: »

I've not read any of the code but I'd say that is a bit of an indication that you are feeding nulls into the file.

Turns out that's not the case, it's just a mistake based on how I was hexifying the file to read it. The file comes out the expected length, but only has 512 bytes of data up top.

Circuitsoft · 2013-09-28 19:20

Don M wrote: »

No what I was refering to was how your logic flows to unmount the card after you have closed the file. So once you press your button I'm assuming you should:

1. finish any writing of data so you have a complete data element
2. close file (includes flushing of data)
3. unmount card

I don't follow your logic in your code as to how this happens. But maybe it's just me... :-)

Writing a complete record happens within the same function that handles the button. So, I only write a line to the file if there is a new line ready (good flag) and if I'm in the running mode (LED_GREEN_BLINK). If something happens within the button code to change the file state, such tests don't happen during writing a record, only after. I guess I'm making the assumption that pflush/pclose/unmount block until said operations complete.

Don M · 2013-09-28 19:26

To me it looks as if you are unmounting the card (holding the button down for 2 seconds) and never closing the file before you unmount it.

Don M · 2013-09-28 19:29

So does a momentary press (less than 2 seconds) open the file and then another momentary press close the file ?

Circuitsoft · 2013-09-28 19:30

Yes. Also, in fsrw (2.6, which I'm using), the unmount method is:

pub unmount
  pclose
  sdspi.stop

So, calling unmount closes the file anyway.

kuroneko · 2013-09-28 19:36

What happens if you just run a simple loop (and nothing else) doing the following (using fake values for now):

repeat 1000
  Dec(week, 1)
  sdfat.pputc(",")
  Dec(centiseconds, 1)
  sdfat.pputc(",")
  Dec(lat, 1)
  sdfat.pputc(",")
  Dec(lon, 1)
  sdfat.pputc(",")
  Dec(alt, 1)
  sdfat.pputc($0d)
  sdfat.pputc($0a)

This loop should be preceded by mount/popen calls and followed by pclose/unmount calls. This is to establish whether there is an issue with fsrw or with your code (could be memory corruption affecting fsrw).

Don M · 2013-09-28 19:36

Sorry for the confusion then. I use FSRW2.6 in a data logging project and don't have any problem with it storing any erroneous characters. I have compared numerous times the data stored on the card to the logic trace data and it has always been exact.

I see what you are saying about some of the methods being included in the others however I still flush and close the file before unmouting.

Circuitsoft · 2013-09-28 23:15

I think I found my problem (buffer overflow in GPS parsing). I fixed that, and now I can't get my button working right...

fsrw isn't flushing all data

Comments