FSRW 2.6 versus upper 32K of EEPROM for storage. Speed compare

Don M · 2012-09-27 10:45

Which method would most likely be faster for storing say 24K of data? Faster as in throughput...

Mike Green · 2012-09-27 10:56

A lot depends on how you write the information to the EEPROM and how much buffering is in the SD card you're using. If you write only a few bytes at a time, almost anything else will be faster. An EEPROM takes about 5ms to write data regardless of how much data is to be written (up to the page size ... often 256 bytes). FSRW buffers the data into blocks of 512 bytes. If the file is kept open and the SD card isn't close to being full, it can be very fast. There have been some threads measuring maximum write speeds to an SD card using FSRW. An SD card is particularly fast if the data to be written fits into the on-card buffers. That way, the program doesn't have to wait for the data to be actually written. Some flash memories have two buffers so one can be written to the flash memory while the other is being filled with new data.

For 24K, I suspect that an SD card will be faster than EEPROM.

Don M · 2012-09-27 11:59

Actually I would be writing it in 512 byte chunks. Thanks for your help.

MagIO2 · 2012-09-27 12:09

I've used FRAMs as a replacement of EEPROM. If I remember right the propeller would be the limiting factor for writing to the FRAM. No wait after reaching the block-size! The advantage is, that you don't need additional PINs and board estate.

Don M · 2012-09-27 12:39

What about those Winbond Flash chips? The W25Q80 1 MB version. How does it compare to EEPROM or uSD card?

rokicki · 2012-09-27 17:17

Don M wrote: »

Which method would most likely be faster for storing say 24K of data? Faster as in throughput...

Can you share a bit more on what you plan to do?

Do you care only about write speed, or read speed, or both?

Will you have 24K in memory all at the same time, or will it come in at a steady pace and you
want to log it persistently?

I'm pretty sure fsrw can write that much data (48 blocks) very quickly, but a lot depends on
how exactly you write it. If you pwrite() 512-byte chunks, and use the normal (fast) SPI driver
I think your throughput can easily exceed a megabyte a second.

Will stuttering (the occasional block that takes longer to write) be a problem? Because SD
cards can "stutter" sometimes due to their internal firmware . . .

-tom

Don M · 2012-09-27 19:34

Actually what I'm logging is data @ 9600 baud polled anywhere from every 10 ms to 50 ms. It's the writing I was concerned about. What I do is receive in the data to a 512 byte (actually a 256 word) buffer then when the buffer has received the 256th word I go ahead and write that buffer to the SD card. It works great thus far.

What I did to check for the "hiccups" in writing was to place in my code some "markers" that would produce a pulse on the logic analyzer so that I could monitor the data being read along with how long it took for the SD write routine to complete. What I found was that near the very beginning of writing to the SD card I would get one instance where the write took around 80 - 90 ms. My data receiver would be interrupted but the receive buffer would more than handle that delay and once the interruption was over then everything went back to normal. All the subsequent writes to the SD card only took .435 ms. Very fast. I let it run this program for 8 minutes of data collection which produced an SD file of 128K and it never missed a beat.

Here's my code:

    repeat
      d := \serial1.rx_check
      if d <> -1
        outa[7]~~ 
        buf1[i] := d
        outa[7]~ 
        i++
        repeat
          d := \serial1.rx_check
          if(d == -1)
            serial1.rx_flush  
            quit
          outa[7]~~  
          buf1[i] := d
          outa[7]~ 
          i++
      s := \serial2.rx_check
      if s <> -1
        if (s == -1)
          serial2.rx_flush
          quit
        outa[7]~~  
        buf1[i] := s
        outa[7]~ 
        i++
      if i => 256
        i := 0
        quit
        
    outa[8]~~    
    sd.pwrite(@buf1[0], 512)
    outa[8]~
    wordfill(@buf1, 0, 520)

buf1 is a 520 word buffer. The "markers" I used were P7 & P8 in the code. I would set them before the routine then reset them after.

Don M · 2012-09-27 19:51

Here's a couple screen shots from the logic analyzer. The top line is the Master transmitting, the second line is the Slave responding. The third line is P7. It goes high then low when there is data being written to the buf1 buffer. See the code in the above post. The fourth line is P8. It goes high at the beginning of the SD write routine then goes low when it's done. Also see code in above post.

The first one shows a long hiccup writing to the SD card. This hiccup was about 22ms long. It was the second write from the beginning of logging. You can also see in line 3 right after the SD write is done the 2 blips indicating writing to the buffer buf1. These will normally appear right after the data is valid as in the next 3 to the right of them. This doesn't seem to affect the data collection at all.

The second here shows a typical SD write speed of .435ms. You can see that it barely interrupted the writing of data to buf1 after it completed writing to the SD card.

If there are any improvements that anyone can suggest I'm all ears.

Don M · 2012-09-27 19:57

I had thought about moving the serial data collection function that writes to buffer buf1 into its own cog and then just transfer data when buffer is full and then the SD write routine could be separate and not affect the collection of data. But unfortunately I've had a difficult time understanding cognew and getting it to work.

Don M · 2012-09-27 20:00

Maybe it would be easier to put FSRW into its own cog and then just send it the data when ready. How would you launch FSRW using cognew?

FSRW 2.6 versus upper 32K of EEPROM for storage. Speed compare

Comments