SD Card or VDRIVE1 Data Logging

Comments

• Bean Posts: 8,129

  2006-12-19 22:28 edited 2006-12-19 22:28

  How would storing 30 million values only use 0.5MB of storage ?
  Even if you stored pure binary it would take 30M * 12 / 8 = 45 MB of storage.
  Storing text would take at least 4*30M = 120MB
  
  Just how fast do you need to store the data ?
  My SD data logger would work, but it's only 19.2Kbaud and only allows 16MB per file and 32MB per media.
  
  Bean.
  
  ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
  Cheap used 4-digit LED display with driver IC·www.hc4led.com
  
  Low power SD Data Logger www.sddatalogger.com
  SX-Video Display Modules www.sxvm.com
  Stuff I'm selling on ebay http://search.ebay.com/_W0QQsassZhittconsultingQQhtZ-1
  
  "USA Today has come out with a new survey - apparently, three out of every four people make up 75% of the population." - David Letterman
• crgwbr Posts: 614

  2006-12-20 12:52 edited 2006-12-20 12:52

  Sorry, did my math wrong; I hate when that happens. Anyway, your SD Data Logger looks really nice, but the slower baud rate and 32 mb limit looks like it might not work. I am really leaning towards the Vinculum device and a big flash drive. Just to avoid reinventing the wheel, I'll ask; has anyone implemented the Vinculum already. If not, guess I'm on my own.
  
  Thanks,
  crgwbr
  
  ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
  NerdMaster
  For
  Life
• Bean Posts: 8,129

  2006-12-20 13:32 edited 2006-12-20 13:32

  Can you devulge more details ?
  We may be able to make other suggestions...
  
  Bean.
  
  ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
  Cheap used 4-digit LED display with driver IC·www.hc4led.com
  
  Low power SD Data Logger www.sddatalogger.com
  SX-Video Display Modules www.sxvm.com
  Stuff I'm selling on ebay http://search.ebay.com/_W0QQsassZhittconsultingQQhtZ-1
  
  "USA Today has come out with a new survey - apparently, three out of every four people make up 75% of the population." - David Letterman
• crgwbr Posts: 614

  2006-12-21 00:37 edited 2006-12-21 00:37

  I don't want to devulge much, because it's of a competitive nature. All I can really say is that, this thing needs to run its full program loop about 1,000,000 times per second; therefore, the speed of USB is very welcome. I'll basically be getting serial data from one source, interperating it, then storing it in the flash drive for later use. I would like to use the SPI interface of the vinculum, probably hosed in the VDRIVE1 module. I hope this is enough info at least for a few tips from the people who have already implemented the vinculum.
  
  Thanks,
  crgwbr
  
  P.S. Is it too unsafe to run an SX48 at 100 MHz or must I stick to 75 MHz. What about with a heatsink?
  
  ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
  NerdMaster
  For
  Life
• JDOhio Posts: 72

  2006-12-21 02:53 edited 2006-12-21 02:53

  I have been using the Rogue Robotics MMC SD Reader/Writer. I have been able to read and write at the default (9600 baud) using an SX-48 running at 20 MHz. I don't have any other experimental information yet. I expect to experiment with the top baud rate of the MMC which is 115,200 baud.
  
  Joe
• PJMonty Posts: 983

  2006-12-21 19:13 edited 2006-12-21 19:13

  crgwbr,
  
  Does your full program loop store all 30 million values? If so, (and assuming you store the 12 bit values in 16 bits words), then you'll need a bandwidth of 30 million value * 1 million times a second * 2 bytes per value = 60 million bytes per second. If this is the case, you won't get that sort of speed with a RAID array of hard drives, let alone a flash memory card.
  
  Thanks,
  PeterM
• crgwbr Posts: 614

  2006-12-21 22:00 edited 2006-12-21 22:00

  PJMonty, thankfully for the both of us, the full program loop stores either 2 or 4 values (user selectable). In your math, I also beleive you made an error; 30 million * 1 million * 2 byte = 60 Trillion bytes per second. However, at my requerment it should be; 2 or 4 * 1 million * 2 bytes = 4 or 8 million bytes per second. Possible? I don't know; what do you think?
  
  ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
  NerdMaster
  For
  Life
• PJMonty Posts: 983

  2006-12-22 18:42 edited 2006-12-22 18:42

  crgwbr,
  
  You're right about my math error. Your actual speed requirement is much more sane. However,
  since you have 12 bit values, I am assuming you'll store those in 16 bit words. If so, 2 values = 4 bytes, and 8 value = 16 bytes, which means your bandwidth is actually either 8 or 16 million bytes per second. This is still mighty fast, and I doubt flash cards can handle anywhere near that speed. The read speed on flash cards is quite fast, but the write speed is pretty pokey.
  

  Thanks, PeterM
• crgwbr Posts: 614

  2006-12-22 18:58 edited 2006-12-22 18:58

  You are probably correct of the write speed of flash cards. However, my want of 2 or 4 values, at 2 bytes each, means 4 or 8 bytes. 4 or 8 bytes at 1 million times per second = 4 to 8 million bytes per second; not 8 or 16. I am still looking into using a VDRIVE1 device and Flash Drive instead of an SD flash card. My hope is that this will allow for faster data transfer. I am also going to resize my hopes a little bit; I'll go for 500 KHz at 2 values or 1 MHz at 2 values. This reduces the speed to a maximum of 2 MBps.
  
  ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
  NerdMaster
  For
  Life
• PJMonty Posts: 983

  2006-12-22 19:08 edited 2006-12-22 19:08

  Crgwbr,
  
  In your original post, you wrote, "I have a project that needs to log a list of 30,000,000 numbers with values ranging from 0-4096 (12 bit)". You have to remember that 12 bits do not fit into a byte, since a byte only has 8 bits. In your previous two posts, you use the word "value", which I assume to mean the 12 bit value referred to in your original post.
  
  If you want to write 2 million values which use 12 bits, under no circumstances do you have 2 million bytes. The smallest byte based unit you can store 12 bits inside of is a word, which has 16 bits. Since 16 bits = 2 bytes, then two million values = 4 million bytes, and 4 million values = 8 million bytes.
  
  In Bean's original reply, he assumed the most efficient approach where you actually wrote 12 bit binary values to the flash memory. Even if you do this, you'll still have 50% more bandwidth required since each 12 bit value = 1 1/2 bytes. Hence, you will require a 3 to 6 million byte per second bandwidth.
  
  Thanks,
  PeterM
• Lord Steve Posts: 206

  2006-12-23 19:05 edited 2006-12-23 19:05

  Use multiple SD or MMC cards in parallel and split the bandwidth requirements across them.
• crgwbr Posts: 614

  2006-12-26 14:58 edited 2006-12-26 14:58

  I think I've figured it out.· Ram and External Ram is super fast, but it's volitale.· Therefore, I save the values in an external RAM (a big one); then after that's done, I copy the vlues over from the RAM to an SD card of Flash drive at a much more leisurely pace.
  
  Would it work?
  
  crgwbr
  
  PS: Lord Steve, also a very good idea for the Propeller, however it wouldn't work for the SX.
  
  ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
  NerdMaster
  For
  Life