Speed up data transfer to Memory Stick Datalogger (#27937)

Pascal P

Hello,

I have a little application in which I collect a set of data (hour, time, second, and button ID) every time one of four button·is pressed, temporarily stored the data in a 1Mb EEPROM, and at the end of the experiment transfer the data on a memory stick using the Memory Stick Datalogger #27937.
It is working BUT the writting onto the memory stick is quick lengthy, i.e. about 2.5 minutes to transfer 250 data points (hence about 1,000 bytes). From your experience, is this a normal rate or is there a way to speed things up? I'm not looking at anything lightning fast but I will eventually have about 1,000 data points (4,000 bytes) and I would like if I could reduce the download time from 10 minutes down to around 1 minute. Note I have a BS2p24 microcontroller.

Thank you very much for any suggestion you may have.

Regards,
Pascal

FYI, I developed my code around the templates provided by the Memory Stick Datalogger.
Here is a copy of the part that does the transfer from the EEPROM to the memory stick. It essentially collects the 4 bytes from the EEPROM that correspond to a particular entry, write it onto the memory stick and then move on to the next 4 bytes until all the data points are transfered:

FOR Ctr_2 = 1 TO Ctr_A + Ctr_B + Ctr_C + Ctr_D
··· ' Read data from EEPROM
··· IF Ctr_2 < 8192 THEN
····· EE_Cmd = 160
····· EE_Addr = Ctr_2 * 8
··· ELSE
····· EE_Cmd· = 168
····· EE_Addr = (Ctr_2 - 8192) * 8
··· ENDIF
··· FOR Ctr_3 = 0 TO 3
····· EE_Addr = EE_Addr + Ctr_3
····· GOSUB I2C_Stop······················· ' Send Start sequence to device
····· GOSUB I2C_Start······················· ' Send Start sequence to device
····· PAUSE 5
····· SHIFTOUT Mem_Data, Mem_Clk, MSBFIRST, [noparse][[/noparse]EE_Cmd]· ' Send Write Command to device
····· SHIFTIN Mem_Data, Mem_Clk, MSBPRE, [noparse][[/noparse]i2cAck\1]·· ' Get Acknowledge Bit
····· SHIFTOUT Mem_Data, Mem_Clk, MSBFIRST, [noparse][[/noparse]EE_Addr.HIGHBYTE]' Send Address to device
····· SHIFTIN Mem_Data, Mem_Clk, MSBPRE, [noparse][[/noparse]i2cAck\1]·· ' Get Acknowledge Bit
····· SHIFTOUT Mem_Data, Mem_Clk, MSBFIRST, [noparse][[/noparse]EE_Addr.LOWBYTE]' Send Address to device
····· SHIFTIN Mem_Data, Mem_Clk, MSBPRE, [noparse][[/noparse]i2cAck\1]·· ' Get Acknowledge Bit
····· GOSUB I2C_Start································ ' Send Start sequence to device
····· SHIFTOUT Mem_Data, Mem_Clk, MSBFIRST, [noparse][[/noparse]EE_Cmd+1]' Send Read command to device
····· SHIFTIN Mem_Data, Mem_Clk, MSBPRE, [noparse][[/noparse]i2cAck\1]·· ' Get Acknowledge Bit
····· SHIFTIN Mem_Data, Mem_Clk, LSBFIRST, [noparse][[/noparse]EE_Value] ' Get Data from device
····· GOSUB I2C_Stop························ ' Send Stop sequence to device
····· 'DEBUG "Address: ", DEC (EE_Addr + Ctr_3), "; Value: ",HEX2 EE_Value, CR
····· IF Ctr_3 = 0 THEN
······· Hours = EE_Value
······· ELSEIF Ctr_3 = 1 THEN
········· Mins = EE_Value
······· ELSEIF Ctr_3 = 2 THEN
········· Secs = EE_Value
······· ELSE
········· Evt_Typ = EE_Value
······· ENDIF
······ 'WRITE DATA TO file
··· EE_Addr = EE_Addr - Ctr_3
··· NEXT
··· 'DEBUG "Record: ",HEX2 Hours,":", HEX2 Mins,":", HEX2 Secs,"-", DEC1 Evt_Typ, CR, CR
··· PAUSE 50
··· SEROUT TX\CTS, DtlgBaud, [noparse][[/noparse]"WRF ", $00, $00, $00, $B, CR,
······················· HEX2 Hours, ",",HEX2 Mins,",",HEX2 Secs,",",DEC1 Evt_Typ, CR]
··· GOSUB Get_Serial_Bytes
··· PAUSE 50
· NEXT

SRLM

Can you clean up the code a little bit and post it in it's entirety as an attachment? Specifically, I'm looking for comments lined up on the right and comments for blocks of code that gives an idea of what it's supposed to be doing.

As for the general questions, I've used the Datalogger at the ~400K baud speed, and it works fine. From the BS manual, you should be able to use the 115K speed to log the data to the datalogger. However, to do that you'll need to send the new speed to the datalogger at the old speed (9600), wait a bit, then switch to the new speed. This should dramatically (10x) speed up your data transfer.

Mike Green

I estimate that you can transfer one byte from the EEPROM in less than 100ms with your code and the 4 bytes of a data point translates to about 30 bytes written to the datalogger which should take less than 200ms to write with your pauses and the response from the datalogger. That's about what you're getting.

I'd try leaving out the PAUSE 50 statements in your code and reading in all 4 bytes of the data point with one I2C transaction. You'd need to increment the EEPROM address by 4 and repeat the SHIFTINs to read the bytes. Then you don't need the FOR/NEXT loop to get the 4 bytes and you don't need the IF/THEN. Use

      SHIFTIN Mem_Data, Mem_Clk, MSBPRE, [noparse][[/noparse]i2cAck\1]   ' Get Acknowledge Bit
      SHIFTIN Mem_Data, Mem_Clk, LSBFIRST, [noparse][[/noparse]Hours] ' Get Data from device
      SHIFTIN Mem_Data, Mem_Clk, MSBPRE, [noparse][[/noparse]i2cAck\1]   ' Get Acknowledge Bit
      SHIFTIN Mem_Data, Mem_Clk, LSBFIRST, [noparse][[/noparse]Mins] ' Get Data from device
      SHIFTIN Mem_Data, Mem_Clk, MSBPRE, [noparse][[/noparse]i2cAck\1]   ' Get Acknowledge Bit
      SHIFTIN Mem_Data, Mem_Clk, LSBFIRST, [noparse][[/noparse]Secs] ' Get Data from device
      SHIFTIN Mem_Data, Mem_Clk, MSBPRE, [noparse][[/noparse]i2cAck\1]   ' Get Acknowledge Bit
      SHIFTIN Mem_Data, Mem_Clk, LSBFIRST, [noparse][[/noparse]Evt_Typ] ' Get Data from device

I wouldn't run the datalogger faster than 9600 Baud. The BS2 can't receive faster than that and I don't think it can reliably send much faster than 19200 Baud.

Post Edited (Mike Green) : 9/10/2009 4:25:11 PM GMT

Mike Green

If you have a BS2p Stamp, you can use the I2CIN / I2COUT statements rather than the SHIFTIN / SHIFTOUT statements and that will speed up the transfer from EEPROM considerably. Look at the chapters in the Stamp Manual for details and there's a Nuts and Volts Column (look at the index under Parallax's Resources tab on their main webpage) that gives examples of the use of an EEPROM with these statements.

Pascal P

Wow... you guys are amazing. You're responding faster than my memory stick is transfering data [noparse]:)[/noparse]
Thank you. I just changed·from a BS2 to a BS2p recently and I was wondering about this I2CIN/I2COUT I noticed in the book.
I will try that tonight along with cleaning up the code as you suggested.

Now the reason why I used a loop for reading from the EEPROM is that I haven't been able to make "page read" work. I can do a random read, a random write, a page write, but·for·a reason I still don't understand I can't·make·page read work. And I've had the same problem with·both a 16kb and 1Mb EEPROM. I am careful about not trying to read across·the physical boundary of the EEPROM map but it just doesn't work. The first byte·is read properly but the·following ones are empty. That's why I eventually used the work around of doing 4 random reads intead of 1·page read of 4 consecutives bytes but now it's biting me back in terms of time.· But I will try again with the code you suggest tonight and hopefully it will work. Thank you again,

Regards,
Pascal

Pascal P

Alright, so I tried to use the IC2IN/IC2OUT but I can't make the I2CIN command·work.

I read the documentation, was careful to connect to pin 0 and 1 and the IC2OUT seems to work but the I2CIN doesn't. To check it wasn't a wiring problem I wrote the following code where I write the value 1 in address 0 of a 24LC16B and then try to read it using IC2IN and SHIFTIN. SHIFTIN reads the right value, i.e. 1, but I2CIN reads 255. I tried at different address in the memory, with different values, nothing worked. I noticed someone else had a similar post·but they used PIN 14 & 15 which probably was there problem.

Any suggestions?

Cheers,

Pascal

·' {$STAMP BS2p}
' {$PBASIC 2.5}

M_Data PIN 0 ' I2C Serial Data Line
M_Clk· PIN 1 ' I2C Serial Clock Line

i2cin_result VAR·· Byte···· ' result from I2CIN
i2cAck······ VAR·· Bit····· ' Ack Bit From Device
shift_result VAR·· Byte···· ' result from shiftin

I2COUT 0,$A0, 0, [noparse][[/noparse]1]

'Read with I2CIN command
I2CIN 0, $A1, 0, [noparse][[/noparse]i2cin_result]
DEBUG CR, "I2CIN read: ",DEC i2cin_result, CR, CR

'Read with SHIFTIN/SHIFTOUT commands
· GOSUB I2C_Start
· SHIFTOUT M_Data, M_Clk, MSBFIRST, [noparse][[/noparse]$A0]········ ' Send "write" command to EEPROM
· SHIFTIN· M_Data, M_Clk, MSBPRE,·· [noparse][[/noparse]i2cAck\1]
· SHIFTOUT M_Data, M_Clk, MSBFIRST, [noparse][[/noparse]0]·········· ' Send address to EEPROM
· SHIFTIN· M_Data, M_Clk, MSBPRE,·· [noparse][[/noparse]i2cAck\1]
· GOSUB I2C_Start
· SHIFTOUT M_Data, M_Clk, MSBFIRST, [noparse][[/noparse]$A1]········ ' Send "read" command to EEPROM
· SHIFTIN· M_Data, M_Clk, MSBPRE,·· [noparse][[/noparse]i2cAck\1]
· SHIFTIN· M_Data, M_Clk, LSBFIRST, [noparse][[/noparse]shift_result] ' Read data from EEPROM
· GOSUB I2C_Stop
· DEBUG CR, "Value stored is: ",DEC3 shift_result,CR,CR

END

I2C_Stop: ' I2C Stop Bit Sequence
· LOW·· M_Data
· INPUT M_Clk
· INPUT M_Data ' SDA --> High While SCL High
RETURN

I2C_Start: ' I2C Start Bit Sequence
· INPUT M_Data
· INPUT M_Clk
· LOW·· M_Data ' SDA --> Low While SCL High
RETURN

Mike Green

A 1MB EEPROM requires a two byte address in addition to modifying the EEPROM address if you're accessing the 2nd 64K of the EEPROM. To read a byte from the EEPROM, you need:

I2CIN 0,$A1,address>>8\address&$FF,[noparse][[/noparse]i2cDataToRead]

To write a byte to the EEPROM, you need:

I2COUT 0,$A0,address>>8\address&$FF,[noparse][[/noparse]i2cDataToWrite]
PAUSE 5 ' allow the EEPROM to finish writing

If you're using the 2nd half of the EEPROM, you'd use addresses $A2 and $A3. You can compute this:

I2CIN 0,i2cPage<<1+$A1, ...

where i2cPage is the 64K page number to use.

Post Edited (Mike Green) : 9/13/2009 9:42:51 PM GMT

Mike Green

You can also read and write multiple bytes in the same statement. For reading, you can read as many bytes as you wish as long as they're all in a 64K "page" of the EEPROM. For writing, the bytes all have to fall in a 128 or 256 byte multiple of addresses depending on the specific model and manufacturer of the EEPROM. If the bytes reside in adjacent groups of 128 or 256 bytes, the address will wraparound to the start of the group of bytes rather than continuing to increment the way you'd normally expect.

Pascal P

Thank you. The problem was the PAUSE. I added one and as long as I put a 3 ms or more PAUSE, it's reading the memory just fine. I think I'm back in business. And thank you too about the word size addressing for the 1MB. I had gone back to the 16K memory while trying to figure out why my I2CIN didn't work but I'll switch back to the big one now that it's working.
Thank you so much and have a great week.
Pascal

Pascal P

Hello,

This is just to let you know I finally got around re-writting my code using IC2IN/OUT instead of SHIFTIN/OUT (as well as removing all the unnecessary PAUSE) and it did speed things up.

From 2.5 minutes to write about·about 250 4-byte data points on a .csv file, I am down to about 22 seconds.

I thought I would share in case anyone is working on a similar project.

And for reference, the 45-lines of codes shown at the beginning of this post are now reduced to the following 9:

·FOR Ctr = 1 TO Ctr_A + Ctr_B + Ctr_C + Ctr_D
··· ' Read data from EEPROM
··· M_Addr_W = Ctr * 4 + 4
··· I2CIN Mem_Data, 161,M_Addr_W.HIGHBYTE\M_Addr_W.LOWBYTE, [noparse][[/noparse]STR Time\3, Evt_Typ]
··· ' Write data to Datalogger
··· SEROUT TX\CTS, DtlgBaud, [noparse][[/noparse]"WRF ", $00, $00, $00, $B, CR,
······················· HEX2 Time(0), ",",HEX2 Time(1),",",HEX2 Time(2),",",DEC1 Evt_Typ, CR]
··· GOSUB Get_Serial_Bytes
· NEXT

This probably is just a coincidence but I just notice while typing this that reducing the number of line of codes by 5 sped up the process by 5.

Anyway, I'm happy and I just want to thank you guys again for the help you provide. I finds this·a great forum and Parallax provides great products.

Cheers, Pascal