Reducing write time of USB Data Logger with BS2p

Engineer09

I'm using the Vinculum USB memory stick data logger to log force data to a flash drive. I need to reduce the write time.

In my device, data from a load cell connected to an ADC is stored in EEPROM. After each run, the program reads the stored data from EEPROM, processes it from raw form to calibrated form , and writes it to the flash drive.

As the program stands, it works flawlessly (when using the suggested flash drive). However, I need to reduce the time it takes to write each file. I log data at a rate of 10 Hz and each run lasts up to several minutes (1-5). So I have 1000-3000 data items to write. Ideally, I would be able to write all of these items to a file in 30 seconds or less.

When I try reducing the pause command (in the provided example code that I heavily relied upon), the writing process freezes. I also tried using a higher baud rate, but again the writing process freezes.

How can I reduce the write time? I've looked at the Vinculum spec sheet and understand what the commands mean, but the timing of the writing process is not so clear. Is there a minimum period between write commands? Can a higher baud rate reduce the write time? Also, I am not so clear on the section of code related to timeout.

I have attached a copy of the relevant code. The other functions (like writing of data to EEPROM) take place in other program slots.

Any help or advice is greatly appreciated. Thanks!

Chris Savage

I will have a closer look at this as soon as I can set something up on my bench. I find it hard to believe you could read data from the EEPROM and write it to the USB drive faster than it can accept data from a BASIC Stamp.

Engineer09

Hi Chris,
I appreciate that. Specifically, the lines in the attached code that I am interested in are: 172 (the pause duration after the data item is written), 53 (setting the baud rate at which data is sent to the logger), and 250 (the duration for timeout). With a 100ms pause after each data item, I get successful writes each time. When I reduce it to 25ms, the drive freezes. I know from other posts that others have trouble at first understanding the Vinculum data logger but I have become familiar with the commands from the spec sheet. As you can see from the attached file above, I stamp each file with either a name or using the current month-date-hour-minute. I am interested in simply getting the data written faster. If need be, I will have to reduce the sampling rate, but it would very nice to keep it at 10Hz. Thanks again.

Tracy Allen

It should be possible and preferable to go directly from the line that writes data (ln 169) to the purge routine without any delay at all. I would set the RTS line HIGH (the not-ready-to-receive state) immediately after the command that writes data (or even before). Otherwise, it seems to me, there is danger of missing the response from the drive. The drive generally finishes its write within a few milliseconds, that is, unless it needs to cross a sector boundary or otherwise fiddle with the directory, in which event it can take longer.

By the way, does your setup in fact implement RTS and CTS in the hardware? If not, it comes down to hard coded delays, which have to be coded for worst case.

The purge routine Get_Data as written (ln 247) will always wait at least 100ms even if the drive responds sooner. It might be better to have the purge routine look for the normal ">" prompt that signals ready for a new command. A timeout means it has either not responded at all, or that it has thrown an error. In either case all you can do is bail out to the synchronization routine or something like that. A simpler purge routine would be,

v_Purge:
  DO
    SERIN vrx\vrts, vbaud, 512, v_tko, [char]
  LOOP UNTIL char=">"
  RETURN

v_tko:
  ' bail out to reset, resync

Other thoughts:
Would it be possible to stream the data directly to the USB in real time, instead of going to eeprom first?

Or, if the intermediate data is in eeprom, could the program stream the entire data set out in one long write command instead of one write command for each record? Say you have 1000 records of 13 bytes each, then tell the write command to expect 13000 bytes and roll them out as fast as possible. The CTS pin will control any potential bottleneck (unlikely with the Stamp!).

Engineer09

Tracy,
First, thank you for the reply.
I'm not sure what you mean by implementing RTS and CTS in the hardware. Can you please clarify?
I will try the suggested software revisions and let post with the result.
I did consider writing directly to the USB in real time rather than to EEPROM but I abandoned that idea early on. First, I assumed writing to EEPROM would be faster than writing to the drive. Since I wanted to collect data at 10Hz or higher, I figured the best option would be write to EEPROM, then do a data dump at the end. Also, I did not upload the rest of the program, but as it stands, the program slots where data is read in from the load cell are nearly full. There wouldn't be any way to add the code for logging data to the drive in the same slot and I did not want to jump back and forth between slots for every data item. I preferred only transitioning among data slots as infrequently as possible.

Tracy Allen

By implemented in hardware I mean that there are in fact wire connections to RTS and CTS as defined by the following pins. Sometimes people skimp and use only RX and TX.

RTS             PIN     8               ' Request To Send --> 27937.6 (CTS)
RX              PIN     9              ' Receive Data    <-- 27937.5 (TXD)
TX              PIN     10               ' Transmit Data   --> 27937.4 (RXD)
CTS             PIN     11              ' Clear To Send   <-- 27937.2 (RTS)

Once the drive is opened for write, it stays open until it receives the number of characters specified. Those do not have to be all contained in one single SEROUT.

If you know the length of a run in advance, say 2 minutes at 10 per second, 1200 samples at 13 bytes each, then after the file is open for write, the output to the drive becomes 1200 iterations of the SEROUT command:

SEROUT TX\CTS, Baud, [DEC3 ((element_number+2)/20),".",DEC1 (((element_number+2)//20)/2), ",", DEC4 calibrated_force_data_to_write, CR, LF]

There are no intervening purge sequences, which are eating up a lot of your time as it stands.

I see that the data_processing routine requires memory space and time too.

Engineer09

Thanks for the clarification: yes, I do have connections to RTS and CTS.
And yes, the number of items to be written is always known. Basically, since each value written is a Word in size, the number of items is equal to the ending EEPROM address divided by 2. So I can see that I could just put the SEROUT command in a loop and have it loop until all the items are written. I'll try the suggested revisions in the morning and post the result. Thanks for the advice.

Engineer09

And yes, processing the data is going to slow the write time down (if several thousand items are being written). I might move the processing routine to the other slot so the raw data is processed before being saved to EEPROM. That way, data will be processed as it is coming in (feasible since the sample rate is only 10Hz).

Engineer09

I've tried removing the purge step after each SEROUT command, but I'm not getting any data to be written. The files get created, but they are blank. I've attached the modified code. Also, in the line of code that you provide, why did you modify the SEROUT command?

Engineer09

Also, I am looking at the SEROUT command that I have:
SEROUT TX\CTS, Baud, [$08, $20, $00,$00,$00,$0D, CR,DEC3 ((element_number+2)/20),".",DEC1 (((element_number+2)//20)/2), ",", DEC4 calibrated_force_data_to_write, CR, LF, CR]
I specify 13 bytes for writing: $0D (13). So I count toward this byte count: carriage return, 3 decimal number, period,1 decimal number, comma, 4 decimal number, carriage return, and the line feed. The final CR does not get counted and is required according to the spec sheet. Does it seem that I cam counting this correctly?

Engineer09

Also, looking at table 4.4 of the vinculum data logger spec sheet, I see a table for different baud rates. So am I understanding this correctly: the three bytes that are set to specify a particular baud rate, are these 3 bytes the first 3 of the 4 that go into the write command (see table 4.1 on file operations)?

Unsoundcode

Hi, the first "CR" is actually part of the command sequence and should not be counted, therefore number of bytes to count should be 12 ( $0C ).

To change baud rate has its own command ( $14, $20,< divisor (3 bytes) LSB first >,$0D ).

My personal preference is to download the Vinculum configuration utility from the FTDI site and preconfigure whether to use ECS/SCS and baud. It cuts out a little code and can be set to boot quicker than default.

Jeff T.


EDIT: Link to Vinculum Firmware Customizer http://www.ftdichip.com/Support/Utilities.htm

Engineer09

Hi Jeff,
Thanks for the reply. I'm not so sure that the first CR should be counted: Here is a quote from a post from Mike Green on this subject:
"The first CR/LF is considered to be data for writing to the file. It's included in the size value. When you read the file from the memory stick, that CR/LF will be there and acts as a delimiter for the variable value. The last CR indicates the end of the command to the Datalogger." (http://forums.parallax.com/showthread.php?133993-Couple-of-Memorystick-questions&highlight=memory+stick+datalogger). You posted on that thread I believe. Are you absolutely sure that the first CR does not count toward the byte count? How about the last CR?

Would you be able to post any example code you have involving the data logger?

Unsoundcode

Well you can look this over http://forums.parallax.com/showthread.php?112872-Datalogger-BS2-or-SX-(-115200-)

There is no implementation of cts/rts but it has a few good points in there.

Your previous code -->> SEROUT TX\CTS, Baud, [$08, $20, $00,$00,$00,$0D, CR,

the above is all a part of the instruction including the first "CR", this should not be counted, the remaining bytes to count total 12 the last CR is also part of the instruction and not counted

Jeff T.

EDIT: using most Stamps the baud has to be kept to 4800, this is one drawback of not using cts/rts, using the BS2px it is possible to run at a higher baud

Tracy Allen

With regard to removing the get_data, that is not what I meant. It still has to be there, but after all the data has been written. It is probably best if I formulate a stripped down example. I'll post that later.

With regard to the number of bytes:

SEROUT TX\CTS, Baud, [$08, $20, $00,$00,$00,$0D, CR,DEC3 datum1 ,".",DEC1 datum2, ",", DEC4 datum3, CR, LF, CR]

There will be 13 bytes written to the drive, starting with the DEC3 after the first CR. The 13 bytes will be include the final CR LF CR sequence. If you only want CR LF then make the number of bytes 12 ($0C).

A final CR is not in my experience and reading of the Vinculum docs required as part of the syntax. There does not have to be a CR there to close the command. The command is closed as soon as the specified number of bytes have been transmitted, period, and the drive responds a few milliseconds later with the prompt ">". If there is an extra CR, no harm done, the monitor will enter that in the command buffer and treat it as though a CR had been entered in response to the prompt, and issue a second ">". The perils are, 1) entering too few characters, in which case the monitor will sit there forever waiting for the rest of them, or on the other hand 2) entering too many characters, in which case the characters followed by a CR are treated as a command, usually returning a "bad command" errror, but also having the potential to really screw things up.

Tracy Allen

Back to the way to avoid so many purges. Here is a framework to send 1000 records of 12 bytes each to the drive, using only one purge. An initial SEROUT tells the monitor how many bytes to expect. Be absolutely sure that is correct. Here is it $2EE0=12000 bytes. Then the loop starts that sends exactly that many, the data only, no framing. Finally comes one single purge with the possibility of a timeout.

file_to_vinculum
  LOW RTS         <--needed due to a peculiarity of the vinculum firmware
   SEROUT TX\CTS, Baud, [$08, $20, $00,$00,$2E,$E0, CR]  <-- specifies 12000 bytes $1EE0 = 1200
   FOR index = 0 to 999   ' 1000 records of 12 bytes each
        ' get the data here, note that this does not include the preamble.  Data only!!!
        SEROUT TX\CTS, Baud, [DEC3 datum1 ,".",DEC1 datum2, ",", DEC4 datum3, CR, LF]
   NEXT
   HIGH RTS  ' hold off the vinculum from sending its response
purge:
  DO
      SERIN RX\RTS Baud, 512, timeout,[char]  ' may have to fiddle with the 512 ms timeout
  LOOP UNTIL char=">"
  STOP

timeout:
  ' 1) try another CR  or two and wait for response.
  ' 2) send lots of e and E character and wait for response (usually means the count of bytes was short)
  ' 3) reset the drive by cycling its power, try again.
  ' 4) use the version of the purge that processes the incoming  characters to see if there is an error message.

Like Jeff points out, you can set a higher baud rate with its own command or in the setup software. On the BS2 there are diminishing returns above 9600 baud though, simply because of the interpreted nature of PBASIC. On the BS2p, though, it might indeed trim the time substantially.

Unsoundcode

I was unaware that the final CR of the WRF command was not required and my first inclination was to argue the point, but I knew better, so I ran some tests with a faster micro (Propeller) to determine exactly what was going on with the response from the data logger.

The syntax is clearly stated

‘WRF’<sp><size in hex(4 bytes MSB first)><cr><data bytes of size><cr> RESPONDS <prompt>$0D

The description of response is inaccurate.

The data bytes are framed inside two CR’s and the data bytes of size must equate to size in hex , data bytes are the only thing that get written to the flash drive.

If the instruction is written as above then by the time the instruction has completed the data logger will have responded with three prompts (“>”)

The first prompt is invoked as a response to the first CR, the second is acknowledgment that all data bytes have been received and the third is in response to the final CR. Omitting the final CR does not appear to compromise the WRF and will reduce the number of prompts returned by one.

Being that the data logger can so easily lock up with erroneous instructions this information is crucial to know when writing a logger application, more so when not using cts/rts as flow control.

I feel better informed, thank you Tracy for pointing this out.

Jeff T.

Engineer09

I just had the chance to test Tracy's sample code. And it does work. Big thanks to Tracy. I have several kloc of code in this embedded system: I was able to get everything else to work as desired but the logger was by far the most troublesome.

One thing I did change:
As Tracy suggested, the write time gets greatly reduced by specifying the number of bytes to write to the file and omitting the purge step with each item. We know the number of data items and each item takes 12 bytes (due to formatting, including the time etc). But the number of data items to be written changes each time - it isn't constant. Sometimes it may be 10, sometimes 3000. Two ways to deal with this: 1. After the data items are written to EEPROM, fill the remaining memory locations with zeroes. That way the total number of bytes being written is always the same. This would make the time needed to log data after each run constant. Or 2. modify the command that tells the logger how many bytes to expect by making it a function of the number of items logged. That is what I did:

number_of_bytes = number_of_elements*12
SEROUT TX\CTS, Baud, [$08, $20, $00, $00,number_of_bytes.HIGHBYTE, number_of_bytes.LOWBYTE, CR]

I am getting very nice write times (around 15sec for 1 minute of data, this includes opening/creating both files), so the timing will not scale linearly (so probably around 20-25sec for 2min of data). I'll see how much I can reduce the write time (diff. baud rates etc) and post the optimized code shortly.

Again, thanks Tracy and Jeff.

Tracy Allen

I'm happy that is working out and will be interested to hear your followup. The drive does have peculiarities, but it is great once all that is ironed out. The suspend options (SUM and SUD) also work well if you need to reduce battery drain.

Engineer09

I wanted to share the final code for the data logger. I'm able to log around 3000 items in under 30 seconds. Code shows how to specify the number of items to log as well as changing the baud rate to 19200bps with the BS2p. Raw data is acquired from an ADC at 10Hz, processed, and stored to EEPROM. I use three program slots of the BS2p for nothing but logging data. Data is then read from EEPROM and logged to the drive.

Also, bear in mind that due to the 8.3 file name format that the logger uses, the file names can only be up to 8 bytes long.

Also, I recommend using the DS1302 real time keeping chip as a source to time stamp your files. A convenient system that allows for minute resolution while staying within the 8 byte limit is: month(00)date(00)hour(00)minute(00), with the hour being 0-24 (the DS1302 being configured for military mode).

Lastly, not all flash drives work. I encountered many that did not.
I had great luck with the Transcend JF V30 1GB.