Memory Stick Datalogger question

rixter

Hi,

I am trying to read text data from a thumb drive using the Parallax Memory Stick Datalogger. Working from the sample code segments, I've tried to use the RDF command in a simple loop to read in 3 "records" of 5 bytes of a TXT file on the thumb drive. See code below. If I perform one read out of a loop, it appears to work, but successive iterations in a loop cause unexpected and inconsistent results. What I keep getting is the drive letter as part of the data read (e.g RickyD:\>) as in:

---

Data:RickyD:\>
\>


Data[noparse]:D[/noparse]:\>
\>
\>


Data:

MarD:\>
\>

---

I have not managed to find decent documentation on this device for the syntax of the file commands. The sample code isn't a very good practical example of this device's use in my opinion. I am not sure if I am supposed to be accounting for CR and LF that are throwing the "alignment" off or what is going on. Any hints or advice appreciated.

Thanks,
Rick

---

TXT file contents:
Ricky
Marty
Sally

Program segment:

' Read data
FOR counter=1 TO 3
PAUSE 120
SEROUT TX\CTS, Baud, [noparse][[/noparse]"RDF ", $00, $00, $00, $05, CR]
SERIN RX\RTS, Baud, 500, No_Data2, [noparse][[/noparse]STR buffer\5]
DEBUG "Data:", STR buffer, CR ' Display stored data value
NEXT

Mike Green

Read the manual from FTDI (www.vinculum.com/documents/fwspecs/Vinculum%20Firmware%20User%20Manual%20V2.4%20Rev%203.pdf). I believe the "D:\>" is a normal response when one command has completed and is sent as a prompt for the next command which would be your next read command.

Tracy Allen

The command buffer needs to be purged after the command, to wait for the > prompt and discard it. The simplest purge is,

SERIN vrx\vrts, vbaud, vtimeout, v_tko, [noparse][[/noparse]wait(">")]

If your file contains a CRLF after each entry, the above line will clear out those too. I'm not quite sure what is going on in your example data. Note that in the DEBUG command, the STR buffer will send characters until it encounters a null in the string.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Tracy Allen
www.emesystems.com

Unsoundcode

Hi rixter, I have attached a txt file I wrote that explains some of the Logger commands. The file uses sub routines to build a menu structure for testing the VDAP commands.

A couple of important notes, the commands use the shortened command set which makes the serial data easier to manage. Secondly there is extensive use of the line SERIN RX, Baud,[noparse][[/noparse]WAIT (">")], this line of code is used to wait for an acknowledge signal from the Logger, unfortunately it does not work well on the BS2 at 9600 so you would have to either reduce the baud rate to 4800 or replace the troublesome lines with a short pause. It works well enough to demonstrate most of the commands though. Lastly the CTS RTS have been omitted and will need inserting.

Even if you don't use whats in the txt file you might find the FILESIZE and SEEK routines useful.

Jeff T.

Tracy Allen

That's a great tutorial Jeff. I printed out a copy to keep with my reference materials on VDAP.

I think the CTS and RTS are pretty important for smooth operation, especially the flow control (vRTS) that tells the Logger that the Stamp is ready to receive data. One peculiarity of the flow control is that the Stamp has to set vRTS low (ready) when transmitting data to the logger. Otherwise the data may be lost. That vRTS line is normally held high by the Stamp until it ready to receive as part of a SERIN vRX\vRTS,... command, so your program has to explicity LOW vRTS before exectuting a SEROUT vTX\vCTS, ... command. It is not a problem AFAIK when the string transmitted is less than 25 characters, but causes loss of data for longer strings. I talked with engineering staff at FTDI about this, and it comes from the assumption on their part that the device will be talking to a fully buffered serial port that will be able to receive characters at the same time that it transmits characters.

They update the Vinculum firmware at www.vinculum.com quite often, and the release notes detail a long list of bug fixes and improvements. When you buy from a distributer such as Parallax or Mouser, you are likely to receive firmware a couple of releases back. A lot has to do with proper detection of different brands of thumb drives. One relatively recent change had to do with the disk scan at startup. Previously it scanned the entire disk for free blocks at startup, but in the recent revision it skips that and starts up MUCH faster. It does not count free blocks until commanded to do so.

It is quite easy to update the firmware, simply by downloading the bootloader from Vinculum.com, changing the name as instructed, and putting it on a thumb drive that you then insert in the Logger interface and it detects the upgrade and painlessly applies it.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Tracy Allen
www.emesystems.com

Unsoundcode

Thank you Tracy. When I first started tinkering with the VDrive (DataLogger) there was a lot less information available than there is now, I went down the road of keeping CTS permenantly low and thats where most of my personal docs are at. Since that time Chris Savage coded a great example using CTS/RTS with a routine to clear the buffer. I havn't fooled with Chris's code much as what I have satisfies my needs but I agree with your comments and take on board what you are saying.

Jeff T

rixter

Thank you all. This is wonderful stuff. In looking further into my code that I didn't display in my first note, I noticed that I wasn't flushing out stuff hanging in 'buffer' as I was using that variable array in a dual purpose fashion. The sample code from Parallax uses the "Get_Serial_Bytes" subroutine extensively to get return data after SEROUT commands. That subroutine uses 'buffer'. I think if I clear that variable out (or use another), before my reads, I should be ok. I could also do byte reads. My goal is to read a decent sized TXT file into EEPROM on a BS2pe to use in my project. Baud rate and byte read efficiency are not an issue as I am only going to read a TXT file in (overwriting EEPROM), very infrequently. I was originally interested in grabbing 20 byte chunks, which would be lines in my TXT file and dumping them to eeprom. But I suppose I could loop single byte reads 20 times and get the same result. But then again I might run into pointer issues. It's probably easier to grab the chunks terminated with CR LF.

In any case, you all have given me plenty of places to find what I need.

Thanks much!

RIck

Chris Savage

We have always had every intention of releasing further examples as time permits. As Tracy pointed out there have been a few firmware updates since we first started carrying the Vinculum chip. Once I get some time I will try to come up with a better example used in an appropriate application. Hopefully I will be able to make use of most of the functions. Until then it is nice to see customers providing their own examples.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Chris Savage
Parallax Tech Support

rixter

Thanks Chris. The key was two-fold... clearing the buffer and realizing that my TXT file is terminated on every line by an invisible CR LF. So this code below (using the sample code's purge subroutine), works when reading 12 lines of a TXT file ("records"), with a fixed length fo 20 characters. Basically, I specified 22 characters (the $16), in the SEROUT "RDF" command to input the 20 real data characters plus the CR LF, then the SERIN command only gets 20 of them, thus dropping the CR LF. The purge subroutine each iteration keeps the buffer clear so I don't get garbage in my input data. Interestingly, in the sample code, most SEROUT commands call the Get_Serial_Bytes subroutine that puts input data into an array (buffer) that never seems to get used elsewhere in the sample code.

I would think that writing and reading multiple records to/from a thumb drive is what most users of this device would be interested in doing. The sample code uses the simplest example of writing one fixed length number and reading it back. The multiple record write sample isn't complimented with a multi-record read. I am also doing the relatively easy fixed length record retrieval, but with text data. While not space efficient (records shorter than 20 being padded with spaces), it simplifies the reads I believe and I'll have the room in eeprom. Otherwise I'd have to implement a record terminator character and do byte reads until I encounter it. Without a firmware update, this thing is reading about two 20 byte records per second, so inputting 200 records into eeprom will take over a minute and a half not including the WRITE commands to eeprom. I'm not bringing in data often, so that is acceptable. I thought this the best way to get a decent chuck of textual data into eeprom.

PAUSE 120
FOR counter=1 TO 12
GOSUB purge
SEROUT TX\CTS, Baud, [noparse][[/noparse]"RDF ", $00, $00, $00, $16, CR]
SERIN RX\RTS, Baud, 500, No_Data2, [noparse][[/noparse]STR result\20]
DEBUG "Data:", STR result, CR ' Display stored data value (WRITE to eeprom code will go here eventually)
<

---

GOSUB purge also works placed here
NEXT


Rick

Chris Savage

Rick,

I’m glad you got it figured out. I will try to put more attention into posts regarding the Datalogger. I know it is not easy to understand some of the things that are going on for some people. Myself, I had a chance to see some code written for it when we first got them. I have since re-written and re-formatted that code, however I also created my own to go with it on the product page. I have two applications in mind which will make use of the device and anything I make is posted along with complete code, so eventually that will be posted as well. At least for now several other members are on the ball and familiar with the device, so…Anyway, take care.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Chris Savage
Parallax Tech Support

Tracy Allen

I'm attaching a terminal interface between the BASIC Stamp and the vDrive or Parallax USB Datalogger. When power is applied, it initializes the drive, displays upgrade and disk information, and then goes into a terminal mode where it accepts keyboard commands and displays the responses from the vDrive, by way of PBASIC. For example, if the user types FWV after the prompt, the drive responds with the firmware version on the debug screen. This is convenient for exploring the vDrive command set.

I make provision to power the vDrive from a separate voltage regulator (LT1763) that can be controlled off and on from PBASIC. Two reasons. First is to save power in low data rate weather logging. Second is to allow reboot of the drive and recovery from lockups. In the program, typing ^C will reboot the drive. The program will also work as intended if both the Stamp and the vDrive are powered up at the same time, but in that case it can't be rebooted on command.

One perplexing problem I still have with the vDrive and PBASIC is in reading long files. For example, if there is a file called "CAPTURE.TXT", it would be convenient to read the whole thing at once using the command,
RD CAPTURE.TXT
That could make Rick's application much more code-efficient. However, I have found that if the file is longer than a about 500 bytes, what I get from that command is every other character in the file, up to near the end, where it starts to act normally and display every character. The same thing happens with the command,
OPR CAPTURE.TXT ' open for read
RDF N ' read N characters from the file
That works fine so long as the number N after the RDF is less than about 500. (512, a sector?). Longer than that and it sometimes drops into the every other character mode. Still trying to get to the bottom of the anomaly. It can be worked around by keeping N<500, but one wonders why. I am wondering if others would see the same anomaly. I'm using Vinculum firmware version 03.63VDAPB.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Tracy Allen
www.emesystems.com

Tracy Allen

A followup. I hooked it up to a 'scope, and it is pretty easy to see what is going on at the times when the Stamp is losing every other character in long read operations. When the Stamp is ready at a SERIN to receive a byte, the flow control line goes low briefly (5 bit periods, which is a little over 500 microseconds at 9600 baud). The vDrive normally sends one character in response to that flow control, and then waits for the next instance to send the next byte. However, when misbehaving, the vDrive sends two bytes instead of 1, and the Stamp misses the second one. This is easily visualized on the 'scope, where each flow control pulse is accompanied by two bytes returned head to tail from the vDrive, and then there is a little wait, another flow control pulse from the Stamp, and two more bytes from the vDrive.

I emphasize that this only happens when the number of bytes requested in a RDF N command is N>=512, or when an RD command asks to capture an entire file. It definitely has to do with sector boundaries in a curiously systematic manner. The result depends on where in a sector the read begins and ends. I'll probably pose the issue to tech support at FTDI.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Tracy Allen
www.emesystems.com

Unsoundcode

Hi Tracy, using the prompt ( > ) as software flow control works quite securely on files >500 bytes reading in 10 byte or greater chunks and serves as a good alternative until the hardware flow control issue is resolved. The only drawback is as I mentioned earlier that anything slower than a BS2px has to run at a slower baud rate. It also appears at first glance that software flow control is more economical on code and Stamp pins wouldn't you agree, does this economy compromise the data it seems fairly secure to me.

Jeff T.

Tracy Allen

Rick, by software flow control, do you mean waiting for the ">" response after every command? OKay, but when errors occur, the ">" is not returned. This can be handled with a timeout, as you have done in a couple of instances, but it needs more coverage I think. It depends on how flexible the interface has to be and the likelihood of an error. An elaborate handler could put all the characters in a buffer and then parse them to detect alternative outcomes, but the added timeouts and error handling make the software flow control much less code efficient. (Often the case, error and exception handling ruins a beautiful program!)

It _is_ indeed attractive to have an interface that uses only rx and tx, instead of the four io's required when rts and cts are also in the mix. But as you pointed out, the baud rate has to be choked back in order to do that. Using the hardware flow control provided by the vDrive does give a patina of certainty to the integrity of the data. The overall throughput is not much better than 9600 baud, but with the flow control I can operate the vDrive at 38400 baud without losing anything. (Except for the anomoly I mentioned above). The vDrive hardware flow control does operate well on a byte by byte basis.

One interesting thing I have found is that there is really no effect in closing a file that has been opened for read. Say you open "myfile.txt" for read, and you read and seek to locations in that file, but if you then close the file, you can still read and seek to new positions in the file just as if the close command had no effect. There is no error. The open and close are more strict of course when a file is opened for write.

Rick, in your tutorial SEEK routine, I don't think there is any need to seek to the position in the file repeatedly. Reads are naturally consecutive in the file.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Tracy Allen
www.emesystems.com

Unsoundcode

Tracy Allen said... I don't think there is any need to seek to the position in the file repeatedly. Reads are naturally consecutive in the file.

Interesting and absolutely right, I removed the seek instruction with the effect of an increase in speed and a reduction of code. I attach a sample that repeatedly reads a file (the file must be named test.txt) at 2400

Yes I am calling the prompt response flow control although that may not strictly be true. If time allows I intend to play around with CTS and RTS and see if I can improve the way I am doing things.

Jeff T.

Tracy Allen

Here is a followup on using the RD command to read an entire file, or RDF to read more than 512 bytes at a time.

The Stamp flow control pin for SERIN vrx/vrts, $54, ... ... holds the vrts pin low for 4.5 bit periods, 469 microseconds at 9600 baud. In response to that pulse duration, the vDrive or memory stick datalogger sometimes as detailed above sends two characters for every one, as though it has already queued up the second character before the Stamp returns the vrts line to the high, "not ready" state. To test this, I rigged it up in hardware to shorten the duration of the ready pulses. Sure enough, when the vrts low time was shorter than about 100 microseconds, the RD and RDF commands acted correctly, that is, one character per control pulse. I tested it at down to 2 microseconds, and it still worked correctly. (one bit period=104 microseconds at 9600 baud.) I think the lesson there for the SX and the Propeller is to keep the vrts pulse less than one bit period. (I haven't tried it at other baud rates, but from other tests, I believe it is tied to the bit period rather than to absolute time.) For the Stamp, we can't change the behavior of the SERIN flow duration without adding external hardware. I've sent a question about this to FTDI support, to see if they might consider a firmware adjustment on their side.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Tracy Allen
www.emesystems.com

SRLM

I found that the easiest thing to do was to take the demo code available and cut it down to suit, then add my own functionality. I dedicated a whole program slot to the datalogger, and simply used the sub routines provided by the demo code. After i figured out that the character count is the fourth data point, I have not had any problem with the setup to record GPS waypoints and data.

sam_sam_sam

SRLM

By any chance could you post the code that you came up with

This would a great help to me if you could· Thanks

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
··Thanks for any··that you may have and all of your time finding them

·
·
·
·
Sam