Datalogger and Assembly Trouble (Resolved)
SRLM
Posts: 5,045
I'm trying to write an assembly program to automatically log a set of variables. The trouble is that the datalogger seems to reject the data after a (short) period of repeated logging. The attached code of interest is "Autodatalogger.spin". I included "Autodatalogger_test.spin" since it launches the first program, but doesn't do anything else.
Background:
I'm using the transmit and receive functions from FullDuplexSerialPlus, with my own wrapper. The datalogger is connected to the Prop with two pins (RX and TX) via a pair of series 2Kohm resistors. The CTS pin on the datalogger is tied low with another 2k resistor. RTS is left unconnected. In the code, I'm currently logging 130 preset values in a 'burst'. 96 of the values are preset numbers, 32 are commas, and two are CR and LF. Through trial and error, I've found that I can transmit at 1_000_000 bps, so that's what I set it at.
The Problem:
The code works fine if I open a file, log a single iteration of the data, close the file and end. The problem arises when I try to log multiple iterations of the burst. Inevitably, it will start logging the first few bursts, then the red LED on the datalogger starts to flash (signalling "Commands from monitor port to USB disk"). Once this happens, it never stops and the program doesn't reach it's conclusion.
My attempts:
I tried opening the file, logging a bunch of bursts, then closing the file. On one attempt, I counted the logs and I got 35 iterations.
I tried looping around the sequence of open, write, close so that on each iteration it reopened the file. I got about 7 writes. With a new USB drive I got 42.
These attempts are after I've isolated it down to this.
My Theories:
At first I thought that it was a buffer overflow on the datalogger. Reasonable, since I didn't implement the RTS pin. So, for the tests above, I've slowed it down to one iteration a second. Although I can't find anything in the datasheet about buffer size, I'm pretty sure that it can process a few hundred bytes per second. After all, you can set the baud to 3 million bits per second. But now, I'm lost. I don't have any idea why it works fine for a single open/write/close, but won't work in a loop.
Any help is appreciated.
Thanks,
SRLM
p.s. I think I've found an undocumented requirement. When changing baud rates, the uC is required to wait for a period of time before sending any more commands. Through testing, I found a wait of 1/16th of a second the minimum delay.
Post Edited (SRLM) : 8/30/2009 5:24:17 PM GMT
Background:
I'm using the transmit and receive functions from FullDuplexSerialPlus, with my own wrapper. The datalogger is connected to the Prop with two pins (RX and TX) via a pair of series 2Kohm resistors. The CTS pin on the datalogger is tied low with another 2k resistor. RTS is left unconnected. In the code, I'm currently logging 130 preset values in a 'burst'. 96 of the values are preset numbers, 32 are commas, and two are CR and LF. Through trial and error, I've found that I can transmit at 1_000_000 bps, so that's what I set it at.
The Problem:
The code works fine if I open a file, log a single iteration of the data, close the file and end. The problem arises when I try to log multiple iterations of the burst. Inevitably, it will start logging the first few bursts, then the red LED on the datalogger starts to flash (signalling "Commands from monitor port to USB disk"). Once this happens, it never stops and the program doesn't reach it's conclusion.
My attempts:
I tried opening the file, logging a bunch of bursts, then closing the file. On one attempt, I counted the logs and I got 35 iterations.
I tried looping around the sequence of open, write, close so that on each iteration it reopened the file. I got about 7 writes. With a new USB drive I got 42.
These attempts are after I've isolated it down to this.
My Theories:
At first I thought that it was a buffer overflow on the datalogger. Reasonable, since I didn't implement the RTS pin. So, for the tests above, I've slowed it down to one iteration a second. Although I can't find anything in the datasheet about buffer size, I'm pretty sure that it can process a few hundred bytes per second. After all, you can set the baud to 3 million bits per second. But now, I'm lost. I don't have any idea why it works fine for a single open/write/close, but won't work in a loop.
Any help is appreciated.
Thanks,
SRLM
p.s. I think I've found an undocumented requirement. When changing baud rates, the uC is required to wait for a period of time before sending any more commands. Through testing, I found a wait of 1/16th of a second the minimum delay.
Post Edited (SRLM) : 8/30/2009 5:24:17 PM GMT
Comments
The buffer size in FDX is only 16 bytes, so I would increase this to 256 or 512. I posted a FDX_rr00? in the ZiCog code for the TriBladeProp thread on this page
http://forums.parallax.com/forums/default.aspx?f=25&p=010&m=329999
Grab the zicog_demo_rr097 zip file. Inside this will be FullDuplexSerial v1.2 that I modified to use a constant to define the buffer size. IIRC it is already set to 256 bytes but you can increase this to 512 bytes if you want. Note to only use the sizes given in my comment.
Second, what are you connected to? A 2K series resistor is most likely way too high and will likely cause problems at that speed.
What is the USB drive you are referring to?
I would try and drop the speed down from 1M baud to 115,200 baud. With the volume of data you are receiving as described, this should be fine. Spin code may not be able to handle this speed either, depending on what else you are doing. I can have a look at your code later.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Links to other interesting threads:
· Home of the MultiBladeProps: TriBladeProp, RamBlade, TwinBlade,·SixBlade, website
· Single Board Computer:·3 Propeller ICs·and a·TriBladeProp board (ZiCog Z80 Emulator)
· Prop Tools under Development or Completed (Index)
· Emulators: Micros eg Altair, and Terminals eg VT100 (Index) ZiCog (Z80) , MoCog (6809)
· Search the Propeller forums·(uses advanced Google search)
My cruising website is: ·www.bluemagic.biz·· MultiBladeProp is: www.bluemagic.biz/cluso.htm
A couple of thoughts:
1. I've taken out the buffer in FDSP. Basically, the only part of the code that I've saved are the actual bit banging (transmit and receive) routines. The routines are modified to run straight though on a call, and have a ret. The calling code fills txdata or reads rxdata (the working variables of the routines).
2. The datalogger is a USB host by the same name from Parallax, page here.. It's a 5V device, so I think I need some resistors. From what Carl says here,, I didn't think that the resistors would be a problem. I might have gone too far though (far as in speed, not distance).
3. The object won't be using any spin code beyond startup (the start method and watch declarations, when I get to that point). It's all assembly.
I think I may try lowering the baud and see if that helps. I want it to be as fast as possible (preferably to log up to 88Hz, but I don't think that's likely).
Clock cycles at 57600: 2,812,840
Clock cycles at 230400: 1,242,136
clock cycles at 460800: 976,860
Clock cycles at 921600: 928,820
Clock cycles at 1000000:930,440
@cluso:
I tried the slow speed (115200 baud) and it did about 57 writes out of 200 before it quit.
I ran it at 9600 baud and it made it through all 200 cycles. So then I rewrote the code to use an infinite cycle without a waitcnt and it ran for about 20 minutes before I turned it off.
@TChapman:
I tried the SCS and hex combination today, but it didn't appear to make a change. I was a little surprised to see the exact same data appear in the text file under HEX as under ASCII without making any other changes to my code. I would have thought that it would be morphed somehow, but I don't really know what "input numbers in hex" really means.
So, the problem still stands: the datalogger won't continuously log faster than 9600 bps.
Has anyone logged continuously faster than 9600 bps in UART?
Anyway here's a couple of things , that have come from a recent discovery regarding a couple of errors I made , that may or may not help.
I have used the logger with a Prop and Spin successfully but have trouble at the higher baud rates·, this could possibly be down to using Extended Full Duplex I am not sure if it can handle those rates.
I know/think the logger can because I have used 115200 with the SX ( although not extensively )
First oddity was using IPA I would write $8,$20,$0,$0,$0,$5A,$0D as the start of a write instruction for 90 bytes and it would fail
If instead I wrote $8,$20,$0,$0,$0,DEC 90,$0D it would work
Last thing I wondered was if your #call receive was actually waiting for the ">" prompt , I see this as being crucial.
You may have these items covered but its worth a mention.
Jeff T.
> If instead I wrote $8,$20,$0,$0,$0,DEC 90,$0D it would work
Unsound - that's pretty freaky, if you ask me.
SRLM,
The UART specs don't say what the speed limit is - but 9600 would be old school. 115k shouldn't be a problem (I have one of these loggers, but sorry, I haven't had a chance to mess with it yet.) 1 or 2 meg might be pushing it. (See threads c.a. last month on Viewpoart data transfer speed issues.)
However, the VDAPFirmwareSpec.pdf @
http://www.parallax.com/Portals/0/Downloads/docs/prod/comm/VDAPFirmwareSpec.pdf
specifically says on p.4
" 2.2 UART Interface Configuration
When using the UART interface as the command monitor port the default baud rate used is 9600 baud, although this can be changed while in command mode. "
BTW, The SPI clock speed max is 12MHz
HTH
- Howard
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
According to the firmware sheet on the Parallax website, the UART speed can be up to 3,000,000 baud. I'm trying for about 1/30th of that for now...
I did take into account the default speed of 9600, and my code includes a section to change it to something faster (using the table, of course).
As for the DEC verses hex issue, I don't know what that may be. I'm just transmitting the number directly without encoding it as ASCII. I assume it works since if I change it, I get the "box" in the resulting text file indicative of something that cannot be displayed.
I'm pretty sure that FDS can handle several hundred baud, although I don't have any specs on it.
I rewrote the receive section and put it into a subroutine call "reply" that tests for bad command, command failed, no disk, and unknown reply (anything else). The flashing red light doesn't ever put it there though.
According to this page, the chip has 4k of internal SRAM, which should be enough for this project. It also mentions something more interesting: that the chip is 3.3v with 5v safe inputs. A question:
***
Is it likely that the resistors are causing bad operation at the higher speed?
***
If the chip really is 3.3v, then would it create problems at these higher speeds?
I've also attached my current code.
Edit: I also found this in the datasheet (here, page 24):
Parameter Description Minimum Typical Maximum Units Conditions
Voh Output Voltage High: Vcc-0.4 to 3.6 V, I source = 8mA
Vol Output Voltage Low: 0.4 V, I sink = 8mA
Vin Input Switching Threshold: 0.8 to 1.4 to 2.0 V
It makes me think that the resistors are not needed. True?
Post Edited (SRLM) : 8/30/2009 12:00:33 AM GMT
The FDX object can handle 115200 baud. However, if you cannot take the characters out fast enough you will lose a block (whatever size your FDX buffer is, standard is 16 bytes) because the buffer within FDX will overrun the characters already in it. The 4KB per Cog has nothing to do with your problem.
Below is a modified FDX that has buffer size set to 256. You can also change it to 512 (max) in the CON section.
You will need to isolate the problem in sections. Firstly, try the bigger buffer in the attached FDX. Slow down the sending to 9600 as this will allow mre time to log the data. Next, try skipping some of the logging.
Alternately, try just logging some data, including a count value. Make sure this works.
You have only posted the datalogger code, not the whole code, so we cannot see what you are doing to get the data.
Hope this may help get you a lttle further down the track :-)
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Links to other interesting threads:
· Home of the MultiBladeProps: TriBladeProp, RamBlade, TwinBlade,·SixBlade, website
· Single Board Computer:·3 Propeller ICs·and a·TriBladeProp board (ZiCog Z80 Emulator)
· Prop Tools under Development or Completed (Index)
· Emulators: Micros eg Altair, and Terminals eg VT100 (Index) ZiCog (Z80) , MoCog (6809)
· Search the Propeller forums·(uses advanced Google search)
My cruising website is: ·www.bluemagic.biz·· MultiBladeProp is: www.bluemagic.biz/cluso.htm
1) I've removed everything out of the FDX object except for the intialization and tx/rx routines. There is no longer a buffer. I feed it data via txdata and get data via rxdata (register labels).
2) The program works okay at 9600 bps per second, both with a 1 second write cycle and continuous logging (at the full speed with no waitcnt delays).
3) The program is just logging test data right now, which is 32 numbers per write cycle starting at 0, counting up to 31, comma separated. Aka, each line should be identical to:
4) I've posted everything needed to run the program, except for the PASD (PASM debugger) code. That can be found here.. Alternately, those sections can be commented out.
Note that none of the errors are a result of a bad command, since I set a breakpoint at the "critical error" routine, which is a catch all for replies from the datalogger that isn't the prompt.
So, in summary, the program works fine at 9600 bps, but not at higher baud. At the higher baud, it works fine for a single log or two, but continuous logging results in the flashing led (on the datalogger, indicative of "Commands From Monitor Port to USB Disk").
Edit: I tried working my way up again from 9600, and it works for 19200 baud, but not 38400 (the two steps following 9600 in the datalogger baud table). I'm on my third day of debugging this problem, and the solution still isn't apparent. I like to think that I'm not one of those who has to have "instant gratification", but this is becoming a drag...
Post Edited (SRLM) : 8/30/2009 1:50:11 AM GMT
unless I misunderstand what you're doing, you haven't followed Cluso's suggestion on buffer size - which he's mentioned twice. You can't just get rid of the buffer, or you'll hear Scotty's voice say "It'll never work like that!"
I think Cluso's assessment is spot on: it works at that slow speed because the goes-in and goes-out stuff just barely keeps up - your mem/registers ARE the buffers in that case. But higher up, the bytes just blast over each other - and the lights probably flashing because it's whacking it just enough to think these are commands. Put the buffers back in and make them BIG and FAT ... and let us know what happens.
- H
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
You are receiving serial data in from something on which pin(s)???
You are then using the Autodatalogger.spin program to write this to an SD card on pins 6 & 7 ???
Are you doing this in 1 cog only ???????? If so, this will fail.
You have removed the buffering from FDX - This is wrong, you will require it.
So, you need 3 cogs
* Spin program to handle data between the two worker cogs
* Pasm program running FDX to receive the data to be logged (possibly with bigger buffers)
* Pasm program to write the data out to the datalogger (SD card)
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Links to other interesting threads:
· Home of the MultiBladeProps: TriBladeProp, RamBlade, TwinBlade,·SixBlade, website
· Single Board Computer:·3 Propeller ICs·and a·TriBladeProp board (ZiCog Z80 Emulator)
· Prop Tools under Development or Completed (Index)
· Emulators: Micros eg Altair, and Terminals eg VT100 (Index) ZiCog (Z80) , MoCog (6809)
· Search the Propeller forums·(uses advanced Google search)
My cruising website is: ·www.bluemagic.biz·· MultiBladeProp is: www.bluemagic.biz/cluso.htm
The goal of the program:
The program is a standalone logger of hub variables. A master program will pass it a series of hub addresses, then set it free (aka, ignore it). This program (running in it's own cog as ASM) will copy those values into the cog and log them to the Datalogger (Parallax product, page here). I guess I should make it a capital D in datalogger, since it's a product name. I'll try to do that from now on. Anyway, it will do this continuously, as fast as possible. Eventually, I'd like to get some nice things like optional column titles, a system clock log, and line numbering going.
The current program state:
For now, there are four main (high level) functions: startup, open file, close file, and log a line. The log a line will just log some test data. This data is under the label "value". As can be seen, it's just 32 sequentially increasing numbers. There are two related labels valueaddr and valuedigits. valueaddr will store the hub address of the data, and value digits stores the number of least significant digits for that particular number. It's arranged in vertical columns in the current code.
So, the data to be logged for the testing right now doesn't change, and is stored in it's entirety in the cog.
The cog is transmitting, then receiving a reply, then transmitting, and so on. It's modified not to do both at the same time.
The only serial communication (besides the PASD debugger) is to and from the Datalogger.
I don't see why I need the buffer, since the data is constant. But if it really is necessary, any ideas on how I should add it back in?
Thanks everyone for your help so far. This is very frustrating.
Post Edited (SRLM) : 8/30/2009 2:38:07 AM GMT
You have to have a buffer because flowing data is never constant. Startup delays, slew rates, cycle time delays, etc etc. Depending on the app, buffers can be small or huge. The greater the differential, the more buffering you need.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Links to other interesting threads:
· Home of the MultiBladeProps: TriBladeProp, RamBlade, TwinBlade,·SixBlade, website
· Single Board Computer:·3 Propeller ICs·and a·TriBladeProp board (ZiCog Z80 Emulator)
· Prop Tools under Development or Completed (Index)
· Emulators: Micros eg Altair, and Terminals eg VT100 (Index) ZiCog (Z80) , MoCog (6809)
· Search the Propeller forums·(uses advanced Google search)
My cruising website is: ·www.bluemagic.biz·· MultiBladeProp is: www.bluemagic.biz/cluso.htm
As for the earlier statement, I'd like to have this running in the background. Using a serial routine straight out of the box will require two cogs: one for the serial, one for the spin.
The data is constant, embedded in the cog. Eventually that will change, but right now I only access the hub during initialization (pin number, baud rates, and all that).
I'm fairly sure that the actual low level serial routines are fairly solid. I did something very similar with a GPS program. In that program, I took the receive routine from FDX and made some wrappers to parse the various GPS strings. There wasn't a buffer in there, and it received okay at 57600 baud.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Links to other interesting threads:
· Home of the MultiBladeProps: TriBladeProp, RamBlade, TwinBlade,·SixBlade, website
· Single Board Computer:·3 Propeller ICs·and a·TriBladeProp board (ZiCog Z80 Emulator)
· Prop Tools under Development or Completed (Index)
· Emulators: Micros eg Altair, and Terminals eg VT100 (Index) ZiCog (Z80) , MoCog (6809)
· Search the Propeller forums·(uses advanced Google search)
My cruising website is: ·www.bluemagic.biz·· MultiBladeProp is: www.bluemagic.biz/cluso.htm
If it looks like there is activity on RTS, the flow control could be a simple addition to the byte level tx routine. Test the flow control input from the datalogger before sending each byte.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Tracy Allen
www.emesystems.com
I believe Tracey is right , I think the only way for larger packets is to use the CTS/RTS .
I attatch the spin file if its of interest , I did use Cluso99's file with the larger buffer but it did not help in this case. I also had a small debug routine looking for the prompt which I don't think was affecting things.
Note: the firmware was reconfigured 115200 baud , IPA , SCS , flow control=none
Jeff T
The top speed that I've got to log so far is 460_800 baud. I tried 921_600, but something is snagging. Maybe after some more work...
Some preliminary testing shows that the logging slows down after a while. For a specific USB stick, logging initially started at about 24 samples per second (that's 137 bytes per sample). After about 1/2 hour, it has dropped to 10 samples per second (in a few second period). With a different stick, I got a start of 44 sps, and I'm testing long term right now.
Another nice feature that I noticed is that I can turn off the system and remove the USB drive at any time, without losing data (except for possibly the last entry). It seems very robust.
Anyway, thanks for all your help! It's really appreciated.
-- You wouldn't think that CTS has to be tied low in order for the datalogger to accept data, because the datalogger does not normally send anything back until it receives the number of characters specified when the file was opened. However, if CTS is not low (ready) the datalogger will pause after about 25 or 30 characters and will wait until CTS does go low. Why? I don't know. That is an issue if you move to a simple half duplex implementation with flow control.. In a half duplex implementation with flow control, CTS would normally stay high (not ready) untill it hits the rx code. That is to say, the datalogger really expects to be working with a full duplex host. That would require PRop to ping-pong between rx and tx as it does in full-duplex-serial.
-- Another issue comes up for receiving lots of bytes (more that one 512 byte sector at a time). If the processor needs any dead time between bytes to go off an do anything, then CTS needs to be brought high (not ready) by the end of the start bit of the current byte. Any later and the datalogger will queue up and transmit another byte after the current one. That only applies for long reads greater than one sector, for example, when using the "RD file" command to read an entire log file.. I discovered that weirdness in trying to use the RD file command with the BASIC Stamp, which sets the flow control to not-ready a little too late in the current byte, so the Stamp would end up receiving every other character upon crossing a sector boundary. Unrelated to baud rate. Anyway, in making your Prop handle CTS, set it high just after the start bit
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Tracy Allen
www.emesystems.com
Post Edited (Tracy Allen) : 8/30/2009 5:55:35 PM GMT
thanks for posting the struggle - it will help others out later (like me [noparse]:)[/noparse]
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
I found out that opening and closing the file is costly in time. Opening and closing on every write about halves the sample rate to start. After about 20 minutes of continuous logging, the sample rate dropped to 1/8th of the full speed. I moved the open and closing out of the loop (so that they're only done once each), and the sample rate jumped up to about 74 samples per second! That's 137 bytes per sample, or about 10,000 bytes per second. It's about 1/4 of the possible bytes per second are data. I have a small concern that at the end of 90,000 samples the rate has dropped by about 3 samples per second (to 71), but at that rate it will take several hours to drop to slow speed.
As for not implementing the CTS (or RTS? I get confused which side we're talking about. I'm talking about ignoring the Propeller ready to receive) the program seems to work okay without it so far. With this program, I won't be reading any data from the datalogger, and so the only information that I receive is the replies. I base this assumption on the test where I ran it for 90,000 samples, and it didn't hang up or crash.
When I get the code working the way I want it to, I'll be posting it (along with other, related code). It should be nice, since there aren't any (in my opinion) good Datalogger programs available in the obex.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Tracy Allen
www.emesystems.com
RE the close - how critical is the data?
On the off-chance of a glitch (power fail, reboot, UFO flyby [noparse]:)[/noparse], the data that's not closed could be lost. Closes routinely flush a disk's caches (temp. buffers) so that the write to disk actually burns.
- H
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
That answers a question I was going to ask , look forward to seeing the completed project.
Jeff T.
Didn't think about the flow control :-(
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Links to other interesting threads:
· Home of the MultiBladeProps: TriBladeProp, RamBlade, TwinBlade,·SixBlade, website
· Single Board Computer:·3 Propeller ICs·and a·TriBladeProp board (ZiCog Z80 Emulator)
· Prop Tools under Development or Completed (Index)
· Emulators: Micros eg Altair, and Terminals eg VT100 (Index) ZiCog (Z80) , MoCog (6809)
· Search the Propeller forums·(uses advanced Google search)
My cruising website is: ·www.bluemagic.biz·· MultiBladeProp is: www.bluemagic.biz/cluso.htm