Reading DEX files with propeller. Need some help with code.
Don M
Posts: 1,654
I am working on a project to read a DEX file from a Vending Machine Controller (VMC). DEX files contain audit data for transactions in a vending machine. It utilizes a 9600 baud 8 bit data link. Nothing fancy. The data is standard ASCII characters and contains line feeds and carriage returns for formatting.
I have tried to condense as much as possible my code to read in the data. Maybe too much
The data stream has a max limit of 245 characters. The messages start with a $10 (DLE) $02 (STX) ... ASCII DATA... $10 (DLE) $17 (ETB) then a 2 byte CRC-16 checksum The data block is then ACK'd by the VMC with a DLE 1 ($10 $31) the first time then a DLE 0 then next time then back to a DLE 1. The ACK alternates between DLE 1 and DLE 0 starting with a DLE 1.
My problem lies beginning in this section. See below:
It sends the ACK's all in a row without waiting for the data block to be received in. I think it has somethng to do with my logic of looking for a DLE ($10) since it is in the beginning and end of a block.
Also I don't know what happens to the end DLE ETB and 2 byte checksum.
Here's the full code I worite. Don't laugh as I am vey much a novice at this yet.
I have tried to condense as much as possible my code to read in the data. Maybe too much
The data stream has a max limit of 245 characters. The messages start with a $10 (DLE) $02 (STX) ... ASCII DATA... $10 (DLE) $17 (ETB) then a 2 byte CRC-16 checksum The data block is then ACK'd by the VMC with a DLE 1 ($10 $31) the first time then a DLE 0 then next time then back to a DLE 1. The ACK alternates between DLE 1 and DLE 0 starting with a DLE 1.
My problem lies beginning in this section. See below:
inx := 1 ' initialize index counter to 1
i := dex.rxtime(20) ' receive in dex data and assign it to variable i
if i == $10 ' check to see if byte is DLE
i := dex.rx ' receive in next byte
if i == $02 ' check to see if next byte is STX. if so start repeat loop
repeat until i == $10 ' do repeat until byte received is DLE then quit
i := dex.rx ' receive in nexy byte and assign to variable i
dexdata[inx] := i ' assign received byte i to dexdata variable indexed by index counter inx
term.tx(dexdata[inx]) ' send data byte to terminal for printing
inx += 1 ' increment index counter
dex.tx($10) ' send ACK - DLE 1
dex.tx($31) ' ACK's alternate between DLE 0 and DLE 1
inx := 1 ' repeat process again for next batch of data
i := dex.rxtime(20)
if i == $10
i := dex.rx
if i == $02
repeat until i == $10
i := dex.rx
dexdata[inx] := i
term.tx(dexdata[inx])
inx += 1
dex.tx($10) ' send ACK - DLE 0
dex.tx($30)
inx := 1 ' repeat process again for next batch of data
i := dex.rxtime(250)
if i == $10
i := dex.rx
if i == $02
repeat until i == $10
i := dex.rx
dexdata[inx] := i
term.tx(dexdata[inx])
inx += 1
dex.tx($10)
dex.tx($31)
It sends the ACK's all in a row without waiting for the data block to be received in. I think it has somethng to do with my logic of looking for a DLE ($10) since it is in the beginning and end of a block.
Also I don't know what happens to the end DLE ETB and 2 byte checksum.
Here's the full code I worite. Don't laugh as I am vey much a novice at this yet.
'' =================================================================================================
''
'' File....... DEX_Prop.spin
'' Purpose.... DEX Reader
'' Author..... Don Merchant
'' -- see below for terms of use
'' E-mail.....
'' Started.... 06 SEP 2011
'' Updated.... 08 SEP 2011 This version excluded PST control and prints data in ASCII format
''
'' =================================================================================================
con
_clkmode = xtal1 + pll16x ' Feedback and PLL multiplier
_xinfreq = 5_000_000 ' External oscillator = 5 MHz
MS_001 = 80_000_000 / 1_000 ' ticks in 1ms
DEX_TX = 0 ' DEX -> VMC
DEX_RX = 1 ' VMC -> DEX
obj
dex : "fullduplexserialplus"
term : "fullduplexserialplus"
var
byte dexstrng[24], dexdata[245]
pub main | i, d, inx
term.start(31, 30, %0000, 115_200) ' start terminal (use PST)
dex.start(1, 0, %0000, 9_600) ' start DEX read
pause (1000) ' pause for PST to start
dex.tx($05) ' send ENQ to start
term.str(string("Start DEX_Prop Ver 0.2", 13))
i := dex.rxtime(100) ' look for response from VMC
if i == $10 ' if DLE received from VMC start handshake
i := dex.rx
if i == $30
term.str(string("Start 1st Handshake", 13))
dex.tx($10) ' send DLE
pause(15)
dex.tx($01) ' send SOH
pause(15)
dex.str(string("1234567890RR01L01")) ' send string
pause(15)
dex.tx($10) ' send DLE
pause(15)
dex.tx($03) ' send ETX
pause(15)
dex.tx($DE) ' send upper byte of CRC checksum
pause(15)
dex.tx($4D) ' send lower byte of CRC checksum
pause(15)
else
term.str(string(2, 1, 3, "No response")) ' If no DLE received from VMC then indicate no communication
i := dex.rx ' In this section we look for a DLE response from the VMC to end first handshake
if i == $15
term.str(string(2, 1, 5, "NAK")) ' if checksum isn't right let us know
else
if i == $10
i := dex.rx
if i == $31
term.str(string("First Handshake Successful", 13))
dex.tx($04) ' send EOT
else
term.str(string(2, 1, 5, "Didn't recognize DLE..."))
i := dex.rx
if i == $05 ' look for ENQ
term.str(string("Start 2nd Handshake"))
dex.tx($10) ' acknowledge the ENQ by sending DLE 0
dex.tx($30)
else
term.str(string(2, 1, 7, "Didn't recognize ENQ..."))
pause(15)
inx := 1 ' receive id data from VMC but do nothing with it
repeat 24
i := dex.rxtime(40)
dexstrng[inx] := i
'term.tx(dexstrng[inx])
inx += 1
term.tx(13)
dex.tx($10)
dex.tx($31)
i := dex.rxtime(100) ' look for response from VMC
if i == $04 ' if DLE received from VMC start handshake
term.str(string("Second Handshake Successful", 13))
i := dex.rx
if i == $05
term.str(string("Begin DEX dump from VMC...", 13))
dex.tx($10)
dex.tx($30)
inx := 1 ' initialize index counter to 1
i := dex.rxtime(20) ' receive in dex data and assign it to variable i
if i == $10 ' check to see if byte is DLE
i := dex.rx ' receive in next byte
if i == $02 ' check to see if next byte is STX. if so start repeat loop
repeat until i == $10 ' do repeat until byte received is DLE then quit
i := dex.rx ' receive in nexy byte and assign to variable i
dexdata[inx] := i ' assign received byte i to dexdata variable indexed by index counter inx
term.tx(dexdata[inx]) ' send data byte to terminal for printing
inx += 1 ' increment index counter
dex.tx($10) ' send ACK - DLE 1
dex.tx($31) ' ACK's alternate between DLE 0 and DLE 1
inx := 1 ' repeat process again for next batch of data
i := dex.rxtime(20)
if i == $10
i := dex.rx
if i == $02
repeat until i == $10
i := dex.rx
dexdata[inx] := i
term.tx(dexdata[inx])
inx += 1
dex.tx($10) ' send ACK - DLE 0
dex.tx($30)
inx := 1 ' repeat process again for next batch of data
i := dex.rxtime(250)
if i == $10
i := dex.rx
if i == $02
repeat until i == $10
i := dex.rx
dexdata[inx] := i
term.tx(dexdata[inx])
inx += 1
dex.tx($10)
dex.tx($31)
inx := 1
i := dex.rxtime(250)
if i == $10
i := dex.rx
if i == $02
repeat until i == $10
i := dex.rx
dexdata[inx] := i
term.tx(dexdata[inx])
inx += 1
dex.tx($10)
dex.tx($30)
inx := 1
i := dex.rxtime(250)
if i == $10
i := dex.rx
if i == $02
repeat until i == $10
i := dex.rx
dexdata[inx] := i
term.tx(dexdata[inx])
inx += 1
dex.tx($10)
dex.tx($31)
inx := 1
i := dex.rxtime(250)
if i == $10
i := dex.rx
if i == $02
repeat until i == $10
i := dex.rx
dexdata[inx] := i
term.tx(dexdata[inx])
inx += 1
dex.tx($10)
dex.tx($30)
inx := 1
i := dex.rxtime(250)
if i == $10
i := dex.rx
if i == $02
repeat until i == $10
i := dex.rx
dexdata[inx] := i
term.tx(dexdata[inx])
inx += 1
term.tx(13)
dex.tx($10)
dex.tx($31)
i := dex.rx ' look for EOT to end routine
if i == $04
term.str(string(" DEX finished"))
pub pause(ms) | t
'' Delays program for ms milliseconds
t := cnt
repeat ms
waitcnt(t += MS_001)
Comments
I don't have time to look over your code but I do wonder if FullDuplexSerial could be causing you trouble because of its small buffer.
I posted a modified version of Tim Moore's four port driver with a 512 byte rx buffer. You should be able to safely receive the whole message and then manipulate it after it has all arrived.
Post #9 of the thread is a modification I frequently use. The driver monitors for an end of message flag so the parent object knows when a complete message has arrived.
Duane
'' ================================================================================================= '' '' File....... DEX_Prop.spin '' Purpose.... DEX Reader '' Author..... Don Merchant '' -- see below for terms of use '' E-mail..... '' Started.... 06 SEP 2011 '' Updated.... 08 SEP 2011 This version excluded PST control and prints data in ASCII format '' '' ================================================================================================= con _clkmode = xtal1 + pll16x ' Feedback and PLL multiplier _xinfreq = 5_000_000 ' External oscillator = 5 MHz MS_001 = 80_000_000 / 1_000 ' ticks in 1ms DEX_TX = 0 ' DEX -> VMC DEX_RX = 1 ' VMC -> DEX '' Parallax Serial Terminal '' Control Character Constants ''------------------------------------- '' CS = 16 ''CS: Clear Screen '' CE = 11 ''CE: Clear to End of line '' CB = 12 ''CB: Clear lines Below '' HM = 1 ''HM: HoMe cursor '' PC = 2 ''PC: Position Cursor in x,y '' PX = 14 ''PX: Position cursor in X '' PY = 15 ''PY: Position cursor in Y '' NL = 13 ''NL: New Line '' LF = 10 ''LF: Line Feed '' ML = 3 ''ML: Move cursor Left '' MR = 4 ''MR: Move cursor Right '' MU = 5 ''MU: Move cursor Up '' MD = 6 ''MD: Move cursor Down '' TB = 9 ''TB: TaB '' BS = 8 ''BS: BackSpace '' BP = 7 ''BP: BeeP speaker obj dex : "fullduplexserialplus" term : "fullduplexserialplus" var byte dexstrng[24], dexdata[245] pub main | i, inx term.start(31, 30, %0000, 115_200) ' start terminal (use PST) dex.start(1, 0, %0000, 9_600) ' start DEX read pause (1000) 'term.tx(16) ' repeat dex.tx($05) ' send ENQ to start term.str(string("Start DEX_Prop Ver 0.2", 13)) i := dex.rxtime(100) ' look for response from VMC if i == $10 ' if DLE received from VMC start handshake i := dex.rx if i == $30 term.str(string("Start 1st Handshake", 13)) dex.tx($10) ' send DLE pause(15) dex.tx($01) ' send SOH pause(15) dex.str(string("1234567890RR01L01")) ' send string pause(15) dex.tx($10) ' send DLE pause(15) dex.tx($03) ' send ETX pause(15) dex.tx($DE) ' send upper byte of CRC checksum pause(15) dex.tx($4D) ' send lower byte of CRC checksum pause(15) else term.str(string(2, 1, 3, "No response")) ' If no DLE received from VMC then indicate no communication i := dex.rx ' In this section we look for a DLE response from the VMC to end first handshake if i == $15 term.str(string(2, 1, 5, "NAK")) ' if checksum isn't right let us know else if i == $10 i := dex.rx if i == $31 term.str(string("First Handshake Successful", 13)) dex.tx($04) ' send EOT else term.str(string(2, 1, 5, "Didn't recognize DLE...")) i := dex.rx if i == $05 ' look for ENQ term.str(string("Start 2nd Handshake")) dex.tx($10) ' acknowledge the ENQ by sending DLE 0 dex.tx($30) else term.str(string(2, 1, 7, "Didn't recognize ENQ...")) pause(15) inx := 1 repeat 24 i := dex.rxtime(40) dexstrng[inx] := i 'term.tx(dexstrng[inx]) inx += 1 term.tx(13) dex.tx($10) dex.tx($31) i := dex.rxtime(100) ' look for response from VMC if i == $04 ' if DLE received from VMC start handshake term.str(string("Second Handshake Successful", 13)) i := dex.rx if i == $05 term.str(string("Begin DEX dump from VMC...", 13)) dex.tx($10) dex.tx($30) inx := 1 ' Routine needs to look for DLE ETB ($10, $17) before checksum repeat 239 ' and then determine whether to send a $10 $30 or $10 $31 to i := dex.rxtime(250) ' start next block of data and to set how long the repeat goes dexdata[inx] := i 'term.hex(dexdata[inx], 2) ' If it finds a DLE ETX ($10, $03) then it should signal end of DEX transmission term.tx(dexdata[inx]) 'term.tx($20) inx += 1 'term.tx($0D) 'term.tx($0D) dex.tx($10) dex.tx($31) inx := 1 repeat 236 i := dex.rxtime(250) dexdata[inx] := i 'term.hex(dexdata[inx], 2) term.tx(dexdata[inx]) 'term.tx($20) inx += 1 'term.tx($0D) 'term.tx($0D) dex.tx($10) dex.tx($30) inx := 1 repeat 236 i := dex.rxtime(250) dexdata[inx] := i 'term.hex(dexdata[inx], 2) term.tx(dexdata[inx]) 'term.tx($20) inx += 1 'term.tx($0D) 'term.tx($0D) dex.tx($10) dex.tx($31) inx := 1 repeat 236 i := dex.rxtime(250) dexdata[inx] := i 'term.hex(dexdata[inx], 2) term.tx(dexdata[inx]) 'term.tx($20) inx += 1 'term.tx($0D) 'term.tx($0D) dex.tx($10) dex.tx($30) inx := 1 repeat 236 i := dex.rxtime(250) dexdata[inx] := i 'term.hex(dexdata[inx], 2) term.tx(dexdata[inx]) 'term.tx($20) inx += 1 'term.tx($0D) 'term.tx($0D) dex.tx($10) dex.tx($31) inx := 1 repeat 236 i := dex.rxtime(250) dexdata[inx] := i 'term.hex(dexdata[inx], 2) term.tx(dexdata[inx]) 'term.tx($20) inx += 1 'term.tx($0D) 'term.tx($0D) dex.tx($10) dex.tx($30) inx := 1 repeat 46 i := dex.rxtime(250) dexdata[inx] := i 'term.hex(dexdata[inx], 2) term.tx(dexdata[inx]) 'term.tx($20) inx += 1 term.tx(13) 'term.tx($0D) dex.tx($10) dex.tx($31) i := dex.rx if i == $04 term.str(string(" DEX finished")) pub pause(ms) | t '' Delays program for ms milliseconds t := cnt repeat ms waitcnt(t += MS_001)I personally would do something like this (completely untested):
... repeat if receive(@dexstr, ACK_token) ' ALternate ACK-token s needed ' Do something with the data received else ' Error recovery actions ... PUB receive(inx, token) | checksum, dle_rec, start '' Receive data from DEX source, Keep looping until data received '' '' Returns TRUE when valid data received '' Returns FALSE when checksum failure or too much data received start := inx repeat ' keep looping until we receive data repeat until dex.rx == $10 ' Wait for DLE if dex.rx == $02 ' Check that next data is STX repeat 245 ' Check for max size i := dex.rx case i $10: ' DLE dle_rec := i ' Save received byte $17: ' ETB if dle_rec ' Did we receive DLE before? checksum := dex.rx << 8 ' Get CRC-16 checksum checksum += dex.rx if crc-16(start, inx-start, checksum) ' Check if received data is OK term.str(string("Data OK",13)) ' Send ACK dex.tx($10) ' Send DLE dex.tx(token) ' Send ACK token return true else term.str(string("Wrong checksum",13)) return false ' Something went wrong no ACK sent else BYTE[inx++] := i ' Store received data other: if prev BYTE[inx++] := dle_rec ' Store DLE as no ETB was received dle_rec~ ' Clear dle_rec BYTE[inx++] := i ' Store data received term.str(string("Too much data received",13)) return false ' Too much data received, no ACK send PUB CRC-16(inx, size, checksum) ' Check CRC-16 chechsum ...Here's the code portion that works and a logic trace of the result-
inx := 1 repeat 239 i := dex.rxtime(250) dexdata[inx] := i term.tx(dexdata[inx]) inx += 1 dex.tx($10) dex.tx($31)Here's the code that gives me trouble and a logic trace of its result. You can see all the ACK's it sends in a row unlike the code above-
inx := 1 ' initialize index counter to 1 i := dex.rxtime(20) ' receive in dex data and assign it to variable i if i == $10 ' check to see if byte is DLE i := dex.rx ' receive in next byte if i == $02 ' check to see if next byte is STX. if so start repeat loop repeat until i == $10 ' do repeat until byte received is DLE then quit i := dex.rx ' receive in nexy byte and assign to variable i dexdata[inx] := i ' dexdata variable indexed by index counter inx term.tx(dexdata[inx]) ' send data byte to terminal for printing inx += 1 ' increment index counter dex.tx($10) ' send ACK - DLE 1 dex.tx($31) ' ACK's alternate between DLE 0 and DLE 1I want to make the second code work as the message length may vary. In the first code I hand coded the message length.
@rosco_pc- yes I know I am not doing anything with the checksums at this point and just assuming they are correct to get started. I was going to implement them in the next go around after I get this portion to work better. Thanks for your code help. I'll look it over and see what I can learn from it.
I'd think you'd want your "dex.tx($10)" to line up with "repeat until" so you only acknowledged after receiving a full message.
Duane
inx := 1 ' initialize index counter to 1 i := dex.rxtime(20) ' receive in dex data and assign it to variable i if i == $10 ' check to see if byte is DLE i := dex.rx ' receive in next byte if i == $02 ' check to see if next byte is STX. if so start repeat loop repeat until i == $10 ' do repeat until byte received is DLE then quit i := dex.rx ' receive in nexy byte and assign to variable i dexdata[inx] := i ' assign received byte i to dexdata variable indexed by index counter inx term.tx(dexdata[inx]) ' send data byte to terminal for printing inx += 1 ' increment index counter i := dex.rx if i == $17 pause(2) dex.tx($10) ' send ACK - DLE 1 dex.tx($31) ' ACK's alternate between DLE 0 and DLE 1Now on to more refinement....
I messed with the code that rosco_pc suggested but I didn't get anywhere with it due to some code needed for CRC-16 calculation which I don't have.... yet.
PUB crc16(buffer,count) result := $FFFF repeat count result ^= byte[buffer++] repeat 8 result := result >> 1 ^ ($A001 & (result & 1 <> 0))You'll need to change the calling of the crc routine to I had never heard of DEX, so I after some quick googling l found some interesting links:
http://saturn.ece.ndsu.nodak.edu/ecewiki/images/0/06/SD0802Technical_Report.pdf
http://bonusdata.ch/Download/DEX-UCS/dexucs_english.htm
First link contains some flow charts on how DEX is supposed to work (there is no source code)
The second link contains some source code (it's C++ though) and you'll have a reference implementation for DEX.
Is there a command to clear the buffer?
This only clears the variable i (reading from the buffer removes the data)
FDS buffer is 64 byte, do you only get 64 characters? If you get more or less then you most likely have a problem with handshaking.
inx := 1 ' initialize index counter to 1 ack := 1 ' initialize ack counter to 1 repeat repeat until i == $10 ' watch for DLE i := dex.rx ' receive data and assign to variable i if i == $10 ' check to see if byte is DLE i := dex.rx ' receive in next byte if i == $02 ' check to see if next byte is STX to start repeat until i == $10 ' receive data until second DLE i := dex.rx ' receive next byte to variable i dexdata[inx] := i ' received byte i to dexdata indexed by inx term.tx(dexdata[inx]) ' send data byte to PST inx += 1 ' increment index counter i := dex.rx ' receive next byte to variable i if i == $17 ' watch for ETB inx := 1 ' reset index counter i~ ' clear variable i pause(2) ' pause for ACK dex.tx($10) ' send DLE if ack == 1 ' this routine toggles ACK between dex.tx($31) ' $30 and $31 depending on ACK index ack-- else dex.tx($30) ack++ else if i == $03 ' else look for ETX pause(2) ' pause for ACK dex.tx($10) ' send DLE if ack == 1 ' this routine toggles ACK between dex.tx($31) ' $30 and $31 depending on ACK index ack-- else dex.tx($30) ack++ repeat until i == $04 ' watch for EOT i := dex.rx term.str(string("DEX Finished !"))' indicate DEX finished quit quitDifferent manufacturers report different amounts of DEX data. It just so happens that one board I have this whole routine works fine on. It takes approx 5 seconds to go through the whole process. The board I'm having trouble with reports quite a bit more data and runs along fine for about 12 seconds then just hangs. There is a DLE ETB sent from the board but my program then decides not to ACK it for some reason after that time even though it was ACKing fine in the seconds preceeding.
This one is from the board that the code works fine on. You can see the long data stream on the bottom line and the ACK's on the top line. It then shows the final EOT on the bottom line from the board. Life is good.
This is the one from the board that hangs after a while. You can see where the data is ACK'd then it decides not to ACK at the end.
You'll also notice that the packets of data are shorter on the second board than the first. Maybe that has something to do with it I don't know.
... else if i == $03 ' else look for ETX pause(2) ' pause for ACK dex.tx($10) ' send DLE if ack == 1 ' this routine toggles ACK between dex.tx($31) ' $30 and $31 depending on ACK index ack-- else dex.tx($30) ack++ repeat until i == $04 ' watch for EOT ...What I did to find this problem was to insert some string statements in different strategic points in my code to print on PST. The problem then showed up where it was quitting. Should have thought of this sooner....
Anyway here is the updated portion that works. I'll work on finding a checksum method that I can add to this.
ack := 1 ' initialize ack counter to 1 repeat ' watch for DLE i := dex.rxtime(250) ' receive data and assign to variable i if i == $10 ' check to see if byte is DLE term.str(string("DLE1", 13)) i := dex.rx ' receive in next byte if i == $02 ' check to see if next byte is STX to start term.str(string("STX", 13)) repeat 'until i == $10 ' receive data until second DLE i := dex.rx ' receive next byte to variable i term.tx(i) ' send data byte to PST if i == $10 ' watch for ETB term.str(string(13, "DLE2", 13)) i := dex.rx if i == $17 term.str(string("ETB", 13, "Checksum ")) i := dex.rx term.hex(i, 2) i := dex.rx term.hex(i,2) term.tx(13) dex.tx($10) ' send DLE if ack == 1 ' this routine toggles ACK between dex.tx($31) ' $30 and $31 depending on ACK index ack-- quit else dex.tx($30) ack++ quit else if i == $03 ' else look for ETX term.str(string("ETX", 13)) pause(2) ' pause for ACK dex.tx($10) ' send DLE if ack == 1 ' this routine toggles ACK between dex.tx($31) ' $30 and $31 depending on ACK index ack-- quit else dex.tx($30) ack++ quit else if i == $04 ' watch for EOT term.str(string("EOT", 13)) term.str(string("DEX Finished !"))' indicate DEX finished quitDon't pay much attention to the comments in the code as they are somewhat messed up from editing code.