RS232 - Ethernet TCP/IP transceiver with ENC28J60

Szabi

Hi guys!

I'm trying to build a transceiver using ENC28J60 and a propeller.
The main purpose of the device is to forward my serial data over standard ethernet so I can have remote access to the serial device from anywhere using an ethernet connection.

The basic idea is simple. Any data coming from serial should be able to be read through telnet, and any data sent in a telent terminal should be sent to the serial port.
I'm using api_telnet_serial.spin, driver_enc28j60.spin and the Extended Full Duplex Serial object.

Everything is working well unless I'm connecting the ral serial device.
When I set up a PC using putty and I'm sending serial data manually by typing in the terminal window things are received on the other end through telnet and viceversa.
But, as I connect the serial device which is sending a continuous data stream things looks like frozen :-( I mean data is not displayed in the telnet terminal anymore until I stop (or disconnect) the data stream, than only all the tata is showing up suddenly like it was somehow buffered and this is not what I would like to see.
I would like to get the data continuously as it is coming in from serial port.

after setting up and calling the telnet server object the code is just a few lines:

repeat while serv.isConnected
SerRx := Serial.Rxcheck
if SerRx <> -1
serv.tx(SerRx)
LCD.char(SerRx)

c := \serv.rxcheck
if SerRx <> -1
serial.tx(c)

if c == 27
\serv.close
ServGoodBye 'goodbye message
delay_ms (3000)

It may be important that data displayed on the LCD (just for test purpose) is continuous in any case and works perfectly in the way I would like to have it sent through telnet.

any help is appreciated

Thanks

VIRAND

I have noticed the buffering delay before but not sure how to fix it.
I think that windows doesn't like to be interrupted by every character, so the driver just buffers it until something else happens,
such as a CR/LF or ctrl-Z or maybe one of the other serial port pins changing state to indicate an end of data being sent.
Just an idea maybe either add whatever signal is indicating end-of-data OR somehow finding the event handler and
programming a response to every incoming character (the latter I don't know how to do).
Summary: Windows is event-driven and it seems to consider serial data input of one character an insignificant event. (I think)

Szabi

VIRAND wrote: »

I have noticed the buffering delay before but not sure how to fix it.
I think that windows doesn't like to be interrupted by every character, so the driver just buffers it until something else happens,
such as a CR/LF or ctrl-Z or maybe one of the other serial port pins changing state to indicate an end of data being sent.
Just an idea maybe either add whatever signal is indicating end-of-data OR somehow finding the event handler and
programming a response to every incoming character (the latter I don't know how to do).
Summary: Windows is event-driven and it seems to consider serial data input of one character an insignificant event. (I think)

Well, I wouldn't blame windows this time 'cause the same thing happens under MAC OS native telnet client.

I do have a data "packet" separator which is "AAh" (DEC 170) but only if it is received twice in a row. I was trying to implement a kind of event handler but with no luck. However the data rate is 2400boud only, he main problem seems to be because the data stream is continuous with almost no delay between the packets.(separated by AAh AAh) If I pause the stream for about 10 sec. it will work but there is no way I could do it in the source hardware.
It is also important that even if I use this "AAh AAh" as a line separator it needs to be re-produced (re-written) at te end of each data packet sent towards the ethernet port because it is needed at the other end. I was thinking to buffer and delay the data stream but with a minute or so, which is not crucial but by separating the packets and re sending with a 10sec delay in between it may result hours of delay after a few packets which is not OK.

I have no intervention in the source serial hardware so I need to work out a solution in my prop device. Probably speeding up the used objects will help but I'm not good in asm.

localroger

This is really a problem with TCP/IP. It's not a very good substitute for serial when you have either continuous data or a high speed challenge-response interface.

The problem is that the TCP stack wants to reduce overhead by putting as much info as possible into each packet. This means it doesn't transmit immediately; when you give the stack some data it waits 0.2 sec (the "Nagle delay") before putting a packet together and transmitting it. Depending on how the stack is implemented it could spend several whole seconds collecting data before sending the packet.

Then, another problem is that the Prop doesn't have enoug RAM to properly implement the packet reordering algorithm, so if you miss a packet or one comes out of order there's no buffer and a lot of re-requests have to be made. Some conventional TCP appliances don't like this hack and won't play with it, and if you're doing these re-requests and there's a steady rain of new packets coming because of continuous data you might just never recover. If the Prop got packet 1101, then 1103, it's going to request 1102 and it will reject 1104, 1105, and 1106 if they come first, then once it gets 1102 it's got to request 1103 again and ignore anything else that wanders in.

Telnet seems to work OK for continuous data up to a packet every quarter second or so; I limit mine to once a second to be sure. For much better performance, open a separate connection for each packet, that way if something gets lost you just slam it shut and start over; the stalled connection doesn't lock up the whole works for 10 seconds as timeouts sort out. It sounds nuts but on an embedded system that works a lot better than a continuous open connection.

Szabi

Thanks for your reply this are really bad news for me :-(
Loosing a packet or two would not be a big problem because the system is re-sending them approx. each 20-30sec, so I may catch it a minute later os so... or by sending a command I could re-request the serial data what I'm looking for, but as you say because the continuous data flow, yep it's difficult and causing me a headache. A way doing it would probably be to buffer 1 min of data or so and than processing it (while skipping new incoming data) and than re-listening to the serial port
Could you please post a working example of yours ?

Multiple connections may also be a good idea but I didn't really got it how do you mean or how I could do it.

Thanks!

localroger

Szabi, do you have a Prop at both ends of your connection or is there a standard device of some sort like a PC doing the transmitting? It makes a big difference. Also, does this have to route over an actual network via IP or is it just a direct connection?

Szabi

At the moment the test setup is: RS232device @ 2400 bout - Prop - Switch - PC Putty, labview (or mac)

The final planned setup needs to be routed through real network like: RS232device - Prop - Router - Internet - Router - (2nd prop can be included if needed) - Client PC (putty, labview etc....) and I would fabulous if in the last stage I could use another prop device which is transforming back to RS232 so that two devices can talk to each other but this is not that much important for now.

localroger

If you are doing props at both ends and it needs to be routable your best bet would probably be to use UDP, but I don't know of any existing code for building the packets. UDP is similar to TCP in that it routes over IP but it's "lossy" in that missing packets aren't re-requested and packets aren't re-ordered when they arrive, so it's much more suitable to what you're trying to do.

I came close to doing this myself a year or so ago but because I didn't have a UDP library for the PC end I ended up using the aforementioned short TCP connections.

Szabi

UDP may be a good idea. It wouldn't be a problem for labview since the TCP/IP library includes UDP as well.
Do you have UDP examples for prop?
Another idea that came, I could dump the serial data on a flash card for example and then download it regularly or to upload regularly to an FTP server. What do you think about? Ok, this wouldn't be anymore a TCP/IP - RS232 transceiver anymore but will fulfill my need.
...
Or any opinion doing this by using a Wiznet W5100? Anyone tried using the Spinneret as a Network to serial bridge ?

$WMc%

Take a look at the Spinneret Forums page.
'
http://forums.parallax.com/forumdisplay.php?82-Spinneret-Web-Server

localroger

A couple of years ago I took a pretty close look at Kye's code and found the routine where he's manually building the packet header using direct writes to the ENC28J60 buffer. That would be the same for UDP as TCP through the IP layer. You'd basically just remove the code that does TCP packet ordering. I'd be willing to take another peek at it but I've got about four other projects going right now.

Mark_T

One approach with the reordering/packet-loss issue is to interpose another machine between the Propeller and the outside internet. A local machine is hopefully never going to loss packets or re-order, and its TCP/IP stack is going to handle the real world internet properly. If doing HTTP then that machine can be a web-proxy and handle HTTPS for you too - I'd imagine this is relatively straightforward using Apache. I'd like to think Apache can handle telnet forwarding too (but haven't checked) - anyway there's bound to be some relevant tools out there to help with this. The machine could also use VPN to the same ends.

My experience using a ENC28J60 (but not with the Prop) is that it seems to just work if doing HTTP with request/responses that fit in one packet (reordering not at issue!).

Szabi

Thanks for the updates guys!

Localroger, thanks I will have a look at the code soon as I have some time. I just hope that the interesting part is not in ASM...otherwise I'm lost :-(

Mark, using another machine will solve my problems 100%, I know that a simple plug PC for $100 will probably do the job, but it's a question about doing it with a prop. If I would use a PC I wouldn't need a prop device anymore, most of the machines have a USB or serial port where I could connect my serial device and re-transmitting the packet would not be a problem anymore.

Rayman

I think Mark_T is onto something too... If the data fits in one packet, I think it should be better...

localroger

The packet building stuff was in Spin. There was a lot of logic for reversing byte order since Ethernet is big-endian and the prop is little-endian. Somewhat confusing, but other than the dependence on PASM SPI code it was all in Spin.