W5100 and TCP packet size

Timothy D. Swieter

Something is puzzling my mind that I haven't been able to resolve yet. (OK - many things puzzle me, but lets stick to electronics/software here!)

I'm working on the W5100 driver. When reading UDP data out of the buffers of the W5100 the IC adds 6 bytes in front of the data payload and the data contains the information related to IP, port and the size of the payload. It is practically a real UDP header. When reading TCP data out of the buffers there is only data. With UDP I am able to read only a packet at a time out of the W5100 buffers. When working with TCP, how is the program suppose to know how much data is in the TCP packet and how much data to get out of the buffer? I suppose if you have a super fast processing program than perhaps there is only ever 1 packet in the buffer, but the purpose of the buffer is to gather the data until it can be processed.

Any insights? The W5100 hides the TCP packet header info as far I understand so I can't get the details from there either. Maybe the different protocols that use TCP have the data such that when parsing the data it can be recognized and therefore the packet size inferred?

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Timothy D. Swieter, E.I.
www.brilldea.com - Prop Blade, LED Painter, RGB LEDs, 3.0" 16:9 LCD Composite video display, eProto for SunSPOT, PropNET, PolkaDOT-51
www.tdswieter.com

pullmoll

Timothy D. Swieter said...
Any insights? The W5100 hides the TCP packet header info as far I understand so I can't get the details from there either. Maybe the different protocols that use TCP have the data such that when parsing the data it can be recognized and therefore the packet size inferred?

Uh! Does that chip handle the TCP layer on its own behalf? If you have no header info, you can't and don't have to deal with all the bits and whatnot of the TCP protocol. Be happy, read the stream until it ends would be my suggestion then. Except: how would you handle multiple streams without any kind of header? Strange.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
He died at the console of hunger and thirst.
Next day he was buried. Face down, nine edge first.

Patrick1ab

pullmoll said...
Except: how would you handle multiple streams without any kind of header? Strange.

I'm not really good at this, but isn't the separation of streams done by opening different sockets?
You can have a maximum of 4 sockets at the same time, so I would believe that you can handle 4 streams.

pullmoll

Patrick1ab said...
I'm not really good at this, but isn't the separation of streams done by opening different sockets?
You can have a maximum of 4 sockets at the same time, so I would believe that you can handle 4 streams.

Yeah, but Timothy is one layer below sockets. A socket is a stream of e.g. TCP data. If the hardware does not receive TCP headers, you have no way to have more than one socket open at a time. How would you distinguish the raw data in the input buffer without any header?
Who says you can have max. 4 sockets? Is that a hardware limitation of the W5100 module?

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
He died at the console of hunger and thirst.
Next day he was buried. Face down, nine edge first.

Harrison.

If I recall correctly, the W5100 handles this by implementing the receive buffers as circular FIFOs. There is a read pointer and a read length register which you use to read from the buffer. The W5100 handles the process of receiving multiple TCP packets and putting the payload into the FIFO. The buffer can overflow if you don't read the W5100 often enough to keep free space in the socket's buffers (although the TCP protocol should protect from data loss in this case).

You tell the chip you have received the data by modifying a read pointer and writing to some acknowledge register. You may have to implement a FIFO on the Propeller if you run into buffer overflows on the W5100. I don't think this is necessary as the W5100 should be smart enough to only ACK packets it can fit into its buffers, and force the remote host to retransmit data when an overflow can occur.

I think this is a great project. I'm very interested to see how fast you can transfer data over ethernet using the W5100. The ENC28J60 only gets me around 100KBytes/sec, mostly constrained by the slowness of the TCP/IP stack written in SPIN. I just wish I had time to work on a W5100 driver.

Patrick1ab

pullmoll said...
Who says you can have max. 4 sockets? Is that a hardware limitation of the W5100 module?

That's correct. It also seems that you have to tell each socket which part of the tx/rx buffer it may use.

Harrison. said...
I think this is a great project. I'm very interested to see how fast you can transfer data over ethernet using the W5100.

Me too. Especially I would like to know how fast it is in "indirect/parallel mode".
I tried the SPI mode of the W5100 some time ago, but the ENC28J60 is much faster and more stable in this mode.

Timothy D. Swieter

Thanks guys for helping work on this concept, but I still don't have it all straight in my mind. Here is what I know and understand:

• The W5100 is a "hardware" TCP/IP stack. In handles the nitty gritty details of UDP and TCP low level stuff. The microcontroller software is at the socket level and close to the application layer level because it doesn't have to deal with stacks and CRC and such....basically set addresses/ports and read/write.
• The W5100 has four sockets. There is a rx buffer on each socket and there is a tx buffer on each socket. Each socket is setup as UDP or TCP or a couple other options like IPRAW or MACRAW.
• The socket buffer size can be configured as 1K, 2K, 4K, or 8K. This is segmenting the memory and the default is that all buffers are 2K. If you make a buffer 4K then one or more of the other buffers must be smaller because the total rx buffer is 8k and the total tx buffer is 8K.
• Harrison is very close to understanding how to control the W5100 to send/receive data. There are receive size resisters which tell how much buffer there is to read. The Prop would read this and then signal to the W5100 how much it read and signal that it did read and the FIFO pointers are adjusted. However I don't see a correlation to the receive size register and the packet size or data size as this number is only the amount of data in the buffer. As I said with UDP one can gain this knowledge by looking at the 8 byte header details that the W5100 does pass over for UDP data, but this header doesn't exist for TCP data passed over.

I think the concept of streams is what may best describe how to process the data coming in over TCP. The stream would only be the pure data/payload. I think I need to review streams and design patterns around streams to see how I would handle this efficiently in the Prop. Anyone care to explain? I have a thought, but am curious how someone would else would describe it.

Harrison - how would you propose that I do a load/speed test on the driver and hardware I have? At the moment the driver is optimized to be fast in that is uses the counters to clock data out, but I am sure the overhead code could be slightly faster in the SPI mode. Later this week I hope to get the indirect/parallel method going. I would like to do speed test to compare the modes. How did you speed test the ECN28J60?

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Timothy D. Swieter, E.I.
www.brilldea.com - Prop Blade, LED Painter, RGB LEDs, 3.0" 16:9 LCD Composite video display, eProto for SunSPOT, PropNET, PolkaDOT-51
www.tdswieter.com