Packet (in)efficiency

Phil Pilgrim (PhiPi)

Is there any way to force the W5100 to slow down and send larger packets? I think it would lead to higher overall throughput. For example, I'm seeing an HTML page with 5K bytes of data requiring 781 (!) packets and 38K overall bytes to send! That's about 6.5 characters per packet, and a data efficiency of 13%. It might not matter on a 100 Mbaud Ethernet local network, but once it leaves from here for the internet, things get choked down to 256 Kbaud or less. So any inefficiency does make a difference.

-Phil

kuisma

Do not call txTCP() for each and every part (line, paragraph, whatever) of the page you are transmitting, but collect it in a transmit buffer, and when the page is ready, call txTCP() with this whole buffer as one invocation.

In an ordinary TCP implementation, this would be controlled by Nagels algorithm, but with the limited resources in the W5100 chip, I guess it will always try to flush its buffers, and if you don't want this, buffer yourself.

Phil Pilgrim (PhiPi)

I do buffer individual characters in a smaller buffer, but having to buffer an entire page presents a rather large burden on hub RAM. Reading the source for TxTCP shows that writing the data to the buffer and sending it are two discrete operations, and TxTCP always does the send, regardless of how little data it's buffered. It seems to me that the W5100's internal buffer could be used to buffer more data before the trigger is pulled. This would entail a separate flush operation when the page is finished, but that's no big deal.

-Phil

kuisma

Good thinking. It didn't occur to me it is actually possible to implement Nagles algorithm in the driver in txTCP, solving this issue of yours.

Who's up to the task?

Phil Pilgrim (PhiPi)

It probably doesn't have to be that complicated. In my case, I'm not doing any reads while the buffering is occurring, so the W5100's buffer would just be used as a substitute for hub RAM unitl it's full or I'm finished with the page/image. Perhaps separate bufTCP and flushTCP methods are all that are needed. If the user is careful, and used them instead of txTCP, the latter would not even have to be changed. (But it would be better if it were aware of pre-buffered data.)

-Phil

kuisma

I think it's best to keep to common practice. One single call to txTCP. This call puts the data into the w5100 transmission buffer, but only issues the transmit command conditionally: if the the packets is large enough, or running out of the internal buffers of the W5100.

Later in the PASM driver main loop, a timeout condition invoking the transmit command after a specified number of milliseconds.

Two new methods needed: setNdelay(ms) to set the timeout (or zero to disable, forcing the driver to behave as today), and flush() to invoke the transmission command immediate.

With those modifications, the driver will still be backward complaint with old applications without need of modifications.

Phil Pilgrim (PhiPi)

Somewhere there still has to be a separate flush method that can be called just before the socket is closed.

-Phil

kuisma

Yes, there will be an explicit flush() method, but it is mainly called from the driver within (e.g. at timeout and at socket close), but may be called explicit to speed up the network I/O - but even if not called, the code will still work as intended, even if the transmissions becomes somewhat delayed.

All this is how TCP transmission is implemented in almost every system.

http://en.wikipedia.org/wiki/Nagle%27s_algorithm