[Beowulf] network question

[David Mathog]

The questions is:  do modern networks bundle multiple TCP "ack"
together into a single packet?  If so, on linux does ifconfig
count all N acks in that single packet as if they were separate packets?

Here's the background:

I have been modifying nettee lately and ran across something which was
a bit mysterious.  The initial observation had TCP_NODELAY set
on the data line.  This was a mistake but it was largely being
compensated for by a minwrite variable which controlled how big the
buffer had to be before it was emptied by a read.  Anyway, when running
in that mode these 3 tests

A->B
   B->C
A->B->C

were performed.  (All 3 are on a single 100baset switch.) The first two
ran at "full speed" (11.x Mb/sec) and the third much slower.  Which is
odd, since B could read and write at "full speed", just not both at the
same time.  So to work on this issue runs on B were instrumented like this:

ifconfig eth0 | grep packet; nettee... ; ifconfig eth0

and the RX/TX counts compared before and after to see how many packets
moved in/out on each test.

For A->B, on B there were 27857 packets in and 13963 packets out.  
For A->B->C on B there were 41887 packets in and 42441 packets out.

Since the sum of in + out on B for the first test is 41820, which is
very close to 41887 for B in the second test, there are definitely
a lot of ack packets coming back to B from C, and that seemed likely
to be the problem. Only it wasn't, at least according to ifconfig.

By varying the minwrite parameter described above the nettee
throughput on B (and so the whole 3 member chain) could be adjusted to
"full speed".  The same speed was obtained by not setting
TCP_NODELAY, in which case the minwrite parameter made no difference. 
Oddly, in these configurations where the program ran fastest the
RX/TX counts changed only very slightly and not as I expected.  In
one typical "optimized" relay B had 41744 in RX and 41822 in TX.
These numbers are only very slightly different from the unoptimized
example shown above.  The one way in which they were remarkable, and
this could just be a coincidence, was that for the highest transfer
rates the observed RX/TX ratio was closer to 1.0 than for other
configurations.

So my best guess for explaining how the data rate could increase so
much is that there really were fewer packets, and ifconfig has somehow
concealed this fact by breaking the multiple acks out and counting
them as separate packets.

Or is there something else going on?

I also discovered that "athcool" really screws up network throughput
on these Athlon machines.  Dropping from about 11.4 Mb/sec to 
about 6.7 Mb/sec.  We run a script that keeps track of CPU usage and
shuts athcool off when CPU use peaks, but nettee only reached  20-30
percent of CPU so that never kicked in.  I wonder if Athlon64's have
the same issue when they are in their power saving mode, but have not
run tests yet to find out.

Thanks,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech