NIC/driver/channel bonding tests

[Bogdan Costescu]

On Fri, 17 Nov 2000, Martin Siegert wrote:

> Remarks
> =======
> 1. For channel bonding use the module bonding.c from the 2.2.17 kernel even
>    if your are using a 2.2.16 kernel. Otherwise bad things (kernel oops)
>    happen. I found out about this the hard way:
>    "/etc/rc.d/init.d/network restart" will crash your machine so badly that
>    it won't even reboot (it hangs on unmounting /proc with "device busy").
>    Only power cycling the box will get it back to life. The problem is
>    with bonding module: "ifconfig bond0 down; rmmod bonding" will yield
>    the same result (oops).
>    The 2.2.17 module seems to fix this.

The bonding module had this problem (or at least, I encountered it) since
the beginning. Recent work was done to eliminate this (and other)
problem(s). If I remember well, even some recent 2.2.18pre kernels had
some changes in this area.

> 2. The 3c59x.c from the 2.2.17 kernel produced basically the same results
>    as the high-performance variant used in the tests above. Somehow these
>    tests don't seem to use the performance enhancing "polling" modes of
>    new the driver. Does anybody have more information on this?
>    The 3c59x.c from kernels < 2.2.17 have bugs and should not be used.

As the author of the modifications, I should answer this. Basicaly, the
polling mode is supported by both Cyclone (905B) and Tornado (905C) cards.
Don's 3c59x driver (as of 0.99Q, I think) enable it for both these chipsets.
However, I didn't have Cyclone cards for testing and I cannot guarantee
that my modifications work, although the documentation says so. So the
polling mode is only enabled for Tornado cards, which I can guarantee that
work (as I also use them). The web page that you mentioned wrongly states
that it's also enabled for Cyclone cards.
If you want to try it, I can  give you a small patch or you can change
every occurence of:

	vp->drv_flags & IS_TORNADO to
	vp->drv_flags & IS_CYCLONE

The idea of the "high-performance" driver was to introduce some features
that are supported by the hardware, but not by the current driver. Given
the fact the both Linus (for 2.4) and Alan Cox (for 2.2) wanted a "feature
freeze" at some point, Andrew Morton (the maintainer of the 3c59x driver)
refused to add them to the mainstream driver. I won't go into details
about these features, but if you want more, either write to
vortex at scyld.com or to me in private (although if you reply and CC:
beowulf, I'll keep it 8-))

>    Channel bonding with the 3c90x driver fails for the similar reasons
>    (in that case not even ifconfig reports identical MAC addresses).

I think that the 3c90x driver does not support this feature (introduced by
Don and present in most - if not all - of his drivers).

>    ... However, some of the cards are "smarter" than others as they
>    can off load some of the cpu tasks. Therefore, a higher throughput in
>    this test indicates a "smarter" card. This should be particularly
>    important when you channel bond the cards. If the cpu is 100% busy
>    with maintaining a high throughput, it is impossible to do computation
>    and communication in parallel. In this area the 3C905B/3c59x outperforms
>    all the other card/driver combinations.

... however you didn't include latency figures and some programs are more
sensitive to latency than to bandwidth. This is important when bonding
interfaces, because you get closer to the maximum PCI bandwidth. The
typical transfers from/to NICs over the PCI bus are done in bursts and
having several NICs trying to share the bus might put one or more of them
"on hold".
Even more, polling mode is a way for the DMA engine on the NIC to query
some main memory locations for flags indicating that there are packets to
be sent (instead of the CPU doing direct I/O to the card and waiting...);
but for this, the DMA engine has to also use the PCI bus. So, having a
too small polling interval for several NICs might lower the performance;
OTOH, having a too large polling interval might generate bursts of
packets, with increased latency (on 3Com cards, the polling interval unit
is 320 ns).

Maybe it's a good time to ask some other knowledgeable people on this
list (especially Don) about usefulness of polling mode when using the
NICs in clusters and bonding configurations...

Sincerely,

Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: Bogdan.Costescu at IWR.Uni-Heidelberg.De