[Beowulf] Re: Cluster Networking

[Dave Love]

Rahul Nabar <rpnabar at gmail.com> writes:

> On Fri, Jun 26, 2009 at 1:30 PM, Jeff Layton<laytonjb at att.net> wrote:
>> Try something like OpenMX over GigE. Much better latencies

∼6μs, if that counts as much better.

>> and should perform and scale better.

Are there data on that?  I'm not clear how much more efficient than TCP
it might be CPU-wise, for instance, and I'm not sure how best to check.

> How close does it get to native Myrinet performance? Or Infiniband.

Not at all for Infiniband.  With the right NICs on two rails, it's
competitive with our Myrinet-2000 system.  See open-mx.org for 10G data,
but they're presumably not relevant to you.

> OpenMX might be a great way for our cluster too to achieve better
> performance without changing our eth backbone.

In principle with Open MPI, it should use the two rails (NICs) to double
the bandwidth as with TCP; that's currently broken, although Manchester
seem to be getting away with it somehow.  Brice will get back to fixing
it when he returns in a couple of weeks.