[Beowulf] IB switches: managed or not?

[Tom Mitchell]

On Tue, Mar 06, 2007 at 12:06:07AM +1100, Andrew Robbie (GMail) wrote:
> Date: Tue, 6 Mar 2007 00:06:07 +1100
> From: "Andrew Robbie (GMail)" <andrew.robbie at gmail.com>
> To: beowulf at beowulf.org
> Subject: [Beowulf] IB switches: managed or not?
> 
> 
>    Hi,
>    I am building a small (~16) node cluster with an IB interconnect. I need to
>    decide whether I will buy a cheaper, dumb switch and run OpenSM, or get a
>    more expensive switch with a built in subnet manager. The largest this
>    system would every grow is 32 nodes (two 24 port switches).

A year ago the hands down answer was "built in subnet manager".
Today, the OpenSM folk have made big improvements.  We may
be at or past the tipping point with OpenSM code quality.
For LARGE clusters OpenSM  or a vendor provided host based
SM may be a requirement because the cards for many "built in
subnet managers" simply run out of memory someplace beyond a
gross (144) nodes and thousands.

One big OpenSM bug/challenge is fail over.  Make sure that exactly one
copy of OpenSM is running.   Once things are fine and dandy explore 
having a second copy but not three+.

As far as I know there is nothing like Ethereal/Wireshark
that applies to IB. There is no raw packet interface that I
know of and if there was the bandwidth/memory issue would be
a challenge for all modern processors.   The managed switches
do give you access to good statistics from the ports in the
switch.

My opinion is that you should save yourself some gray hair and
get a managed switch as your first switch.  The second IB switch
can be managed by the first switch.   Try and get all the IB parts
from the same vendor.


>    Various  vendors (integrators, not switch OEMs) have stated to me that
>    managed switches are the go, and that OpenSM is (a) buggy, and (b) very time
>    consuming to set up. But, a managed name brand switch seems to cost a lot
>    more  than  a  non-managed one using the Mellanox reference design kit
>    (rebadged, but I suspect made by Flextronics...).
>    My other query is about diagnostic software. With an ethernet switch it is
>    pretty easy to fire up Ethereal (sorry Wireshark, but it is such a silly
>    name) or Etherape and get a look at what is going on. If I buy a Cisco or
>    Voltaire  etc  do  they  come  with  tools  that  let  me get accurate
>    representations of what is going on? Or are their tools really for large IB
>    networks?
>    Regards,
>    Andrew
>    [v2]

> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf


-- 
	T o m  M i t c h e l l 
	Found me a new place to hang my hat :-)
	Now it got bought.