[Beowulf] Gbe switch comparision

[Håkon Bugge]

Hi,

we did a study of two different Gigabit ethernet switches from one 
particular vendor on a 16-node XEON cluster (2.4GHz dual XEON, 400MHz FSB, 
Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet (rev 15)). We run a 
number of benchmarks every night on the cluster as part of our regression 
testing for Scali MPI Connect(synthetic, application benchmarks and 
applications). Although we ran on a _very_ little system, that is not of 
the fastest these days, we observed significant differences. I thought the 
findings might be of interest to the community, not necessarily as a guide 
to which switch to purchase, but merely as a motivation to assess the 
quality of the switches to consider in a purchase.

Best regards, Hakon


Benchmark                Switch A   Switch B       A/B
======================================================
Pnt2pnt (usec - lower is better)
Ping pong latency            30.9       34.6
Ping pong bw 128              3.6        3.3
Ping pong bw 1k              16.8       14.2
Ping pong bw 1M             116.6      116.4
All2all 1k                  57.57       60.4
All2all 128k                78.93      79.57
Barrier latency            139.27      143.9
-------------------------------------------------------
Pallas snapshots (usec - lower is better)
Reduce 4                   250.79     268.91      0.933
Redyce 1M                   61241      34575      1.771
Allreduce 4                167.69     172.39      0.973
Allreduce 1M                62540      41208      1.518
Bcast 4                    255.23     250.43      1.019
Bcast 1M                    72173      72227      0.999
Allgather 4                571.33     634.86      0.900
Allgather 1M               612035     617526      0.991
Alltoall 4                 255.27     254.77      1.002
Alltoall 1M               1500884     856977      1.751
-------------------------------------------------------
NPB (Total Mflops/sec - higher is better)
bt                         3631.4     4800.8      0.756
cg                         313.15    1628.62      0.192
ep                         387.49     389.58      0.995
ft                         2643.3     3985.6      0.663
is                          61.29     168.56      0.364
lu                         8835.2    10306.5      0.857
mg                         1760.9     2913.8      0.604
sp                         1499.4     2377.4      0.631
-------------------------------------------------------
Applications (Jobs/day - higher is better)
MM5                          4.64       4.84      0.958
Fluent fl5m3                475.8      499.3      0.953
Fluent fl5l2               1122.1     1116.3      1.005
Eclipse E100 100k          178.14     188.24      0.946
Eclipse E100 1M             28.13      29.54      0.952
Eclipse E300 100k           84.74     105.45      0.804
Ls-dyna                      44.7      44.49      1.005
Powerflow                  413.48     414.69      0.997
StarCD Aclass              141.34     144.18      0.980
StarCD Eng20              1367.09    1353.81      1.010
StarCD large custom         40.13      39.76      1.009