[Beowulf] Q: IB message rate & large core counts (per node)?

[Gilad Shainer]

PCIe Gen2 at 5GT, and 8/10 bit encoding and current chipsets
efficiencies gives you around 3.3GB/s per direction, so one IB QDR port
can handle that. For more BW out of the host, you can use more adapters
(single port ones are the cost effective solution for that). 

 

Gilad

 

From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org]
On Behalf Of Brian Dobbins
Sent: Tuesday, February 23, 2010 2:42 PM
To: Patrick Geoffray
Cc: beowulf at beowulf.org
Subject: Re: [Beowulf] Q: IB message rate & large core counts (per
node)?

 

 

	  I thought PCIe Gen2 x 8 @ 500 Mhz gives 8GB/s?  I know there
are 250 and 500 Mhz variants in addition to the lane sizes, so while a
250 Mhz x8 link wouldn't provide enough bandwidth to a dual-port card,
the 500 Mhz one should.  But I'm woefully out of date on my hardware
knowledge, it seems.  Of course, EDR (eight data-rate) IB is on the
roadmap for 2011, so if we're in no rush that could help, too.


Er,.. I thought about this a half-second after I sent it.  First, that's
5.0 Ghz, not 500 Mhz, and second, that's probably aggregate
bi-directional throughout at 8 GB/s, isn't it?  So, I got it, 4 GB/s per
direction, which means dual-port won't get us much in terms of
unidirectional throughput.

  - Brian

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20100223/8881a7ea/attachment.html>