[Beowulf] GPFS on Linux (x86)

[Scott Atchley]

On Sep 14, 2006, at 9:41 PM, Greg Lindahl wrote:

> On Thu, Sep 14, 2006 at 08:08:12AM -0400, Scott Atchley wrote:
>
>> If, however, you are considering something faster than gigabit
>> Ethernet, bear in mind that GPFS only can use TCP (as far as I know).
>
> The July 2006 presentation posted to the list about the same time as
> your email says they're adding IB protocols in the future. I thought
> that GPFS used to support GM directly?

I believe that they used Ethernet emulation (opened the myri0  
Ethernet interface) and used it. That is how they currently do it  
with both GM and MX.

If they do modify the network layer to support alternate APIs, that  
will be a good thing for GPFS.

>> You will be limited to the performance of the TCP on your
>> interconnect. If you are using TCP/IP/Ethernet, then expect a very
>> high CPU load.
>
> This is debatable. It is often the case that a large cluster does big
> I/O (say 10+ GB/s) on so many nodes (1000+) that individual nodes are
> doing < 100 MB/s. In that case the TCP overhead is not that large.
>
> And frequently you're limited by the disk system, not the network.
>
> But this depends, of course.
>
> -- greg

Ahh, I was thinking of the server side which has to handle the 1000  
clients. I also tested Lustre using its TCP driver (SOCKLND) with the  
kernel zero-copy patch on top of our 10Gb/s NICs using our native  
Ethernet driver (no MX). With the TCP zero-copy patch, CPU usage was  
~10-30% for the server. The client was 12-28% (single client to a  
single server). Details at https://mail.clusterfs.com/wikis/lustre/ 
Myri-10G_Ethernet.

For GPFS without the zero-copy TCP, I would expect loads to be higher.

Clearly, disk will be the limiting factor.

Regards,

Scott