[Beowulf] NFS over RDMA performance confusion

[Joe Landman]

On 09/13/2012 07:52 AM, holway at th.physik.uni-frankfurt.de wrote:

[...]

> If I set up a single machine to hammer the fileserver with IOzone I see
> something like 50,000 IOPS but if all four machines are hammering the
> filesystem concurrently we got it up to 180,000 IOPS.

I wouldn't recommend IOzone for this sort of testing.  Its not a very 
good load generator, and it has a tendency to report things which are 
not actually seen at the hardware level.  I'd noticed this some years 
ago, when running some of our benchmark testing on these units, that an 
entire IOzone benchmark completed with very few activity lights going on 
the disks.  Which suggested that the test was happily entirely cached, 
and I was running completely within cache.

Use fio.

Second, are the disks behind the NFS/ZFS server solid state, ram disk, 
or spinning rust?

> Can anyone tell me what might be the bottleneck on the single machines?
> Why can I not get 180,000 IOPS when running on a single machine.

50k  IOPs x 16k/IOP =  819 MB/s

180k IOPs x 16k/IOP = 2949 MB/s (close to pragmatic limits on QDR)

Some observations ... these don't sound like disk subsystems.  A 15k RPM 
drive will give you ~300 IOPs.  To get 50k IOPs, you would need 167 disk 
drives per machine, operating in a best performance case scenario (RAID0 
or JBOD).  To get 180k IOPs, you'd need 600x 15k RPM disks.  I am 
guessing you don't have that.

Are you asking why a single machine cannot fill your QDR bandwidth?

I'd recommend running traces on the individual machines to see where 
things are getting lost.  One you have the traces, post em, and see if 
people can help.

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615