[Beowulf] how fast can NFS run?

[Dan Stromberg]

On Wed, 2006-02-01 at 00:13 -0500, Joe Landman wrote:
> 
> Bruce Allen wrote:
> 
> >> b) If this is gigabit, you get about 110 MB/s max in best case 
> >> scenarios, with the wind at your packets, along with a nice 
> >> gravitational potential, an a good switch to direct packets by.  If 
> >> this is IB, you should be able to see quite a bit higher, though your 
> >> PCI is going to limit you.  PCI-e is better (and HTX is *awesome*).
> > 
> > 
> > This is either 10Gb/s or three or four channel-bonded Gb/s links.  This 
> > should give substantially faster than 100MB/sec.
> > 
> > Note that even PCI-X is 133 MHz x 8 bytes = 1064 MB/sec.
> 
> Most folks don't see more than 850 MB/s due to overhead and other 
> issues.  This is one of the reasons why PCIe is so interesting/useful.
> 
> The idea is that you shouldn't be filling this link up.
> 
> A simple test between 2 systems like this would be whether or not you 
> can push the data over something as simple as a socket (not even NFS) to 
> a receiver at the "wire" speed through the switch.  I assume you could 
> read it off the disk and shove it out the wire (might even want to just 
> stream from disk out the wire, and avoid buffering as this looks like it 
> will get in the way of the disk for this scenario).  You could even have 
> 4 single wire systems to which you open one socket each, and just toss 
> data.  c.f. http://netcat.sourceforge.net/

I may be mistaken, but I've been assuming that my
http://dcs.nac.uci.edu/~strombrg/pnetcat.html program (a python
reimplementation of netcat, with different options - IMO, they're more
coherent, and also it can easily to file to file or socket to socket,
not just file to socket or socket to file, and also has a "don't write
the data anywhere, just drop it" mode) is better for testing raw
bandwidth than netcat, because it:

1) Allows you to use huge block sizes, to facilitate the IP stack being
able to maximize your jumbo frames, avoid context switches, etc.

2) Allows you to adjust your TCP window sizes, which can give a big
performance boost.

If I'm assuming incorrectly, I hope someone will speak up :)

Also, for something closer still to ivory-tower performance, iperf is a
good bet.