[Beowulf] diskless booting over jumbo frames

[Bogdan Costescu]

On Wed, 25 Apr 2007, Amrik Singh wrote:

> I agree that Jumbo Frames would not be a great help with the root 
> file system but we hope to get a better performance from other NFS 
> servers.

I don't quite follow you here: the link that was sent earlier had 
shown some kernel level autoconfig problem. If this is indeed still 
the case with newer kernels, this should only affect the root FS - 
before mounting the other NFS exports, you should have the chance to 
perform a proper initialization of the network, which would give you 
the possibility to use a larger MTU.

> As all the machines on the same subnet have to be using the jumbo 
> frames,

Why ? The MTU specifies a _maximum_ value; even when using 1500, not 
all packets are exactly 1500 bytes. The larger MTU _allows_ larger 
values, but doesn't _force_ them.

> Even though NFS is extremely slow, copying files over scp is still 
> very fast between a client and server.

If by this you mean that NFS is slower than scp, then this should be 
exactly the other way around, because scp needs CPU time for its 
encryption. I would have said that you have some network problems and 
use NFS over UDP which leads to high retransmission rates; scp would 
adapt better to the network problems due to using TCP... but you just 
mention that you tried switching between TCP and UDP.

> We have tried all different ways to tune the NFS for a better 
> performance (increasing NFS deamons on the servers, changing rsize & 
> wsize, using TCP vs UDP, using async vs sync, noatime, timeo).

Could it be that you tried so hard to tune it that you just used too 
many settings that don't play well together in your particular 
situation ? I've found the NFS client and server in the kernels of the 
recent years to perform reasonably well in their default 
configuration; although they could be further optimized, a NFS 
transfer in these conditions would always beat a scp one.

> initrd=bzImage ramdisk=40960

dhclient could be located in the ramdisk. Actually even a whole root 
FS could be located in the ramdisk, if there are not too many nodes 
booting at the same time and leading to UDP packet loss.

... and you already have a rather large ramdisk. Have you created it 
yourself (and therefore know what's in it and how to add some more) ?

-- 
Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: Bogdan.Costescu at IWR.Uni-Heidelberg.De