[Beowulf] 10G and rsync

[David Mathog]

On Thu, 2 Jan 2020 13:32:17 Michael Di Domenico wrote:
> On Thu, Jan 2, 2020 at 12:44 PM David Mathog <mathog at caltech.edu> 
> wrote:
>> 1. Is a single large file transfer rate reasonable?
>> 2. Ditto for several large files?
> 
> yes, if i transfer files outside of rsync performance is reasonable
> 
>> Are you sure there is not a patrol read ongoing on one system or the
>> other?  That can cause this sort of disk head issue.
> 
> yes, i control both sides.  the client side is totally idle and the
> lustre system is quiet.

Double checking - you queried the RAID card (if present) to see that it 
was not doing a patrol read or SMART analysis?  In my experience SMART 
commands do not light the disk activity lights, so physically looking at 
the array may show no or little activity when in fact the disks are 
working quite hard.

> 
>> Also it might be this "hugepage" issue:
>> https://www.beowulf.org/pipermail/beowulf/2015-July/033282.html
> 
> ah forgot about that one.  tried it, no change

Hmm.  Let's see if you can take the file systems more or less out of the 
equation.  Something along these lines:

1. Create 100 FIFOs with matching names on each end in a similarly named 
directory.
2. On the receiving machine spin out 100 processes doing:

   dd if=/PATH/FIFOname12 of=/dev/null &

3. On the the sending side spin out similar process to write to the FIFO

   dd if=/dev/zero of=/path/FIFOname12 bs=8196 count=10000 &

4. Start up rysnc on the directory holding the FIFOs.

I never tried coercing rsync into working like that, but if it can be 
done then it emulates a storage system to storage system transfer 
without ever actually reading or writing to any file systems.

Regards,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech