[Beowulf] LSI Megaraid stalls system on very high IO?

[Gavin W. Burris]

Hi, David.

I've been pretty happy with various megaraid controllers.  Some systems
do get unresponsive under high storage load.  One thing that helped was
changing the IO scheduler.  You may want to consider deadline, instead of
cfq.

# echo deadline > /sys/block/sdX/queue/scheduler

G'luck.

On Thu 07/31/14 09:37AM -0700, mathog wrote:
> Any pointers on why a system might appear to "stall" on very high IO through
> an LSI megaraid adapter?  (dm_raid45, on RHEL 5.10.)
> 
> I have been working on another group's big Dell server, which has 16 CPUs,
> 82 GB of memory, and 5 1TB disks which go through an LSI Megaraid (not sure
> of the exact configuration and their system admin is out sick) and show up
> as /dev/sda[abc], where the first two are just under 2 TB and the third is
> /boot and is about 133 Gb.  sda and sdb are then combined through lvm into
> one big volume and that is what is mounted.
> 
> Yesterday on this system when I ran 14 copies of this simultaneously:
> 
>   # X is 0-13
>   gunzip -c bigfile${X}.gz > resultfile${X}
> 
> the first time, part way through, all of my terminals locked up for several
> minutes, and then recovered.  Another similar command had the same issue
> about half an hour later, but others between and since did not stall.  The
> size of the files unpacked is only about 0.5Gb, so even if the entire file
> was stored in memory in the pipes all 14 should have fit in main memory.
> Nothing else was running (at least that I noticed before or after, something
> might have started up during the run and ended before I could look for it.)
> During this period the system would still answer pings.  Nothing showed up
> in /var/log/messages or dmesg, "last" showed nobody else had logged in, and
> overnight runs of "smartctl -t long" on the 5 disks were clean - nothing
> pending, no reallocation events.
> 
> Today ran the first set of commands again with "nice 10" and had "top" going
> and nothing untoward was observed and there were no stalls. On that run
> iostat showed:
> 
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda            6034.00         0.00    529504.00          0     529504
> sda5           6034.00         0.00    529504.00          0     529504
> dm-0          68260.00      2056.00    546008.00       2056     546008
> 
> 
> So why the apparent stalls yesterday?  It felt like either my interactive
> processes were swapped out or they had a much lower priority than enough
> other processes so that they were not getting any CPU time. Is there some
> sort of housekeeping that the Megaraid, LVM, or anything normally installed
> with RHEL 5.10, might need to do, from time to time, that would account for
> these stalls?
> 
> Thanks,
> 
> David Mathog
> mathog at caltech.edu
> Manager, Sequence Analysis Facility, Biology Division, Caltech
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

-- 
Gavin W. Burris
Senior Project Leader for Research Computing
The Wharton School
University of Pennsylvania