[Beowulf] Geriatric computer does not stay up

[Gerald Creager]

David Mathog wrote:
> So we have a cluster of Tyan S2466 nodes and one of them has failed in
> an odd way. (Yes, these are very old, and they would be gone if we had a
> replacment.)  On applying power the system boots normally and gets far
> into the boot sequence, sometimes to the login prompt, then it locks up.
>  If booted failsafe it will stay up for tens of minutes before locking.
>  It locked once on "man smartctl" and once on "service network start". 
> However, on the next reboot, it didn't lock with another "man smartctl",
> so it isn't like it hit a bad part of the disk and died.  Smartctl test
> has not been run, but "smartctl -a /dev/hda" on the one disk shows it as
> healthy with no blocks swapped out.  Power stays on when it locks, and
> the display remains as it was just before the lock.  When it locks it
> will not respond to either the keyboard or the network.  (The network
> interface light still flashes.)  There is nothing in any of the logs to
> indicate the nature of the problem.
> 
> The odd thing is that the system is remarkably stable in some ways.  For
> instance, the PS tests good and heat isn't the issue: after running
> sensors in a tight loop to a log file, waiting for it to lock up, then
> looking at the log on the next failsafe boot, there were negligible
> fluctuation on any of the voltages, fan speeds, or temperatures.  It
> will happily sit for 30 minutes in the BIOS, or hours running memtest86
> (without errors).  The motherboard battery is good, and the inside of
> the case is very clean, with no dust visible at all.  Reset the BIOS but
> it didn't change anything.
> 
> Here are my current hypotheses for what's wrong with this beast:
> 
> 1. The drive is failing electrically, puts voltage spikes out on some
> operations, and these crash the system.
> 2. The motherboard capacitors are failing and letting too much noise in.
>  The noise which is fatal is only seen on an active system, so sitting
> in the BIOS or in Memtest86 does not do it. (But the caps all look good,
> no swelling, no leaks.)  It will run memtest86 overnight though, just in
> case.
> 3. The PS capacitors are failing, so that when loaded there is enough
> voltage fluctuation to crash the system.  (Does not agree very well with
> the sensors measurements, but it could be really high frequency noise
> superimposed on a steady base voltage.)
> 4. Evil Djinn ;-(
> 
> Any thoughts on what else this might be? 


I'd also be suspicious of memory failures.  We have had DIMM failures 
that were unseen on repeated MemTest86 runs until they failed hard, 
hard, HARD.  While they were still trying to decide, they'd pass MemTest 
and we'd try using them.

Capacitor failures are a potential problem but if the systems have been 
in a stable environment and not subject to a lot of thermal stressors, 
they should be fine.  Especially the power supply caps shouldn't decide 
to get old and fail (I'm assuming you're talking electrolytics).  The 
old paper electrolytics might have exhibited this behavior, but not even 
tantalums will do this.  And, if tantalum caps go, they tend to be more 
spectacular and take lots of other parts with them.

More to the point, (ceramic) chip caps that haven't been in a 
wet/moist/temp-varying, humid environment shouldn't crack and fail.

Option 4 has potential, though.

gc