[Beowulf] Syslog Server-Traffic

[Mark Hahn]

> I'm researching setting up a cluster and I'm curious as to whether or
> not it's a good idea to set up a syslog server.   The question I have

depends on how much syslog activity you have, and whether you care 
to look at it (in one spot).  there _are_ scalable and more robust 
system/event logging approaches, but plain old syslog is pretty good.

> is whether the traffic created from logging is going to slow down my
> network to the point of poor performance?

a syslog message is normally a smallish UDP packet - say 100 bytes.
if you have 200 nodes each doing 5 per second, that's still only 
100KB/s - a pretty small fraction of a server's gigabit bandwidth.
and if you actually have 5/s, something's probably wrong...

> experiences with logging.  The cluster will have a management network
> and a computational network.

I'm always skeptical about this advice - it's obvious that it might be good
in cases where a node sustains a nontrivial stream of management traffic
(say, NFS traffic using jumbo frames) which would interfere with possible 
latency-sensitive/small MPI packets.

but how often does that happen?  consider that with a non-jumbo gigabit net,
a full packet is only 15 us more than the ~40 or so for a minimal one.
further, I observe MPI codes mostly getting packed into full nodes,
and not interfering with themselves much (distinct MPI and file IO phases
to the program.).

I could more readily imagine segregating traffic to two nets based on 
packet size or TOS.  or bonding them in the first place.  in any case,
I think you'd have to work pretty hard to generate enough syslog traffic
to matter much.

> Greater than 64 Nodes, Yes.
> Greater than 64 and less than 128, Maybe?
> Greater than 128 No

for a smallish cluster like 64 nodes, I don't think I'd worry about 
syslog even if the net were just 100bT.  for going above 200 nodes,
I'd probably try to do some measurements and extrapolation, but the 
numbers above make it look minor.