Load balancing and other issues

[Erik Arjan Hendriks]

On Wed, 4 Aug 1999, Mike Perry wrote:

> Thus spake Erik Arjan Hendriks (hendriks@cesdis1.gsfc.nasa.gov):
> 
> I still don't see what's wrong with my suggestion of replacing fork, and
> having it call bproc_rfork if the file descriptors are stdio only, and the
> load is above a certain threshold.. This would still be transparent for the
> most part, and with this we have the advantage that programs don't need to be
> rewritten or modified to take advantage of the cluster to some extent.

I think replacing fork with bproc_rfork() (at least bproc rfork as it
exists now, more or less) would have a LOT of hidden gotchas.  I'll
try and give a few examples of what I think would be serious problem
cases.

* $ vi /etc/passwd
  Shells have to fork before they exec.  If bash rforked here we'd end
  up editing the password file on the wrong machine.  You couldn't catch
  this by looking at open fd's either.  The only place to find out that
  bash is intending to run vi w/ the argument '/etc/passwd' is somewhere
  in bash's memory space.  Where it's extremely hard to look in a
  reliable fashion.

  You could think up a million examples along these lines 'cat
  /tmp/debug.out', 'gdb foo core'.  A lot of this could be ok if users
  operated on files in their home dir which was NFS mounted but 'ok most
  of the time' isn't good enough if you're gonna mess with a basic
  system function like fork.  Also, if the binary you wanted to run
  wasn't available on NFS, you'd run into trouble too.


* Host names, IP addresses, etc could change.  For example, you'd
  never want sshd to get started anywhere but the front end.  Building
  a list of things to "not run remotely" wouldn't work because of the
  fork then exec issue mentioned above.


* Daemons, etc which are often hard wired to look in /etc for their
  configuration info would always fail if they got started on another
  machine.


* Any program using SysV IPC would be in extreme peril.  (We might be
  able to bring in DIPC to fix some of this but I haven't looked closely
  enough at this know if it could be a generic solution.)


* Programs using (or planning on using) shared mmaps would break.

I guess the basic point is that it's bad to switch file systems on
processes w/o their consent.

If you want to experiment along the lines of changing fork (without
jumping in head first and changing the syscall), I'd suggest using
LD_PRELOAD to replace the fork function for some processes.  Most
everything is dynamically linked these days so that should work fairly
well.

- Erik
------------------------------------------------------------
Erik Hendriks
hendriks@cesdis.gsfc.nasa.gov