Load balancing and other issues

[Mike Perry]

Thus spake Erik Arjan Hendriks (hendriks@cesdis1.gsfc.nasa.gov):

> These file issues (transparency issues, really) are a big deal.
> (Bigger than they seem at first (to me anyway)) I think what you're
> proposing here is essentially Mosix.  Mosix is cool because it handles
> all these things.  The reason we don't use Mosix here is because it
> provides these features at an unacceptably high (IMHO) cost.
> 
> Still if you want to replace regular fork, Mosix is the only way to
> go.  Mosix (in a nutshell) runs processes remotely but performs all
> the processes's syscalls on the node where the process originated
> (including file IO) so that it appears nothing has changed.
I still don't see what's wrong with my suggestion of replacing fork, and
having it call bproc_rfork if the file descriptors are stdio only, and the
load is above a certain threshold.. This would still be transparent for the
most part, and with this we have the advantage that programs don't need to be
rewritten or modified to take advantage of the cluster to some extent.

> If that's what you're looking for, you should give it a try it.  It's
> even GPL'ed now.
> 
> <Philosophical Rambling>
> 
> The way I see it, we'd like the cluster to look like a single machine
> kind of like Mosix provides but we don't want the take performance hit
> that Mosix will incur for many (if not most) applications.
> 
> Here's one way to look at the whole thing that I like:
> 
> There are basically two extremes here.  Mosix is at one end.  PVM/MPI
> distributing processes with rsh is on the other.  I believe there's a
> happy medium in the middle there somewhere.
> 
> Mosix it working its way towards the middle.  They're talking about
> providing temp files, etc on local file systems.  (i.e. where the
> process is really running, not where it thinks its running.)  They
> also talk about "migratable sockets".  (I'd really like to see how
> they're going to do that.  That'll be a really good trick if they pull
> it off.)
> 
> bproc starts on the PVM/MPI+rsh end is working its way in the other
> direction.  It gives you a way to move + create new processes but
> makes no promises about sockets or files.  The move is not
> transparent.  On the other hand, there's NO performance penalty on
> file IO or network IO.
> 
> Being the author of bproc, I'm obviously biased towards my approach.
> You're all entitled to make your own decisions as to whether or not
> I'm some kind of nut-job.
> 
> </Philosophical Rambling>
See, I do agree for the most part with your Philosophical Rambling. However, I 
do believe that it would be relatively easy to move closer to the middle of the
road you describe by providing a new fork(). 

As I see it, the system would incur very very little extra overhead, and in 
situations where we can't yet handle the file descriptors or pipes, no biggie,
we simply revert back to local fork(). I too think Mosix introduces a bit too
much overhead, especially after you pointed out that all the system calls are 
executed on the head node (something I didn't get from my quick perusal the 
docs on their website).

> That being said, bproc's IO forwarding does need serious work.  I've
> actually cut back on it a bit in my latest development version because
> it was too sketchy.  You may have noticed goofiness if you've tried to
> pipe stuff around.  A replacement is in the works there.
>