[Beowulf] copying big files (Henning Fehrmann)

[David Mathog]

Perry E. Metzger wrote:
> Is there a reason bittorrent isn't suited to this application?

That would probably work too.  

I also recall reading about "broadcast"
methods for doing this sort of distribution.  I originally wrote nettee,
which is a derivative of "dolly", because Ghost (which uses a broadcast 
method) was dreadfully slow.  This was many moons ago, and there is
probably much better broadcast software around today, even in Ghost
itself.

nettee has a few whistles and bells for this "data push" application. 
bittorrent is more of a "data pull" application.  So on the top node,
nettee can easily tell you when all of the clients are in the chain (or
not), monitor the progress of the download, and indicate if anything
went wrong.  To do that with the bittorrent clients I assume you would
have to wrap them in scripts to report the desired information back to a
central point.  It looks to me like it would be easiest to have each
client report its own completion status, perhaps through a wget with a
carefully formed URL to a cgi script on a central web server.  Because
torrents are "pull", determining from the sending side(s) if a
particular torrent client has successfully completed a file download
looks like a nightmare.  It would seem to require examining the log
information on (potentially all) of the other clients.

Regards,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech