[Beowulf] dedupe filesystem

[Loic Tortay]

Mark Hahn wrote:
[...]
> 
> this seems like a bad design to me.  I would think (and I'm reasonably
> familiar with Lustre, though not an internals expert) that if you're 
> going to touch Lustre interfaces at all, you should simply add cheaper, 
> higher-density
> OSTs, and make more intelligent placement/migration heuristics.  I guess 
> that CEA already has a vast investment in some existing HSM, so can't do 
> this.
> 
Last time I talked with one of the people in charge of this at CEA, they
had something like 5 petabytes of Lustre based disk storage and, if I'm
not mistaken, a bit more in their HSM (HPSS from IBM).
That was about 18 months ago, I'm pretty sure they now have much more
Lustre storage.

Similar HSM interfaces for other cluster filesystems have been available
for some time.  For instance, QFS w/ SAM-FS, GPFS w/ HPSS (in several
forms), GPFS w/ "external pools" and probably others (like StorNext).


Regarding your question about tapes, these are still widely used in HEP 
data processing where data volumes are in the petabytes scale for the 
older experiments and will be, at least, in the tens of petabytes scale 
for the LHC experiments at CERN.


Loïc.
-- 
|   Loïc Tortay <tortay at cc.in2p3.fr>  -     IN2P3 Computing Centre     |