[Beowulf] Power per area

[Stu Midgley]

It’s really a matter of what technology you want.

KNL we run at about 320W so 800/320*3=7.5TF
V100 we run at about 300W so 800/300*6=16TF

BUT our KNL’s sit turbo boosted nearly all the time so more like 3.5TF
each.




On Wed, 11 Mar 2020 at 7:04 pm, Scott Atchley <e.scott.atchley at gmail.com>
wrote:

> Hi Stu,
>
> The rolling weight is only an issue when moving equipment during
> installation/removal. It causes point loads and we typically lay steel
> plate down to spread the load over multiple tiles.
>
> What is your performance density (FLOPS/ft^2) in Houston if you do not
> mind me asking?
>
> Scott
>
> On Tue, Mar 10, 2020 at 8:30 PM Stu Midgley <sdm900 at gmail.com> wrote:
>
>> Immersion cooling makes a lot of sense :)
>>
>> We run it on the 21st floor of a building in Kuala Lumpur, on the 1st
>> floor in Perth and on a slab-on-ground in Houston.
>>
>> The tanks+fluid are light.  When full of equipment, about 1.2 tonnes
>> spread over about 2m2.
>>
>> In Houston, we have the lowest rated raised floor - since the tanks
>> spread the load across multiple tiles/floor stands, and there is no rolling
>> weight (its spread evenly over 2m2)
>>
>> So ~ 2200lbs/10sqft ie. about 220lbs/sqft .  We run a power density of
>> 8.5kW/sqm (~800W/sqft) across our whole DC (which includes all the internal
>> white space/CRAC space etc).
>>
>> We cool the whole facility with evaporation (compressor cooling is only
>> for comfort cooling).
>>
>> We have hit a PUE of 1.045 in Houston...   and 1.035 in Perth :)
>>
>> Come and have a look at our Houston DC :)
>>
>>
>>
>> On Wed, Mar 11, 2020 at 3:37 AM Scott Atchley <e.scott.atchley at gmail.com>
>> wrote:
>>
>>> Hi everyone,
>>>
>>> I am wondering whether immersion cooling makes sense. We are most
>>> limited by datacenter floor space. We can manage to bring in more power (up
>>> to 40 MW for Frontier) and install more cooling towers (ditto), but we
>>> cannot simply add datacenter space. We have asked to build new building and
>>> the answer has been consistently "No."
>>>
>>> Summit is mostly water cooled. Each node has cold plates on the CPUs and
>>> GPUs. Fans are needed to cool the memory and power supplies and is captured
>>> by rear-door heart exchangers. It occupies roughly 5,600 ft^2. With 200 PF
>>> of performance and 14 MW of power, that is 36 TF/ft^2 and 2.5 kW/ft^2.
>>>
>>> I am wondering what the comparable performance and power is per square
>>> foot for the densest, deployed (not theoretical) immersion cooled systems.
>>> Any ideas?
>>>
>>> To make the exercise even more fun, what is the weight per square foot
>>> for immersion systems? Our data centers have a limit of 250 or 500
>>> pounds per square foot. I expect immersion systems to need higher loadings
>>> than that.
>>>
>>> Thanks,
>>>
>>> Scott
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>>> To change your subscription (digest mode or unsubscribe) visit
>>> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
>>>
>>
>>
>> --
>> Dr Stuart Midgley
>> sdm900 at gmail.com
>>
> --
Dr Stuart Midgley
sdm900 at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20200311/b4180e8f/attachment-0001.html>