[Beowulf] Advice on 4 CPU node configurations

[Prentice Bisbal]

Craig Tierney wrote:
> On 2/9/11 7:38 AM, Prentice Bisbal wrote:
>> I don't think Numascale/ScaleMP has much of a cost advantage anymore.
>>
>> About 6 months ago, I purchased a couple of Dell PowerEdge R815s with
>> 128 GB of RAM and 32 cores. We looked at similar RAM configurations a
>> couple of years ago. and the cost premium for that much RAM was
>> prohibitive (the price for additional RAM seemed to go up exponentially
>> with the amount of RAM) so we stayed at 32 GB.  This time around, the
>> premium for the additional RAM seemed marginal. Not sure how large you
>> can go and keep that "marginal" relationship, but it's much larger than
>> it was a couple of years ago.
>>
> 
> There is only so much memory you can put in a box, regardless of the
> cost.  ScaleMP/Numascale let you get around that issue.  What if I need
> a TB of RAM?  Yes, I might be able to convert my algorithms, but if I can
> add $3k per node to get cache-coherent memory, why not go that route?

Agreed. I was going to add the caveat that there are limits to how much
RAM you can add to a single box, or there eventually be a point the
amount of RAM in a single box vs. price is no longer nonlinear, but then
I got lazy. ;)

> 
> 
>> And don't forget to figure in the cost of the interconnect. I don't know
>> what Numascale requires, but ScaleMP require IB, which can add
>> significant costs if you don't have an existing IB fabric to use.
>>
> 
> ScaleMP is a software solution, Numascale is a cache-coherent hardware solution.
> The list cost of the 3D torus version Numascale card with 4GB Ram (highest end card).
> Cables are similar to IB, about $120 each.
> 
> It would be really cool to build SMP system to sit next to my cluster.  When users
> need large memory, or want to do things with UPC or other similar languages,
> an SMP system like this would be handy.
> 
> - I am not advocating for Numascale, or any other products here.  I am just
> trying to provide information to the community.
> 
> Craig
> 
> 
> 
>> Prentice
>>
>>
>> Douglas Eadline wrote:
>>> Although I wrote about the SMP and the Numascale hardware,
>>> I have not yet had a chance to use it.
>>>
>>> To me there are two issues worth considering. First,
>>> if you need a big SMP this might be a low
>>> cost solution. Second, should your application
>>> not scale best in a node-based ccNuma system (ScaleMP
>>> or Numascale), MPI is still an option. Indeed, no need
>>> to rewrite the codes. And, management is probably
>>> much easier.
>>>
>>> Of course for large systems, clusters work best.
>>>
>>> --
>>> Doug
>>>
>>>
>>>> On Thu, 2011-02-03 at 09:45 +0000, Hearns, John wrote:
>>>>> Also look at ScaleMP and Numascale
>>>>> Here's a damn good article from Doug Eadline:
>>>>> http://www.linux-mag.com/id/7947
>>>>>
>>>>> I must admit though I don't know how far a budget of 30K takes you
>>>>> there!
>>>> My understanding is that the Numascale cards cost "a bit more" than IB
>>>> cards but you don't need a switch (it's a 3D taurus I think) so it would
>>>> be worth looking into.
>>>>
>>>> Rob
>>>>
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> 

-- 
Prentice Bisbal
Linux Software Support Specialist/System Administrator
School of Natural Sciences
Institute for Advanced Study
Princeton, NJ