[Beowulf] memory bandwidth scaling

Fri Oct 2 08:28:03 PDT 2015

Hi All,
The main problem here is that in modern systems the memory bus often becomes a scarce resource with a lot of processes competing for it.
And for many programs it is beneficiary not to use all cores in the system as John points out.

However, one interesting aspect of the memory bus contention is that many programs experience quite a large degree of slowdown even at moderate memory bandwidth usage levels. There are examples where two programs from the NPB benchmarks which only utilize 1/3rd of the available memory bandwidth but still they are slowed down quite significantly. There are also some examples of programs which experience a super linear slowdown, caused by memory bandwidth sharing.

The problem however is to determine the correct number of processes to run on each node or if the workload consists of many programs how these should be combined in order to achieve the highest resource usage. If you have one type of nodes and one program you can just execute the program with using a few different allocation schemes and see which one works best. However, this approach does not scale.

Now I will shamelessly plug a recent article addressing this problem:
Andreas de Blanche and Thomas Lundqvist, "Addressing Characterization Methods for Memory Contention Aware Co-scheduling", The Journal of Supercomputing: Volume 71, Issue 4, Page 1451-1481, Springer Verlag, 2015.

Allocating processes to nodes in order to avoid any, not only memory or memory bus, resource exhaustion is becoming more and more important.

Best Regards
//Andreas

-----Original Message-----
From: Beowulf [mailto:beowulf-bounces at beowulf.org] On Behalf Of John Hearns
Sent: den 2 oktober 2015 10:31
To: Orion Poplawski <orion at cora.nwra.com>; Beowulf List <beowulf at beowulf.org>
Subject: Re: [Beowulf] memory bandwidth scaling

Orion, that is a good question.

I have to say though that we are heading for a world where you don't get to make that choice - the lowest core count SKUs will just get higher and higher.
Couple this with the current way many folks specify systems with a minimum amount of RAM per core (which is quite sensibly based on looking at their applications and how much memory they consume!) also is leading us to specify higher memory for two processor nodes, and to keep a balanced DIMM configuration too. I was discussing this with some researchers only yesterday.

I will throw something open to the floor:
You don't HAVE to use all the cores in a CPU socket.
Yes, I realise that you have paid for them and that they are a resource which is available.

I am perhaps not explaining myself very well, and I imagine lots of sites already allocate different numbers of cores per node depending on job type.

We may be looking a getting a couple new compute nodes.  I'm leery though of going too high in processor core counts.  Does anyone have any general experiences with performance scaling up to 12 cores per processor with general models like CM1/WRF/RAMS on the current crop of Xeon processors?

--
Orion Poplawski
Technical Manager                     303-415-9701 x222
NWRA, Boulder/CoRA Office             FAX: 303-415-9702
3380 Mitchell Lane                       orion at nwra.com
Boulder, CO 80301                   http://www.nwra.com
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
#####################################################################################
Scanned by MailMarshal - M86 Security's comprehensive email content security solution.
#####################################################################################
Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Employees of XMA Ltd are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to company policy and outside the scope of the employment of the individual concerned. The company will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising. XMA Limited is registered in England and Wales (registered no. 2051703). Registered Office: Wilford Industrial Estate, Ruddington Lane, Wilford, Nottingham, NG11 7EP _______________________________________________
Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf