[Beowulf] Scaling issues on Xeon E5-2680

[Joe Landman]

On 02/28/2016 10:27 AM, Jon Tegner wrote:
> Hi,
>
> have issues with performance on E5-2680. Each of the nodes have 2 of 
> these 12 core CPUs on SuperMicro SuperServer 1028R-WMR (i.e., 24 cores 
> on each node).
>
> For one of our applications (CFD/OpenFOAM) we have noticed that the 
> calculation runs faster using 12 cores on 4 nodes compared to when 
> using 24 cores on 4 nodes.
>
> In our environment we also have older AMD hardware (nodes with 4 CPUs 
> with 12 cores each), and here we don't see these strange scaling issues.
>
> System is CentOS-7, and communication is over FDR Infiniband. BIOS is 
> recently updated, and hyperthreading is disabled.
>
> Feel a bit lost here, and any hints on how to proceed with this are 
> greatly appreciated!

Need lots more info.  What does

     numastat

report for example?  Are you running openfoam in a way to enforce 
affinity?  Do you have hugepages enabled/disabled?  Is openfoam using 
RDMA or do you have it running IPoIB?

>
> Thanks,
>
> /jon
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit 
> http://www.beowulf.org/mailman/listinfo/beowulf

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
e: landman at scalableinformatics.com
w: http://scalableinformatics.com
t: @scalableinfo
p: +1 734 786 8423 x121
c: +1 734 612 4615