[Beowulf] Researchers break million-core supercomputer barrier

[Eugen Leitl]

http://www.futuretimeline.net/blog/2013/01/29.htm

Researchers break million-core supercomputer barrier

Stanford Engineering's Center for Turbulence Research (CTR) has set a new
record in computational science by successfully using a supercomputer with
more than 1 million computing cores. This was done to solve a complex fluid
dynamics problem – the prediction of noise generated by a supersonic jet
engine.

Joseph Nichols, a research associate in the centre, worked on the newly
installed Sequoia IBM Bluegene/Q system at Lawrence Livermore National
Laboratories (LLNL). Sequoia recently topped the list of the world's most
powerful supercomputers, boasting 1,572,864 compute cores (processors) and
1.6 petabytes of memory connected by a high-speed five-dimensional torus
interconnect.

Because of Sequoia's impressive numbers of cores, Nichols was able to show
for the first time that million-core fluid dynamics simulations are possible
– and also to contribute to research aimed at designing quieter aircraft
engines.

The physics of noise

The exhausts of high-performance aircraft at takeoff and landing are among
the most powerful man-made sources of noise. For ground crews, even for those
wearing the most advanced hearing protection available, this creates an
acoustically hazardous environment. To the communities surrounding airports,
such noise is a major annoyance and a drag on property values.

Understandably, engineers are keen to design new and better aircraft engines
that are quieter than their predecessors. New nozzle shapes, for instance,
can reduce jet noise at its source, resulting in quieter aircraft.

Predictive simulations – advanced computer models – aid in such designs.
These complex simulations allow scientists to peer inside and measure
processes occurring within the harsh exhaust environment that is otherwise
inaccessible to experimental equipment. The data gleaned from these
simulations are driving computation-based scientific discovery as researchers
uncover the physics of noise.

More cores, more challenges

Parviz Moin, a Professor in the School of Engineering and Director of CTR:
"Computational fluid dynamics (CFD) simulations, like the one Nichols solved,
are incredibly complex. Only recently, with the advent of massive
supercomputers boasting hundreds of thousands of computing cores, have
engineers been able to model jet engines and the noise they produce with
accuracy and speed."

CFD simulations test all aspects of a supercomputer. The waves propagating
throughout the simulation require a carefully orchestrated balance between
computation, memory and communication. Supercomputers like Sequoia divvy up
the complex math into smaller parts so they can be computed simultaneously.
The more cores you have, the faster and more complex the calculations can be.

And yet, despite the additional computing horsepower, the difficulty of the
calculations only becomes more challenging with more cores. At the
one-million-core level, previously innocuous parts of the computer code can
suddenly become bottlenecks.

Ironing out the wrinkles

Over the past few weeks, Stanford researchers and LLNL computing staff have
been working closely to iron out these last few wrinkles. This week, they
were glued to their terminals during the first "full-system scaling" to see
whether initial runs would achieve stable run-time performance. They watched
eagerly as the first CFD simulation passed through initialisation then
thrilled as the code performance continued to scale up to and beyond the
all-important one-million-core threshold, and as the time-to-solution
declined dramatically.

"These runs represent at least an order-of-magnitude increase in
computational power over the largest simulations performed at the Center for
Turbulence Research previously," said Nichols. "The implications for
predictive science are mind-boggling."

A homecoming

The current simulations were a homecoming of sorts for Nichols. He was
inspired to pursue a career in supercomputing as a high-school student when
he attended a two-week summer program at Lawrence Livermore computing
facility in 1994 sponsored by the Department of Energy. Back then, he worked
on the Cray Y-MP, one of the fastest supercomputers of its time. "Sequoia is
approximately 10 million times more powerful than that machine," Nichols
noted.

The Stanford ties go deeper still. The computer code used in this study is
named CharLES and was developed by former Stanford senior research associate,
Frank Ham. This code utilises unstructured meshes to simulate turbulent flow
in the presence of complicated geometry.

In addition to jet noise simulations, Stanford researchers are using the
CharLES code to study advanced-concept scramjet propulsion systems, used in
hypersonic flight at many times the speed of sound.