[Beowulf] [EXTERNAL] Re: ***UNCHECKED*** Re: Spark, Julia, OpenMPI etc. - all in one place

[Lux, Jim (US 7140)]

From: Beowulf <beowulf-bounces at beowulf.org> on behalf of Oddo Da <oddodaoddo at gmail.com>
Date: Wednesday, October 14, 2020 at 11:09 AM
To: Michael Di Domenico <mdidomenico4 at gmail.com>
Cc: "beowulf at beowulf.org" <beowulf at beowulf.org>
Subject: [EXTERNAL] Re: [Beowulf] ***UNCHECKED*** Re: Spark, Julia, OpenMPI etc. - all in one place

On Wed, Oct 14, 2020 at 1:24 PM Michael Di Domenico <mdidomenico4 at gmail.com<mailto:mdidomenico4 at gmail.com>> wrote:
On Wed, Oct 14, 2020 at 11:53 AM Oddo Da <oddodaoddo at gmail.com<mailto:oddodaoddo at gmail.com>> wrote:

i'll agree that in some respects software engineering has gotten
better in the last 20yrs, but it's subjective.  there are a lot of
things that have gotten better and there are a lot of things that are
much worse.  but i'm not sure you can apply that statement to HPC.
HPC code doesn't churn like business code or even more volatile cloud
code.  HPC code is usually written to solve something specific and
gets incremental updates over time.  usually that something specific
hasn't changed the last 20yrs (think physics/chemistry) the models we
use to describe or solve the problems likely have, but the underlying
code is probably basically the same with tweaks along the way to fit
the new model.

I disagree. I think yes, there is old code that does not churn but there are always new people/grad students coming into the field. They too are being pointed in the same direction of how to do things, which is what we are discussing here ;-)



Yes, there are new people coming in.  But they’re not developing new modeling codes from scratch – they’re typically “improving” the existing codes.   And as Michael pointed out, there’s significant resistance to change when you’ve got a code base that is known, debugged, and has known warts.    Big changes occur when a modeling paradigm shift occurs.  And those are not real common.





an evolution to MPI.  but it goes back to technical debt.  to re-write
something in chapel is non-trivial and may not be worth the time.
writing something new and choosing chapel is really left up to the
developer.  i have some chapel users here and there, but they're a
minority.  and since chapel is largely only found on cray machines its
exposure is low

It seems that in your world nothing new ever gets written? You are talking only about re-writes ;).

For many HPC science applications this is true – Physics models change very slowly. Once you have a gridded finite element model, it “just works” and there’s not huge demand for new modeling approaches.

Changes in language usage usually occur because of a technical problem with the existing code that means it just cannot be modified.  More than one ambitious person has said “let’s re-do program X from Fortran to C++ or Java or Ada or Python” and found it a bigger challenge than expected.  There is a definite preference for the devil you know.





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20201014/defab826/attachment-0001.html>