[Beowulf] Thoughts on EasyBuild?

[Chris Samuel]

On 17/1/19 7:32 am, Faraz Hussain wrote:

> Some folks I work with are using EasyBuild but I am not sure what to 
> make of it.

I used easybuild for a number of years at VLSCI and at Swinburne and I 
can say that whilst the "easy" can be a bit of a misnomer it is very 
powerful and does simplify the job of managing HPC software.

You want OpenFOAM?  Pick the version, tell it to build it in robot mode 
and it'll go off and build it from the compiler all the way to the 
finished install.

What I really like is that it codifies a lot of the knowledge about 
building these applications so even if you don't use it yourself the 
easyconfigs can help for software you may be struggling to build.  The 
community around it is also strong and helpful.

They also use checksums for the files they download to confirm nothing 
went wrong, it also has the useful side-effect of catching projects who 
spot a bug in a release and then re-release it without updating the 
version number. :-)

One complication I did find is when you build things that you want to 
apply custom configuration to - like ensuring that all your OpenMPI 
versions are built with Slurm integration enabled for example - it means 
modifying all those easyconfigs in advance. I don't know if that's 
changed recently.

One other nit is that it can lead to an explosion of versions of GCC, 
various MPI's, etc because the easyconfigs encode all those.  What we 
did was use custom versions and pick a GCC v6, v7, etc version to use as 
well as a version of Python2, Python3, Perl, etc for those modules.

What I liked is that it is very flexible - I really liked the way I 
could create an easyconfig for a bundle of Python modules for a group 
and they would just load that one bundle to get everything they wanted. 
It's not embedded into the existing Python install so you can do this 
over and over again and not accidentally upgrade one groups versions of 
a module simply because another group wants a module that depends on a 
later version.

They also have a sense of humour, Kenneth Hoste the lead developer has a 
great talk called "How to make package managers cry" which goes into 
detail about how to make your software hard to install, with examples.

https://www.youtube.com/watch?v=NSemlYagjIU

Hope that's useful!

All the best,
Chris
-- 
  Chris Samuel  :  http://www.csamuel.org/  :  Berkeley, CA, USA