Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

WORK IN PROGRESS!

Table of Contents

Introduction

EasyBuild is a framework for building and managing software on HPC systems. It automates compiling, installing, and configuring scientific applications while handling dependencies, toolchains, and environment modules.

A key concept is easyconfig files — text-based receipts that define build parameters, dependencies, versions, and toolchains. By using these receipts, EasyBuild installs software along with all its dependencies, ensuring a consistent, reproducible, and optimized HPC software environment.

How it works? The dependency resolution mechanism constructs a full dependency graph for the software package(s) being installed, after which a list of dependencies is composed for which no module is available yet. Each of the retained dependencies will then be built and installed, in the required order as indicated by the dependency graph.

This page This tutorial shows how to install IMAS ecosystem using EasyBuild. The installation will be performed on the Gateway cluster but it can be done anywhere.

...

Before you start, make sure that:

  • Easybuild is installed and publicly available via module system

  • Your

    Easybuild configuration file exists and is properly configured

  • IMAS related receipts are

    downloaded from all repositories being used
  • Your TMP  directory is set up to use $ITMWORK

Software installation receipts

Sources of the receipts (aka 'easyconfig' files):

  • EasyBuild built-in receipts
  • EasyBuild receipts repository
  • Gateway specific receipts
  • 'Limbo' of the IMAS receipts
  • ITER repository of the IMAS receipts 

Downloading configuration (EB) files

  • downloaded 

  • All modules are purged and EasyBuild module is the only module loaded:

    Code Block
    module purge
    module load EasyBuild
  • $EASYCONFIG_PATHS system variable is set to point to all directories with easyconfig files 


Downloading easyconfigs

To allow the users to seamlessly switch between EUROfusion and ITER clusters, the environment that is built on the Gateway mimics (to some extent) the one of the ITER SDCC machines. 

To achieve this a receipts from the ITER cluster are utilised to install software on the GW.

Downloading configuration (EB) files
Source SDCC: /work/imas/opt/EasyBuild/ebfiles_repo/ 

Code Block
languagebash
titleSynchronisation of the ITER receipts
rsync -av --delete -e "ssh -J <user>@sdcc-login.iter.org"  <user>@sdcc-login02.iter.org:/work/imas/opt/EasyBuild/ebfiles_repo/ ./all_iter/-receipts/
System variable EASYCONFIGS_PATH

Set system variable EASYCONFIGS_PATH  that points to the directory with receiptsClone these repositories and  specify the paths to them in the --robot  in the next step.

Code Block
languagetext
titleclone repo
TBD

Remember to change the --robot and pass paths to your  and TBD  directories. 

Installation of IMAS HLIs - 5.2.0dev

Code Block
languagebash
titleIMAS
eb IMAS-AL-Cpp/IMAS-AL-Cpp-5.2.0dev-foss-2020b-DD-3.40.1.eb --allow-modules-tool-mismatch --robot=/pfs/work/g2pbloch/test_instruction/easybuild-imas-files:/pfs/work/g2pbloch/test_instruction/eb_files_repo 
eb IMAS-AL-Fortran/IMAS-AL-Fortran-5.2.0dev-foss-2020b-DD-3.40.1.eb  --allow-modules-tool-mismatch --robot=/pfs/work/g2pbloch/test_instruction/easybuild-imas-files:/pfs/work/g2pbloch/test_instruction/eb_files_repo 
eb IMAS-AL-Java/IMAS-AL-Java-5.2.0dev-foss-2020b-Java-11-DD-3.40.1.eb  --allow-modules-tool-mismatch --robot=/pfs/work/g2pbloch/test_instruction/easybuild-imas-files:/pfs/work/g2pbloch/test_instruction/eb_files_repo 
eb IMAS-AL-Python/IMAS-AL-Python-5.2.0dev-foss-2020b-DD-3.40.1.eb  --allow-modules-tool-mismatch --robot=/pfs/work/g2pbloch/test_instruction/easybuild-imas-files:/pfs/work/g2pbloch/test_instruction/eb_files_repo 

Remember to change the --robot and pass paths to your easybuild-files  and easybuild-imas-files  directories. 

Installation of IMAS

IMAS bundle

...

  • IMAS-AL-Cpp
  • IMAS-AL-Fortran
  • IMAS-AL-Java
  • IMAS-AL-Matlab
  • IMAS-AL-Python

...

  • IDStools

...

export EASYCONFIGS_PATH=`pwd`/iter-receipts


Warning

This simple mechanism will be replaced with a more complex one, where receipts would be maintained in GIT repositories.


Installation of software components

Searching for receipts

EasyBuild needs a receipt to install any software component. The following command can be used to look for receipts:

Code Block
eb --search <software_name> --robot-paths=$EASYCONFIG_PATHS

Info

 --robot-paths=<path1>:<path2>  configuration option supports prepending or appending to the default robot search path:

  • To prepend one or more paths, a list of paths followed by a ‘:’ should be specified.

  • To append one or more paths, a list of paths preceded by a ‘:’ should be specified.

See EasyBuild documentation for details and examples

A <software_name>  could be only part of the name being searched. Additionally EasyBuild search is case insensitive. E.g. "dox"  string can be provided while looking for "Doxygen" 

Code Block
shell> eb --search dox --robot=$EASYCONFIG_PATHS

== found valid index for /gw/swimas/easybuild/opt/software/EasyBuild/5.0.0/easybuild/easyconfigs, so using it...
 * /afs/eufus.eu/user/g/g2bpalak/work/easybuild/all_iter/Doxygen/Doxygen-1.9.8-GCCcore-13.2.0.eb
 * /afs/eufus.eu/user/g/g2bpalak/work/easybuild/all_iter/Doxygen/Doxygen-1.13.2-GCCcore-13.2.0.eb
...
 * /gw/swimas/easybuild/opt/software/EasyBuild/5.0.0/easybuild/easyconfigs/d/Doxygen/Doxygen-1.9.8-GCCcore-13.2.0.eb
 * /gw/swimas/easybuild/opt/software/EasyBuild/5.0.0/easybuild/easyconfigs/d/Doxygen/Doxygen-1.11.0-GCCcore-13.3.0.eb  

Usually receipt name uses following syntax:

Code Block
<software name>-<software version>-<toolchain>-<toolchain version>

e.g.:
Doxygen-1.9.8-GCCcore-13.2.0.eb

Installation


Code Block
languagebash
titleIMAS
eb <receipt name>.eb --robot --robot-paths=$EASYCONFIG_PATHS

e.g.:
eb Doxygen-1.9.8-GCCcore-13.2.0.eb --robot --robot-paths=$EASYCONFIGS_PATHS 

Info
titleHints
  •  --robot/-r command line option enables dependency resolution See EasyBuild documentation for details
  • It is strongly advised to change temporary directories to use $ITMWORK

  • It is recommended to run before actual installation:
    • eb with option -missing-modules/-M  to check what dependencies will be installed

    • eb  with option --dry-run/-D  or --extended-dry-run/-x,  to check what and how will be installed

  • All 'massive' builds should be performed on the worker nodes 
  • See EB help for the other useful eb switches

Installation of the 'production' software

To install the 'production' version of the software belonging to IMAS ecosystem, a few extra options have to be added to point out EB to the dedicated public directory /gw/swimas/easybuild  


Code Block
languageplain
--prefix='/gw/swimas/easybuild/opt' 
--installpath-modules='/gw/swimas/easybuild/etc'
--ignore-checksums (optional)


# TO BE DISCUSSED: Accessing sources while debugging. Should 'source' directory be public, as well? 

Installation of the IMAS bundle

The IMAS bundle is (usually) provided by ITER and loads all IMAS High Level Interfaces and related modules:

Content of IMAS bundle

IMAS bundle consist of:

  • IMAS High Level Interfaces
    • IMAS-AL-Cpp 
    • IMAS-AL-Fortran 
    • IMAS-AL-Java 
    • IMAS-AL-Matlab 
    • IMAS-AL-Python 
  • Related modules:
    • IDStools 

IMAS bundle receipts

The IMAS bundle receipts name usually contains the 'IMAS' string, Data Dictionary version, date of release and toolchain version, e.g.: IMAS-3.42.0-2024.09-foss-2023b.eb  or IMAS-4.0.0-2024.12-foss-2023b.eb  

Before installation search for a proper IMAS bundle version

Code Block
shell> eb --search IMAS-3 --robot-paths=$EASYCONFIG_PATHS

== found valid index for /gw/swimas/easybuild/opt/software/EasyBuild/5.0.0/easybuild/easyconfigs, so using it...
...
 * /gss_efgw_work/work/g2bpalak/easybuild/iter-receipts

...

/IMAS/IMAS-3.42.0-2024.08

...

-foss-2023b.eb
 * /gss_efgw_work/work/g2bpalak/easybuild/iter-

...

receipts/IMAS/IMAS-3.42.0-2024.08-intel-2023b.eb
...
 * /gss_efgw_work/work/g2bpalak/easybuild/iter-receipts/IMAS/IMAS-3.42.0-2024.09-foss-2023b.eb
 * /gss_efgw_work/work/g2bpalak/easybuild/iter-receipts/IMAS/IMAS-3.42.0-2024.

...

09-intel-2023b.eb
...
 


Warning

Every IMAS High Level Interface depends on the IMAS Access Layer Core ( IMAS-

...

AL-Core

Make sure that all IMAS HLIs use exactly the same version of  IMAS-AL-Core and other common dependencies !!!

Installation

Once the receipt with a proper version of Data Dictionary and toolchain is found, the IMAS bundle can be installed as any other software:

Code Block
eb <IMAS bundle receipt>.eb --robot --robot-paths=

...

$EASYCONFIG_PATHS 

e.g.:
eb IMAS-3.42.0-2024.08.1-intel-2023b.eb --robot --robot-paths=$EASYCONFIG_PATHS

imasenv module

How to use IMAS

After installation you can load IMAS  modules i.e:

Code Block
languagebash
titleLoad module
module use {prefix}/modules/all
module load IMAS-AL-Cpp/5.2.0dev-foss-2020b-DD-3.40.1
module load IMAS-AL-Fortran/5.2.0dev-foss-2020b-DD-3.40.1
module load IMAS-AL-Java/5.2.0dev-foss-2020b-Java-11-DD-3.40.1
module load IMAS-AL-Python/5.2.0dev-foss-2020b-DD-3.40.1

How it works

  • eb - EasyBuild
  • IMAS-AL-Cpp/IMAS-AL-Cpp-5.2.0dev-foss-2020b-DD-3.40.1.eb - path to EasyBuild configuration file

  • --allow-modules-tool-mismatch - flag
  • --robot - paths to configuration files

Installation of the 'production' version

...

languageplain



...

imasenv module

Adding software components to existing installation

  • Set  MODULEPATH system variable 
    Code Block
     export MODULEPATH=/gw/swimas/easybuild/etc/all
  • Add --use-existing-modules  switch to eb  command

Useful tips

Installing dependencies as hidden modules

Warning

TO BE CHECKED !!!!!

{{--hide-deps}}

Purge loaded modules before using Easybuild

Warning

EasyBuild requires no modules to be loaded ! 

Please execute:

Code Block
module purge

 

The key feature of the EasyBuild is the ability to install a software component with all its dependencies. Unfortunatlely EasyBuild installs module not only for the requested software but for every dependency as well.

It results with the number of available modules growing with every installation, becoming less and less hard to navigate for the users. 

There are few options to hide modules, unfortunately every of them has its disadvantages, 

  • eb --hide-deps=<moduleName1>,<moduleName2>,..., <moduleNameN>   option of eb requires for all modules to be explicitly listed

  • eb --hide-all-dependencies -  implemented in working branch but PR#3217 has been not merged since 2020
  • module-hide <module-name> - Lmod module system command (see documentation). . Modules have to be hide one by one.

Change the default tmp directory Change the default tmp directory
Anchor
tmp_dir_change
tmp_dir_change

Easybuild produces a lot of temporary content (logs, intermediate build files, etc)

...