WORK IN PROGRESS!


Introduction

This tutorial shows how to install IMAS ecosystem using EasyBuild. The installation will be performed on the Gateway cluster but it can be done anywhere.

Prerequisites

Before you start, make sure that:

  • Easybuild is installed and publicly available via module system
  • Your Easybuild configuration file exists and is properly configured
  • IMAS related receipts are downloaded from all repositories being used


Software installation receipts

Sources of the receipts (aka 'easyconfig' files):

  • EasyBuild built-in receipts
  • EasyBuild receipts repository
  • Gateway specific receipts
  • 'Limbo' of the IMAS receipts
  • ITER repository of the IMAS receipts 

Downloading configuration (EB) files

/work/imas/opt/EasyBuild/ebfiles_repo/

Synchronisation of the ITER receipts
rsync -av --delete -e "ssh -J <user>@sdcc-login.iter.org"  <user>@sdcc-login02.iter.org:/work/imas/opt/EasyBuild/ebfiles_repo/ ./all_iter/


Clone these repositories and  specify the paths to them in the --robot  in the next step.

clone repo
https://gitlab.eufus.psnc.pl/g2pbloch/easybuild-files 
https://gitlab.eufus.psnc.pl/g2pbloch/easybuild-imas-files 

# [Bartek Palak] Probably only the ITER repository of EB receipts is required (ssh://git@git.iter.org/imex/easybuild-easyconfigs.git) To be checked.

Installation of Data Dictionary (DD)

Run the DD installation (in the easybuild-imas-files directory):

IMAS install
eb Data-Dictionary/Data-Dictionary-3.40.1-GCCcore-10.2.0.eb --allow-modules-tool-mismatch --robot=/pfs/work/g2pbloch/test_instruction/easybuild-imas-files:/pfs/work/g2pbloch/test_instruction/eb_files_repo  --http-header-fields-urlpat="^https://git.iter.org::$HOME/.config/easybuild/secret.txt"

Remember to change the --robot and pass paths to your easybuild-files  and easybuild-imas-files  directories. 

Configuring paths in IMAS configuration (EB) files

After DD installation change path in this line in the HLI easyconfigs to your  {prefix}/software/Data-Dictionary/3.40.1-GCCcore-10.2.0/dd_3.40.1/

DD path
 preconfigopts = 'export IMAS_VERSION="0.0.0" &&  export IMAS_PREFIX="/pfs/work/g2pbloch/test_instruction/software/Data-Dictionary/3.40.1-GCCcore-10.2.0/dd_3.40.1/" &&' 

In Python configuration files IMAS-AL-Python, IMAS-AL-Python-Bindings  you have to change path  to python to $prefix/software/Python/3.8.6-GCCcore-10.2.0/ 

Installation of IMAS HLIs - 5.2.0dev

IMAS
eb IMAS-AL-Cpp/IMAS-AL-Cpp-5.2.0dev-foss-2020b-DD-3.40.1.eb --allow-modules-tool-mismatch --robot=/pfs/work/g2pbloch/test_instruction/easybuild-imas-files:/pfs/work/g2pbloch/test_instruction/eb_files_repo 
eb IMAS-AL-Fortran/IMAS-AL-Fortran-5.2.0dev-foss-2020b-DD-3.40.1.eb  --allow-modules-tool-mismatch --robot=/pfs/work/g2pbloch/test_instruction/easybuild-imas-files:/pfs/work/g2pbloch/test_instruction/eb_files_repo 
eb IMAS-AL-Java/IMAS-AL-Java-5.2.0dev-foss-2020b-Java-11-DD-3.40.1.eb  --allow-modules-tool-mismatch --robot=/pfs/work/g2pbloch/test_instruction/easybuild-imas-files:/pfs/work/g2pbloch/test_instruction/eb_files_repo 
eb IMAS-AL-Python/IMAS-AL-Python-5.2.0dev-foss-2020b-DD-3.40.1.eb  --allow-modules-tool-mismatch --robot=/pfs/work/g2pbloch/test_instruction/easybuild-imas-files:/pfs/work/g2pbloch/test_instruction/eb_files_repo

Remember to change the --robot and pass paths to your easybuild-files  and easybuild-imas-files  directories. 


Installation of IMAS/3.42.0-2024.08.1

  • Clone repo with easyconfigs: `https://gitlab.eufus.psnc.pl/g2pbloch/ebfiles_repo
  • Replace the paths to your python installed by easybuild in the files: /IMAS-AL-Core/IMAS-AL-Core-5.3.0-foss-2023b.eb, /IMAS-AL-Core/IMAS-AL-Core-5.3.0-intel-2023b.eb ,  /IMAS-AL-Python/IMAS-AL-Python-5.3.0-foss-2023b-DD-3.42.0.eb ,`/IMAS-AL-Python/IMAS-AL-Python-5.3.0-intel-2023b-DD-3.42.0.eb`
  •  Run installation: 
    eb ./ebfiles_repo/IMAS/IMAS-3.42.0-2024.08.1-foss-2023b.eb --allow-modules-tool-mismatch --robot=<path_to_ebfiles_repo>  --accept-eula-for=Intel-oneAPI
eb ./ebfiles_repo/IMAS/IMAS-3.42.0-2024.08.1-intel-2023b.eb --allow-modules-tool-mismatch --robot=<path_to_ebfiles_repo> --accept-eula-for=Intel-oneAPI

How to use IMAS

After installation you can load IMAS  modules i.e:

Load module
module use {prefix}/modules/all
module load IMAS-AL-Cpp/5.2.0dev-foss-2020b-DD-3.40.1
module load IMAS-AL-Fortran/5.2.0dev-foss-2020b-DD-3.40.1
module load IMAS-AL-Java/5.2.0dev-foss-2020b-Java-11-DD-3.40.1
module load IMAS-AL-Python/5.2.0dev-foss-2020b-DD-3.40.1

How it works

  • eb - EasyBuild

  • IMAS-AL-Cpp/IMAS-AL-Cpp-5.2.0dev-foss-2020b-DD-3.40.1.eb - path to EasyBuild configuration file

  • --allow-modules-tool-mismatch - flag
  • --robot - paths to configuration files
  • --http-header-fields-urlpat - secret to downloading files from ITER repo


Installation of the 'production' version

To install the 'production' version of IMAS ecosystem, a few extra options have to be added to point out EB to the dedicated public directory /gw/swimas/easybuild  


--prefix='/gw/swimas/easybuild/opt' 
--installpath-modules='/gw/swimas/easybuild/etc'

# TO BE DISCUSSED: Accessing sources while debugging. Should 'source' directory be public, as well? 

# following modules must be installed 
# pkgconfig
# Cmake
# IMAS_AL-Core receipt - to be checked!!!
    '-D Python3_ROOT_DIR=/gw/swimas/easybuild/opt/software/Python/3.11.5-GCCcore-13.2.0 '
    '-D Python3_EXECUTABLE=/gw/swimas/easybuild/opt/software/Python/3.11.5-GCCcore-13.2.0/bin/python3 '
    '-D Python_ROOT_DIR=/gw/swimas/easybuild/opt/software/Python/3.11.5-GCCcore-13.2.0 '
    '-D Python_EXECUTABLE=/gw/swimas/easybuild/opt/software/Python/3.11.5-GCCcore-13.2.0/bin/python3 '


Adding software components to existing installation


  • Set  MODULEPATH system variable 
     export MODULEPATH=/gw/swimas/easybuild/etc/all
  • Add --use-existing-modules  switch to eb  command


Useful tips



Installing dependencies as hidden modules

TO BE CHECKED !!!!!

{{--hide-deps}}


Purge loaded modules before using Easybuild

EasyBuild requires no modules to be loaded ! 

Please execute:

module purge

 

Change the default tmp directory

Easybuild produces a lot of temporary content (logs, intermediate build files, etc)

The default /tmp  dir on the GW offers quite small capacity. Once tmp is full, system cannot work properly.

It is strongly recommended to change the default /tmp  to a directory in $ITMWORK 


.bash_profile
tmpdir=$ITMWORK/tmp

export TMP=${tmpdir}
export TEMP=${tmpdir}
export TMPDIR=${tmpdir}

Run a massive builds on the worker nodes


# Run 'interactive' batch job using SLURM 'srun' command
# -p gw : 'gw' queue offering the longer walltime than the default 'gwdebug'
# --ntasks-per-node=48 : number of cores requested (1 node provides up to 48 cores)
# --time=48:0:0 : walltime (documentation says that the default walltime in 'gw' queue is 48h but not explicitly specified is set to 1h - To Be Checked)
# --x11 : X11 forwarding (optional)
# --pty /bin/bash : pseudoterminal
bash> srun -p gw --ntasks-per-node=48 --time=48:0:0 --pty /bin/bash


Once resources are allocated, a new shell is being opened on the working node.  A file system error is reported in most cases:

bash: /afs/eufus.eu/user/g/g2<user>/.bashrc: Permission denied


It means that the Kerberos token should be set/renewed, to allow access to the filesystem:

bash> kinit; aklog


Now .bash_profile  could be 'sourced' to enable user's startup settings:

bash> source .bash_profile


Now EasyBuild can be used to build a software.


Hints

  • Add --parallel=48  to EB commands (on the working nodes EB doesn't recognise properly  a number of available cores and runs sequentially - To Be Checked)
  • EB has built-in feature that allows to submit build to the batch system (see this link) - To Be Checked


Increasing the length of error/warning message

Easybuild shows only the first 300 characters of error/warning message and cuts (usually the most interesting) remaining part of the message.

The message length value is hardcoded and can be change only by 'hacking' the EB codes:

  1. Open <easybuild virtual env>/lib/python3.10/site-packages/easybuild/framework/easyblock.py 
  2. Look for value "300" (line 4295 in EB version 4.9.4) 
    easyblock.py
    except EasyBuildError as err:
        first_n = 300   # <=== HERE is the limit hardcoded 
        errormsg = "build failed (first %d chars): %s" % (first_n, err.msg[:first_n])  
        _log.warning(errormsg)
        result = False
  3. Increase the limit to bigger value (e.g. 3000) or remove the message cutting err.msg[:first_n])  ==> err.msg 
  4. Save the file    


Other

  • If you have compilation problems try adding the --parallel=1  option
  • If you have problem with Perl : 
    make[1]: Entering directory `/afs/gw/swimas/easybuild/opt/build/Perl/5.38.0/GCCcore-13.2.0/perl-5.38.0/utils'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/afs/gw/swimas/easybuild/opt/build/Perl/5.38.0/GCCcore-13.2.0/perl-5.38.0/utils'

        Everything is up to date. Type 'make test' to run test suite.
./perl -Ilib -I. installperl --destdir=
mkdir /afs/.gw: Read-only file system at ./install_lib.pl line 146.
make: *** [install-all] Error 30
 (at easybuild/tools/run.py:682 in parse_cmd_output)
    The reason may be the installation path starting with /afs . Try changing it.
  • If you have problem with cairo  and old ttx you can link old ttx  to /usr/bin/true .
  • Problem with IMAS modules → change easyblock to Bundle 

The scientific work is published for the realization of the international project co-financed by Polish Ministry of Science and Higher Education in 2019 from financial resources of the program entitled "PMW"; Agreement No. 5040/H2020/Euratom/2019/2.

This work has been carried out within the framework of the EUROfusion Consortium and has received funding from the Euratom research and training programme 2014–2020 under grant agreement No 633053.

This work has been carried out within the framework of the EUROfusion Consortium, funded by the European Union via the Euratom Research and Training Programme (Grant Agreement No. 101052200-EUROfusion). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the European Commission can be held responsible for them.

The project activities are co-financed by Polish Ministry of Science and Higher Education from financial resources of the programme entitled "PMW" in 2023, 2024.

The views and opinions expressed herein do not necessarily reflect those of the European Commission or ITER