Installing JOREK on Eurofusion gateway.
Part of the instructions is copied from JOREK wiki pages, eventually instructions found here should overlap with those from JOREK wiki.
How to obtain JOREK source code - ITER git platform
- Request an account for the ITER platform via email.
- Request access to JOREK code
- Create an ssh keypair (help)
Run the command
ssh-keygen -t rsa -C "<key identifier>"
Example:
ssh-keygen -t rsa -C "git.iter.org"
You will be prompted for a location to save your key. Be careful not to overwrite any existing keys that you still need!
You can add a password for an additional layer of security (encrypting your private key file) or proceed without a password. Push Enter when prompted for a password if you don't want to add a password.
Upload your public ssh key: https://git.iter.org/plugins/servlet/ssh/account/keys
The public key is stored in the .pub file.
The .pub file is a regular text file. You will need to copy its contents into the web form that you get after clicking “Add key”.
If you are using other keys, configure a .ssh/config file by adding:
Host git.iter.org HostName git.iter.org IdentityFile ~/.ssh/id_rsa_iter
if you named your file id_rsa_iter,
- Get the code from ITER platform
Your home directory is located on AFS file system (/afs/eufus.eu/user/g/<username>) which is protected by ACL and by default is not accessible by batch. If you intend to submit your jobs to batch system it is better to use GPFS file system. Save your repository for example in /pfs/work/<username>/jorek folder. You can read more about Gateway Storage Areas here.
Clone JOREK repository
git clone ssh://git@git.iter.org/stab/jorek.git
Compiling JOREK
The following modules need to be loaded e.g. by including in the ~/.bashrc
module purge module load cineca module load intel/pe-xe-2017--binary module load intelmpi/2017--binary module load mkl/2017--binary module load fftw/3.3.4--intelmpi--2017--binary module load szip/2.1--gnu--6.1.0 module load zlib/1.2.8--gnu--6.1.0 module load hdf5/1.8.17--intelmpi--2017--binary
Before compilation one has to specify which compiler and libraries will be used. This depends on the machine. For the Eurofusion Gateway the following Makefile.inc for compiling JOREK needs to be copied to JOREK repository directory
The above Makefile.inc can be modified. In line 2 the model is defined. There are several models available 199, 303, 401, 500, 501, 502, 600, 710, 711 and 712. The list of input parameters for each model can be found here. For the demonstration purposes model 303 is chosen.
Most of the necessary libraries are already available at the Gateway. There are two libraries Pastix (5.2.3) and Scotch (5.1.12) which are not available and have to be provided separately. The precompiled libraries can be found in ~g2iholod/public folder. The apropriate path is set in line 23 of Makefile.inc. Depending on the setting MUMPS library may also be needed but it is not used in the example run (see line19 of Makefile.inc).Compile JOREK with
make -j 8
This creates an executable file "jorek_model<model_number>" which is "jorek_model303" in our case.
Submitting jobs
Example of the batch job submission script
#!/bin/bash #SBATCH --job-name=jorek #SBATCH --nodes=1 #SBATCH --ntasks=2 #SBATCH --cpus-per-task=24 #SBATCH --time=00:10:00 #SBATCH --partition=gw export OMP_NUM_THREADS=24 srun /pfs/work/$USER/jorek/jorek_model303 < ./inxflow > logfile.out
The "inxflow" file with example input parameters can be found in ~g2iholod/public/jorek_dev/tmp directory. Since the example run is expected to finished within 30 minutes it is possible to change the standard partition "gw" to the debugging partition "gwdbg" by setting
#SBATCH --partition=gwdbg
in submit.sh script and omit the queue for jobs that take more time.
It is advised to run the script in a different folder than the repository. Output files are saved in a working directory and running jorek in the repository results in mixing source files with input and output files.
Submit the bash script with the command
sbatch submit.sh
Check your submission status with
squeue -u $USER
Creating namelist
Procedure of creating a JOREK namelist for a given model number is a variation of running a "JOREK for the first time case" instruction described in jorek wiki. We describe the procedure step by step here.
It is reccomended that you have access to JOREK repository. This way, you are sure that you are using the latest version of the code and namelist. Furthermore, it allows you to test for different models easily yourself (the namelists of the models are not exactly identical).
Take the very simple intear test case from the tutorial and run it for model199 with 0 time steps (nstep=0):
===== - Running a simple test case =====
We will run a very simple test case: a **tearing mode in a large aspect-ratio circular plasma**. You will have to find your own way of how to structure directories for your simulations. For the prupose of this instruction we assume you have created the following directory structure with the source code and the ''Makefile.inc'' prepared according to the instructions section in 1.2:
jorek-namelist/ jorek.git/ # the source code intear_ntor3/ # run directory
If you want to clone the repository to a specific folder, 'jorek.git' in our example, you can do it by adding 'folder.git' to 'git clone ...' command (see Section 1.1):
cd jorek-namelist/ git clone ssh://git@git.iter.org/stab/jorek.git jorek.git
To write out the namelist, you need to add right before the endif in line 193 of in models/model199/initialise_parameters.f90 the following three lines:
open(42, file='out.nml', status='replace', action='write')
write(42,in1)
close(42)
Same for the other models although the line number may change.
First, we compile the code with appropriate settings for the test case:
cd jorek.git/ ./util/config.sh model=199 n_tor=3 n_period=1 n_plane=4 # run with toroidal mode numbers n=0,1 make clean make -j 8 make -j 8 jorek2vtk jorek2_postproc
Then, we copy the compiled binaries and the input file for the test case into the run directory (the input file ''intear'' contains the information necessary to run our test case, more information about the namelist input files is given further below):
cp jorek_model199 jorek2vtk jorek2_postproc namelist/model199/intear ../intear_ntor3 cd ../intear_ntor3
Now, open with your favorite editor the file ''intear'', which contains the description of the test case to be simulated. Many things here will not immediately be self-explanatory of course. The only thing we do now is to change the settings for the time stepping by setting
tstep = 3000. nstep = 0
Important notice: You can run the other models using the same intear input file as well. However, it is not guaranteed that it works for all the models.
Managing Jorek parameters using HDF5
Presentation of the tool
A new tool written in C has been developed to manage Jorek namelist parameters files using HDF5.
The tool has been presented in May 2022 to the Jorek team at the Max Planck Institute.
Here are the slides:
Compiling and using the tool from the command line
The tool can be currently executed from the command line. Soon, it will be integrated to the Jorek fortran code in order to be called from Jorek in different use-cases.
COMPILING THE SOURCES
$ cd jorek/util/nmlist2h5 Requirements: HDF5 version >= 1.12 To use the provided Makefile, be sure that: - The HDF5_HOME environment variable is defined in order to find the HDF5 header files - The HDF5 lib directory is referenced by the LD_LIBRARY_PATH variable, for linking - BOOST library is available in your system Compiling: $ make -f Makefile_namelist Example: [nmlist2h5]$ make -f Makefile_namelist gcc -c -o namelist2h5.o namelist2h5.c gcc -c -o h5_2_namelist.o h5_2_namelist.c gcc -c -o h5_utils.o h5_utils.c gcc -c -o namelist_api.o namelist_api.c g++ -pthread -Wl,--no-undefined -o namelist_api namelist2h5.o h5_2_namelist.o h5_utils.o namelist_api.o -L/Applications/libraries/hdf5/1.12.0/gcc/6.4.0/lib -lhdf5
API
Namelist to HDF5 Converter API: //--------------------------------------------------------------------- int new_simulation(const char* h5_namelist_folder, const char* nml_file_name); int restart(const char* h5_namelist_folder, const char* nml_file_name, const char* jorek_restart_file_name, const char* time_index); int write_restart_file (const char* h5_namelist_folder, const char* jorek_restart_file); int extract_from_h5_namelist (const char* h5_namelist_folder, const char* nml_file, const char* time_index); int extract_from_jorek_restart_file (const char* h5_namelist_folder, const char* jorek_restart_file_name, const char* nml_file_name, const char* time_index); //--------------------------------------------------------------------- NEW SIMULATION: int new_simulation(const char* h5_namelist_folder, const char* nml_file_name); /** Creates a new HDF5 namelist file from an existing JOREK parameters file in CSV format. Parameters are written in the '00000' HDF5 group which will be created. * Inputs: * h5_namelist_folder (string) : directory path in which the new HDF5 namelist file will be created * nml_file_name (string) : name of the JOREK namelist file in CSV format, the file must be supplied in the 'h5_namelist_folder', an error will occur otherwise. * Returns: * 0 if the operation is successfull or a code status < 0 otherwise. **/ RESTART: int restart(const char* h5_namelist_folder, const char* nml_file_name, const char* jorek_restart_file_name, const char* time_index); /** If the HDF5 namelist file already exists in the folder specified by 'h5_namelist_folder', 'restart' will update this file from the specified JOREK restart file 'jorek_restart_file_name': * - all HDF5 groups YYYYY >= time_index=XXXXX are removed from the HDF5 namelist file * - the parameters from the file 'nml_file_name' (in CSV format) are added to the HDF5 namelist file in the HDF5 group XXXXX * If the HDF5 namelist does not exist in the folder specified by 'h5_namelist_folder', 'restart' will create a new HDF5 namelist file in the folder 'h5_namelist_folder': * - all parameters of the JOREK restart file specified by 'jorek_restart_file_name' are added to the HDF5 namelist file * - the parameters located in the file 'nml_file_name' (in CSV format) are added to the HDF5 namelist file in the HDF5 group time_index=XXXXX * Inputs: * h5_namelist_folder (string) : directory path in which the HDF5 file is located. If the file does not exist, it will be created in this directory. * nml_file_name (string) : name of the JOREK parameters file in CSV format, the file must be supplied in the 'h5_namelist_folder', an error will occur otherwise. * jorek_restart_file_name (string) : name of the JOREK restart file in HDF5 format, the file must be supplied in the 'h5_namelist_folder' if the HDF5 namelist file does not exist, an error will occur otherwise. * time_index (string) : name of the HDF5 group of the HDF5 namelist file where the namelist parameters from the file 'nml_file_name' will be added * Returns: * 0 if the operation is successfull or a code status < 0 otherwise. **/ WRITE RESTART FILE: /** Creates a new HDF5 JOREK restart file. All parameters from the existing HDF5 namelist file are written in the 'namelist' HDF5 group. * If the HDF5 namelist file does not exist in the specified 'h5_namelist_folder' directory, an error will occur. * Inputs: * h5_namelist_folder (string) : directory path in which the new HDF5 JOREK restart file will be created * jorek_restart_file_name (string) : name of the JOREK restart file in HDF5 format * Returns: * 0 if the operation is successfull or a code status < 0 otherwise. **/ EXTRACT FROM H5 NAMELIST: /** Creates a new namelist parameters file in CSV format. All parameters from the supplied HDF5 namelist file are extracted from the HDF5 group 'time_index' and copied to the CSV namelist parameters file. * Inputs: * h5_namelist_folder (string) : directory path in which the CSV namelist file will be created. This directory should contain the supplied HDF5 namelist file, an error will occur otherwise. * nml_file_name (string) : name of the CSV namelist file to be created * time_index (string) : time index with format 'XXXXX' * Returns: * 0 if the operation is successfull or a code status < 0 otherwise. **/ EXTRACT FROM JOREK RESTART FILE: /** Creates a new namelist parameters file in CSV format. All parameters from the existing JOREK HDF5 restart file are extracted from the HDF5 group 'namelist/time_index' and copied to the CSV namelist file. * Inputs: * h5_namelist_folder (string) : directory path in which the CSV namelist file will be created. This directory should contain the supplied HDF5 namelist file, an error will occur otherwise. * jorek_restart_file_name (string) : name of the JOREK restart file * nml_file_name (string) : name of the CSV namelist file to be created * time_index (string) : time index with format 'XXXXX' * Returns: * 0 if the operation is successfull or a code status < 0 otherwise. **/
EXAMPLES
NEW SIMULATION
./namelist_api new_simulation /Home/LF218007/jorek out711.nml
analysing file, 13453 line(s) found with identifier, (n_line=15864)
File analysis ended.
Number of identifiers=13453.
Number of parameters=301.
HDF5 dataset for parameter GRID_TO_WALL already previously defined, ignoring new definition
HDF5 dataset for parameter FIX_AXIS_NODES already previously defined, ignoring new definition
Checking the file with h5dump:
h5dump namelist.h5 |less
HDF5 "namelist.h5" {
GROUP "/" {
GROUP "00000" {
DATASET "ADAPTIVE_TIME" {
DATATYPE H5T_STRING {
STRSIZE H5T_VARIABLE;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
(0): "F
"
}
}
DATASET "AKI_NEO_CONST" {
…
RESTART
./namelist_api restart /Home/LF218007/jorek out711.nml jorek_restart.h5 00001
copying time index: 00001...
analysing file, 13453 line(s) found with identifier, (n_line=15864)
File analysis ended.
Number of identifiers=13453.
Number of parameters=301.
HDF5 dataset for parameter GRID_TO_WALL already previously defined, ignoring new definition
HDF5 dataset for parameter FIX_AXIS_NODES already previously defined, ignoring new definition
Checking the file with h5ls:
$ h5ls namelist.h5
00000 Group
00001 Group
WRITE RESTART FILE:
./namelist_api write_restart_file /Home/LF218007/jorek/jorek_repo/jorek/util jorek_restart.h5
Checking the file with h5ls:
h5ls -r jorek_restart.h5
/ Group
/namelist Group
/namelist/00000 Group
/namelist/00000/ADAPTIVE_TIME Dataset {SCALAR}
/namelist/00000/AKI_NEO_CONST Dataset {SCALAR}
/namelist/00000/AMIN Dataset {SCALAR}
/namelist/00000/AMIX Dataset {SCALAR}
...
/namelist/00001 Group
/namelist/00001/ADAPTIVE_TIME Dataset {SCALAR}
/namelist/00001/AKI_NEO_CONST Dataset {SCALAR}
/namelist/00001/AMIN Dataset {SCALAR}
/namelist/00001/AMIX Dataset {SCALAR}
...
/namelist/00001/Z_LIMITER Dataset {1000/Inf}
EXTRACT FROM H5 NAMELIST:
./namelist_api extract_from_h5_namelist /Home/LF218007/jorek test.nml 00001
found group 00001
matched group 00001
Converting HDF5 file /Home/LF218007/jorek/jorek_repo/jorek/util/namelist.h5 ...
more test.nml
&IN1
ADAPTIVE_TIME = F
,
AKI_NEO_CONST = 0.000000e+00,
AMIN = 1.000000e+00,
AMIX = 0.000000e+00,
AMIX_FREEB = 8.500000e-01,
AMU_NEO_CONST = 0.000000e+00,
AUTODISTRIBUTE_MODES = T
,
…
EXTRACT FROM JOREK RESTART FILE:
./namelist_api extract_from_jorek_restart_file /Home/LF218007/jorek jorek_restart.h5 test2.nml 00001
found group 00001
matched group 00001
Converting HDF5 file /Home/LF218007/jorek/jorek_repo/jorek/util/jorek_restart.h5 ...
more test2.nml
&IN1
ADAPTIVE_TIME = F
,
AKI_NEO_CONST = 0.000000e+00,
AMIN = 1.000000e+00,
AMIX = 0.000000e+00,
AMIX_FREEB = 8.500000e-01,
AMU_NEO_CONST = 0.000000e+00,
AUTODISTRIBUTE_MODES = T
,
…
The scientific work is published for the realization of the international project co-financed by Polish Ministry of Science and Higher Education in 2019 from financial resources of the program entitled "PMW"; Agreement No. 5040/H2020/Euratom/2019/2
This work has been carried out within the framework of the EUROfusion Consortium and has received funding from the Euratom research and training programme 2014–2020 under grant agreement No 633053. The views and opinions expressed herein do not necessarily reflect those of the European Commission or ITER