Markdown |
---|
# Container image - Images are binary files containing all the data and metadata required to start the container - They can be built locally or downloaded from remote locations - The most common standard is the Docker image format ## Docker image format - Docker image is a `tar` archive with metadata and *layers* - Each layer consists of its own metadata and another `tar` archive with the set of changes the layer introduces - The first metadata file in the image is `manifest.json`: ``` json [ { "Config": "f63181f19b2fe819156dcb068b3b5bc036820bec7014c5f77277cfa341d4cb5e.json", "RepoTags": [ "ubuntu:latest" ], "Layers": [ "151ae8ef4f042fd5173fd2497f0a365b4413468163e7bd567146f29dcfea3517/layer.tar", "2872658e1abe34d0c7391abbc0848fdeddb456659e39511df0574fcfc8b7ad70/layer.tar", "2b83a9243dd8405d0811beeb14aeb797745b100e4538d056adb63fcc6b47c59f/layer.tar" ] } ] ``` - It contains: - `Config` -- path to configuration file (architecture requirements, etc.), - `RepoTags` -- the list of tags used, - `Layers` -- paths to `tar` files containing layer information ## OCI image format - An alternative format was proposed by OCI (*Open Container Initiative*) - It is also a `tar` archive containing metadata and layers in the form of embedded `tar` archives - The first metadata file is `index.json`: ``` json { "schemaVersion": 2, "manifests": [ { "mediaType": "application/vnd.oci.image.manifest.v1+json", "digest": "sha256:7ad481b55901a1b5472c0e1b3fbf0bf2867dc38feb6eb7a18cd310f00208e05c", "size": 658 } ] } ``` - The manifest contains paths to configuration and layers: ``` json { "schemaVersion": 2, "config": { "mediaType": "application/vnd.oci.image.config.v1+json", "digest": "sha256:10bdc2317d43a5421151e135881e172002c7d61e934de7e1e79df560a151f112", "size": 2427 }, "layers": [ { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:f3f8f4bd7c131f4d967bc162207ab72c24f427915682f895eb4f793ad05d7e35", "size": 29989546 }, { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:0188b501936213b7cd0b5333245960781a8b035249cfa427fe9a229fe557c624", "size": 924 }, { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:db861e57845ea7ba52a2ac277abbdd8cd04bda5db69c49bf95be49d11e5a47e1", "size": 202 } ] } ``` ## Local vs remote - Both formats describe how the image is stored as a *local file* - When transferring the image from remote server, the client asks for a list of layers, then checks its cache contents and finally downloads only the missing layers - This allows to reuse layers in images depending on each other - For example: - Let's say image `A` is based on `ubuntu` - Image `B` extends `A` by adding Python executable on top - Image `C` also extends `A`, but it adds Apache HTTP server instead - A first-time user of `B` image will download layers from `ubuntu`, then from `A` and finally from `B` - When he/she wants to use `C` image, only the layers with Apache HTTP server will be downloaded as all previous are still in cache # Building container images - The most straightforward approach is by using the [`Dockerfile` format](https://docs.docker.com/engine/reference/builder/) - The `Dockerfile` is a text file with commands describing the recipe to build the image - The first command is `FROM <image>` which instructs what to base the image on - The `RUN <command>` executes a command in the builder context - The `CMD <command>` configures the default command a container will run upon creation - The `EXPOSE <port>` adds metadata about a port a service inside the image will listen on (**Important!** The container creator decided which ports to publish and how. Exposing a port in `Dockerfile` serves as a form of documentation.) - The `ENV <key>=<value>` sets environment variables' values - The `COPY <src> <dest>` allows to copy files from the host to the image - The `WORKDIR <dir>` sets the current working directory as seen inside the container - The `ARG <name>=<default>` configures a build-time argument that can be changed by the image builder ## Example - Contents of `index.html`: ``` html <h1>Hello World from Docker!</h1> ``` - Contents of `Dockerfile`: ``` dockerfile FROM ubuntu ENV DEBIAN_FRONTEND=noninteractive COPY index.html /var/www/html/index.html RUN apt-get update -y RUN apt-get install -y apache2 EXPOSE 80 CMD ["/usr/sbin/apachectl", "-DFOREGROUND"] ``` # Optimization - The layer system allows caching, but it also has its consequences you need to be aware of - **Scenario 1**. Image `A` creates `/bigfile`, image `B` extending `A` deletes it. This fact is merely masked -- i.e. the user of `B` will not see `/bigfile`, but the file will still be part of image `B` and it will still take a lot of space - **Scenario 2**. One of the layers contains secrets (passwords, unencrypted private keys, etc.), the next layers delete them. Even though the secretes are not directly available, one can extract `tar` archives of every layer and get access to them anyway - In `Dockerfile`, each command creates a separate layer, so you should usually do this: - Combine subsequent `RUN` commands: ``` diff -RUN touch /test1 -RUN date > /test2 +RUN touch /test1 && date > /test2 ``` - In a combined `RUN`, make sure to delete all temporary files: ``` diff -RUN apt-get update -y -RUN apt-get install -y git -RUN rm -rf /var/lib/apt/lists/* +RUN apt-get update -y \ + && apt-get install -y git \ + && rm -rf /var/lib/apt/lists/* ``` ``` diff -RUN curl URL > archive.tar -RUN tar xf archive.tar -RUN rm archive.tar +RUN curl URL | tar x ``` - Building procedure also makes use of caching - **Scenario 1**: - Layer 1: Install Apache HTTP server - Layer 2: Copy `index.html` - **Scenario 2**: - Layer 1: Copy `index.html` - Layer 2: Install Apache HTTP server - Both scenarios will create equivalent images, but if you change `index.html`, then in **Scenario 2** both layers will be rebuilt, while in **Scenario 1** only the second one - A good practice is to order the layers according to probability of being changed (the more probable, the later should it be) ## Example after optimization ``` dockerfile FROM ubuntu ENV DEBIAN_FRONTEND=noninteractive RUN apt-get update -y \ && apt-get install -y \ apache2 \ && rm -rf /var/lib/apt/lists/* EXPOSE 80 CMD ["/usr/sbin/apachectl", "-DFOREGROUND"] COPY index.html /var/www/html/index.html ``` # Multi-stage Dockerfiles - Sometimes to build an image you need to generate the resources e.g. compile and build a JAR file from a Java project - In such cases, usually the source codes and a set of tools required to process them is not necessary in the final image - To optimize the image, you could try performing the following in **a single `RUN` command**: - Transfer source codes - Install build-time dependencies (compilers, etc.) - Build everything - Transfer the generated resource to its final destination - Remove all immediate files - This is a lot to be done in a single command, so it is prone to errors and hard to debug - To overcome this, you can use multi-stage building, which is like building multiple images simultaneously and freely transferring files between them - Each stage starts with its own `FROM <image> AS <stage>` command - You can copy files between images created in separate stages using `COPY --from=<stage>` syntax ## Example - Contents of `hello.go`: ``` go package main import "fmt" func main() { fmt.Println("hello world") } ``` - Contents of `Dockerfile`: ``` dockerfile FROM golang AS builder COPY hello.go hello.go RUN go build hello.go FROM ubuntu COPY --from=builder /go/hello /usr/bin/hello CMD /usr/bin/hello ``` # BuildKit - Starting with version 18.09, Docker is shipped with two build engines: the legacy one (used by default) and [the BuildKit](https://docs.docker.com/develop/develop-images/build_enhancements/) - To use the new engine, you have to set the following environment variable `DOCKER_BUILDKIT=1` - The new engine has the following advantages: - Independent stages' steps are executed in parallel - You can pass private SSH keys to the build process and be sure they do not end up in any layer or metadata: ``` dockerfile RUN --mount=type=ssh <command> ``` ``` sh docker build --ssh default . ``` - Similarly, you can pass any secrets to the build process: ``` dockerfile RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret ``` ``` sh docker build --secret id=mysecret,src=file.txt . ``` - And the following disadvantage: - It is harder to debug problems # Debugging (legacy engine) - Many things might go wrong during Docker image preparation - When building with the legacy engine, each command in Dockerfile creates a layer, which gets stored under unique id - If something goes wrong, you can instantiate an interactive container from the last layer that was built successfully and look for clues ## Example - Contents of `Dockerfile` ``` dockerfile FROM alpine RUN date > /tmp/build-date.txt RUN cat /tmp/build-dat.txt > /tmp/final-date.txt ``` - Results of `docker build .`: Sending build context to Docker daemon 2.048kB Step 1/3 : FROM alpine ---> 14119a10abf4 Step 2/3 : RUN date > /tmp/build-date.txt ---> Running in 3fe480f490d7 Removing intermediate container 3fe480f490d7 ---> 7eadcff6ea01 Step 3/3 : RUN cat /tmp/build-dat.txt > /tmp/final-date.txt ---> Running in 7cba04ecd3e3 cat: can't open '/tmp/build-dat.txt': No such file or directory The command '/bin/sh -c cat /tmp/build-dat.txt > /tmp/final-date.txt' returned a non-zero code: 1 - The result of `FROM` command is stored as `14119a10abf4` - The result of the first `RUN` command is stored as `7eadcff6ea01` - The second `RUN` fails, so you debug it by starting an interactive container (the `-it` switch): ``` sh docker run -it 7eadcff6ea01 ``` - Now you can try to execute the command that failed: `cat /tmp/build-dat.txt` - Then figure out why it failed, what could cause it, etc. # Debugging (BuildKit) - With BuildKit layers are no longer stored after each command in the Dockerfile - To improve performance, BuildKit only stores the image when a stage is finished - To debug problems, you have to abuse this rule and introduce artificial stage beginnings and ends ## Example - For the same `Dockerfile` as before, with BuildKit you will see this output (for TTY output style `plain` configured by running `docker build --progress plain .`): #2 [internal] load .dockerignore #2 sha256:28b059ecac284a33ba98daa285c6a068d86485b54afc2e67f18e2bd1640d871a #2 transferring context: 2B done #2 DONE 0.1s #1 [internal] load build definition from Dockerfile #1 sha256:cbd3d6400308afcf33c0910b894d2e44156fc4127a0db290d19df5a4e8eae37e #1 transferring dockerfile: 37B done #1 DONE 0.3s #3 [internal] load metadata for docker.io/library/alpine:latest #3 sha256:d4fb25f5b5c00defc20ce26f2efc4e288de8834ed5aa59dff877b495ba88fda6 #3 DONE 0.0s #4 [1/3] FROM docker.io/library/alpine #4 sha256:665ba8b2cdc0cb0200e2a42a6b3c0f8f684089f4cd1b81494fbb9805879120f7 #4 CACHED #5 [2/3] RUN date > /tmp/build-date.txt #5 sha256:684d70446f71e64256b21f59555e6fedc1eac55780675519af54f9e174fd16e1 #5 DONE 1.2s #6 [3/3] RUN cat /tmp/build-dat.txt > /tmp/final-date.txt #6 sha256:bde8a93c3b05727094dd3d24e010c506f451a33718cc07f32f4e6b1ccab0b645 #6 0.925 cat: can't open '/tmp/build-dat.txt': No such file or directory #6 ERROR: executor failed running [/bin/sh -c cat /tmp/build-dat.txt > /tmp/final-date.txt]: exit code: 1 ------ > [3/3] RUN cat /tmp/build-dat.txt > /tmp/final-date.txt: ------ executor failed running [/bin/sh -c cat /tmp/build-dat.txt > /tmp/final-date.txt]: exit code: 1 - Although you can notice lines starting with `sha256:...`, they do not correspond to intermediate layers' ids - To be able to debug the problem as previously, you have to alter the `Dockerfile` by (1) naming the debugged stage if not yet done and (2) adding `FROM scratch` line just before the command you need to debug: ``` diff -FROM alpine +FROM alpine AS debug RUN date > /tmp/build-date.txt +FROM scratch RUN cat /tmp/build-dat.txt > /tmp/final-date.txt ``` - Now you can set BuildKit to only consider `debug` target: ``` sh docker build --progress plain --target debug . ``` #1 [internal] load build definition from Dockerfile #1 sha256:ffe58018ac4c453ce043471e51216f7528c1fe315a9a83fa1ad276df0ac9f8a6 #1 transferring dockerfile: 157B done #1 DONE 0.1s #2 [internal] load .dockerignore #2 sha256:dda1a34cfb4f2eb8169d58953a937062de5c70a0ae78ca49118e49ea8279a7b7 #2 transferring context: 2B done #2 DONE 0.2s #3 [internal] load metadata for docker.io/library/alpine:latest #3 sha256:d4fb25f5b5c00defc20ce26f2efc4e288de8834ed5aa59dff877b495ba88fda6 #3 DONE 0.0s #4 [debug 1/2] FROM docker.io/library/alpine #4 sha256:665ba8b2cdc0cb0200e2a42a6b3c0f8f684089f4cd1b81494fbb9805879120f7 #4 DONE 0.0s #5 [debug 2/2] RUN date > /tmp/build-date.txt #5 sha256:684d70446f71e64256b21f59555e6fedc1eac55780675519af54f9e174fd16e1 #5 CACHED #6 exporting to image #6 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00 #6 exporting layers #6 exporting layers 0.4s done #6 writing image sha256:0846d70d4f562fa95835da5893cb53beb7f623cf228f57460bd298a0f87da680 0.0s done #6 DONE 0.5s - Finally you can start an interactive container by taking the hash value of the written image: ``` sh docker run -it 0846d70d4f562fa95835da5893cb53beb7f623cf228f57460bd298a0f87da680 ``` # IMAS Docker ## General remarks - The IMAS Docker build project is available at git.iter.org as `IMEX/imas-container` - It contains `ansible-container/`, `buildah/` and `docker/` subdirectories - Only the last one is actively developed - The `Dockerfile` requires BuildKit as it uses `--mount=type=ssh` - This also means that if you want to build IMAS Docker, you need to configure SSH agent and add there the private key which gives access to git.iter.org projects ``` ssh://git@git.iter.org/imas/access-layer.git ssh://git@git.iter.org/imas/data-dictionary.git ssh://git@git.iter.org/imex/fc2k.git ssh://git@git.iter.org/imas/installer.git ssh://git@git.iter.org/imex/kepler-core-actors.git ssh://git@git.iter.org/imex/kepler-installer.git ssh://git@git.iter.org/imex/kepler-patches.git ssh://git@git.iter.org/imas/uda.git ssh://git@git.iter.org/imas/uda-plugins.git ssh://git@git.iter.org/imex/kepler.git ``` ## `build.sh` - This Bash script accepts the following command line options: -f disable cache (build everything from scratch) -u build with UDA -c CPUs number of CPUs [default=$(nproc)] -t target build only one target - There are three auxiliary Bash functions defined: - `latest_git_tag url blacklisted`. Returns the latest (in the meaning of `sort --version-sort`) tag in a git repository given with the `url`. If `blacklisted` is given, such tag will be ignored. For example, the UDA repository has tag `code_camp_cadarache` which should be ignored. - `latest_stable_git_tag url`. As above, but the tag has to contain `stable` keyword (applicable for MDS+) - `latest_released_git_tag url`. As above, but the tag has to contain `rel` keyword (applicable for Ant) - Next, in the `build.sh` script you can set either of these tags to specific values: ``` sh tag_al=4.8.7 # FIXME: 4.8.7 is used by ETS tag_ant=1.10.6 # FIXME: later versions of ant fail to build... tag_blitz= tag_cmake=v3.20.0 tag_dd=3.31.0 # FIXME: 3.31.0 is used by ETS tag_fc2k= tag_hdf5=hdf5-1_12_0 tag_installer= tag_kca= tag_kepler_installer= tag_kp= tag_lapack= tag_mdsplus=stable_release-7-96-15 # FIXME: later versions of mdsplus fail to build... tag_tigervnc= tag_uda=2.3.1 # FIXME: uda/2.3.1 is known to work well with uda-plugins/1.2.0 tag_uda_plugins=1.2.0 # FIXME: uda/2.3.1 is known to work well with uda-plugins/1.2.0 ver_kb= ``` - All unset values will be checked using functions defined previously ## `files/{base,ual,kepler,fc2k,gui}` - There are several files used in `Dockerfile` available in `files/*` directories - In `files/base` you can find `blas-ifort.pc` and `lapack-ifort.pc` files preconfigured for BLAS and LAPACK (they do not come with the upstream package, so they were crafted manually) - In `files/kepler` you can find modulefile for JAXFront (it also had to be crafted manually) - In `files/{ual,kepler}` you can find `Makefile.Docker.Ubuntu` which is a configuration file used by IMAS Installer and Kepler Installer - In these files you select what to build e.g. you can switch off `gfortran` compilation of Fortran interface - In `files/fc2k` you can find `install_Docker.Ubuntu.xml` and `settings_Docker.Ubuntu.xml` which control the installation and usage of FC2K - In `files/{ual,kepler,fc2k}` you can find `docker-entrypoint.sh` which is set to be the entrypoint of corresponding images. The role of these scripts is to load necessary modules (e.g. IMAS or UDA) and execute `imasdb test` ## `Dockerfile`  - There are 14 stages in the `Dockerfile`, all based on Ubuntu 18.04 - `common-builder` has compilers and libraries for building Blitz++, HDF5 and MDS+. It installs CMake from GitHub, because the version in Ubuntu 18.04 repo is too old ``` dockerfile FROM ubuntu:18.04 AS common-builder ENV DEBIAN_FRONTEND=noninteractive RUN apt-get update \ && apt-get install -y \ curl \ g++ \ gfortran \ git \ libreadline-dev \ libxml2-dev \ make \ openjdk-8-jdk-headless \ openssh-client \ python \ tclsh \ && rm -rf /var/lib/apt/lists/* # Ubuntu 18.04 has old version of CMake, so blitz++ and hdf5 cannot be built -> install it manually ARG tag_cmake RUN curl -L https://github.com/Kitware/CMake/releases/download/${tag_cmake}/cmake-${tag_cmake#v}-linux-x86_64.tar.gz | tar xz ``` - `blitz-builder` builds Blitz++ in `/opt/blitz`, `hdf5-builder` builds HDF5 in `/opt/hdf5`, `mdsplus-builder` builds MDS+ in `/opt/mdsplus` ``` dockerfile # blitz-builder ############################################################### FROM common-builder AS blitz-builder # Ubuntu 18.04 does not have blitz++ -> have to install it manually ARG tag_blitz ARG cpus=1 RUN curl -L https://github.com/blitzpp/blitz/archive/${tag_blitz}.tar.gz | tar xz RUN mkdir blitz-${tag_blitz}/build \ && cd blitz-${tag_blitz}/build \ && /cmake-${tag_cmake#v}-linux-x86_64/bin/cmake \ -DCMAKE_INSTALL_PREFIX:PATH=/opt/blitz \ .. \ && make -j ${cpus} lib \ && make install # hdf5-builder ################################################################ FROM common-builder AS hdf5-builder # Ubuntu 18.04 packages hdf5 in a different way than IMAS expects it to be packaged ARG tag_hdf5 ARG cpus=1 RUN curl -L https://github.com/HDFGroup/hdf5/archive/refs/tags/${tag_hdf5}.tar.gz | tar xz RUN mkdir hdf5-${tag_hdf5}/build \ && cd hdf5-${tag_hdf5}/build \ && /cmake-${tag_cmake#v}-linux-x86_64/bin/cmake \ -DCMAKE_INSTALL_PREFIX:PATH=/opt/hdf5 \ .. \ && make -j ${cpus} \ && make install \ && version=$(echo ${tag_hdf5} | sed -e 's/^hdf5-//' -e 's/_/./g') \ && ln -s hdf5-${version}.pc /opt/hdf5/lib/pkgconfig/hdf5.pc \ && ln -s hdf5_cpp-${version}.pc /opt/hdf5/lib/pkgconfig/hdf5-cpp-gnu.pc # mdsplus-builder ############################################################# FROM common-builder AS mdsplus-builder ARG tag_mdsplus RUN curl -L https://github.com/MDSplus/mdsplus/archive/${tag_mdsplus}.tar.gz | tar xz RUN cd mdsplus-${tag_mdsplus} \ && ./configure --prefix=/opt/mdsplus \ && make \ && make install ``` - `imas-git-puller` pulls all repositories from git and it is the only stage that accesses SSH keys ``` dockerfile FROM common-builder AS imas-git-puller RUN mkdir -p /root/.ssh \ && chmod 700 /root/.ssh \ && ssh-keyscan git.iter.org > /root/.ssh/known_hosts ARG tag_fc2k RUN --mount=type=ssh true \ && git clone --single-branch -b ${tag_fc2k} ssh://git@git.iter.org/imex/fc2k.git ARG tag_uda RUN --mount=type=ssh true \ && git clone --single-branch -b ${tag_uda} ssh://git@git.iter.org/imas/uda.git ARG tag_uda_plugins RUN --mount=type=ssh true \ && git clone --single-branch -b ${tag_uda_plugins} ssh://git@git.iter.org/imas/uda-plugins.git ARG tag_kepler_installer RUN --mount=type=ssh true \ && git clone --single-branch -b ${tag_kepler_installer} ssh://git@git.iter.org/imex/kepler-installer.git COPY files/kepler/Makefile.Docker.Ubuntu /kepler-installer-site-config/ ARG ver_kb ARG tag_kp ARG tag_kca RUN --mount=type=ssh true \ && cd kepler-installer \ && make update \ SITECONFIG=/kepler-installer-site-config/Makefile.Docker.Ubuntu \ VER_KB=${ver_kb} \ TAG_KP=${tag_kp} \ TAG_KCA=${tag_kca} ARG tag_installer ARG tag_dd ARG tag_al RUN --mount=type=ssh true \ && git clone --single-branch -b ${tag_installer} ssh://git@git.iter.org/imas/installer.git \ && cd installer \ && make checkout \ TAG_DD=${tag_dd} \ TAG_AL=${tag_al} COPY files/ual/Makefile.Docker.Ubuntu /installer/site-config/ ``` - `base` contains a long list of applications, libraries and environment variables that will be used by any other `imas/*` image ``` dockerfile FROM ubuntu:18.04 AS base ENV DEBIAN_FRONTEND=noninteractive RUN apt-get update \ && apt-get install -y \ autoconf \ bsdtar \ cmake \ curl \ doxygen \ environment-modules \ g++ \ gawk \ gcc \ gfortran \ git \ libboost-dev \ libboost-filesystem-dev \ libboost-system-dev \ libmemcached-dev \ libopenmpi-dev \ libreadline-dev \ libsaxonhe-java \ libssh-dev \ libssl-dev \ libxml2-dev \ make \ openjdk-8-jdk-headless \ openmpi-bin \ pkg-config \ python3 \ python3-dev \ python3-pip \ rsync \ tclsh \ tcsh \ unzip \ xinetd \ xsltproc \ && rm -rf /var/lib/apt/lists/* \ && update-alternatives --install /usr/bin/python python /usr/bin/python3 10 \ && pip3 install \ cython \ numpy \ matplotlib \ setuptools \ wheel \ && ln --symbolic /usr/share/java/Saxon-HE.jar /usr/share/java/saxon9he.jar \ && ln --symbolic /usr/sbin/xinetd /usr/local/sbin/xinetd \ && ln --symbolic --force /bin/bash /bin/sh COPY --from=blitz-builder /opt/blitz /opt/blitz COPY --from=hdf5-builder /opt/hdf5 /opt/hdf5 COPY --from=mdsplus-builder /opt/mdsplus /opt/mdsplus ENV CFLAGS=-pthread \ CLASSPATH=/usr/share/java/saxon9he.jar \ CXXFLAGS=-pthread \ HDF5_HOME=/opt/hdf5 \ ITMSCRATCH=/tmp \ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \ LANG=C.UTF-8 \ MDSPLUS_DIR=/opt/mdsplus \ PKG_CONFIG_PATH=/opt/blitz/lib/pkgconfig:/opt/hdf5/lib/pkgconfig ENV CPLUS_INCLUDE_PATH=${HDF5_HOME}/include:${MDSPLUS_DIR}/include \ C_INCLUDE_PATH=${HDF5_HOME}/include:${MDSPLUS_DIR}/include:/usr/lib/jvm/java-8-openjdk-amd64/include/linux:/usr/lib/jvm/java-8-openjdk-amd64/include \ LD_LIBRARY_PATH=${HDF5_HOME}/lib:${MDSPLUS_DIR}/lib:/usr/lib/x86_64-linux-gnu \ LIBRARY_PATH=${HDF5_HOME}/lib:${MDSPLUS_DIR}/lib:/usr/lib/x86_64-linux-gnu \ MDS_PATH=${MDSPLUS_DIR}/tdi ``` - `base-devel` adds Intel compilers on top of that and it uses them to compile BLAS and LAPACK ``` dockerfile FROM base AS base-devel COPY --from=intel/oneapi-hpckit:devel-ubuntu18.04 /opt/intel /opt/intel ARG tag_lapack ARG cpus=1 RUN curl -L https://github.com/Reference-LAPACK/lapack/archive/refs/tags/${tag_lapack}.tar.gz | tar xz \ && . /opt/intel/oneapi/compiler/latest/env/vars.sh \ && cd /lapack-${tag_lapack#v} \ && cp INSTALL/make.inc.ifort make.inc \ && sed -i \ -e 's/CFLAGS =.*/& -fPIC/g' \ -e 's/FFLAGS =.*/& -fPIC/g' \ -e 's/FFLAGS_NOOPT =.*/& -fPIC/g' \ -e 's/LDFLAGS =.*/& -fPIC/g' \ make.inc \ && make -j ${cpus} \ && install -Dm644 librefblas.a /opt/blas/lib/librefblas.a \ && install -Dm644 liblapack.a /opt/lapack/lib/liblapack.a \ && install -d /opt/blas/lib/pkgconfig /opt/lapack/lib/pkgconfig \ && ln -s librefblas.a /opt/blas/lib/libblas.a \ && ln -s blas-ifort.pc /opt/blas/lib/pkgconfig/blas.pc \ && ln -s lapack-ifort.pc /opt/lapack/lib/pkgconfig/lapack.pc COPY files/base/blas-ifort.pc /opt/blas/lib/pkgconfig/blas-ifort.pc COPY files/base/lapack-ifort.pc /opt/lapack/lib/pkgconfig/lapack-ifort.pc ENV PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:/opt/blas/lib/pkgconfig:/opt/lapack/lib/pkgconfig \ LIBRARY_PATH=${LIBRARY_PATH}:/opt/blas/lib:/opt/lapack/lib ``` - `ual-devel` does the following: (1) compiles UDA and UDA Plugins, (2) compiles IMAS without Fortran interface, (3) compiles Fortran interface, (4) installs IMAS, (5) builds UDA Plugins again - UDA has a cyclic dependency: IMAS requires UDA, UDA Plugins require IMAS - Fortan interface takes the longest to compile and requires the most amount of RAM. If you have trouble building the image, set CPU count in parallel building to a smaller value (see `-c CPUs` in `build.sh` description) ``` dockerfile FROM base-devel AS ual-devel COPY --from=imas-git-puller /uda /uda COPY --from=imas-git-puller /uda-plugins /uda-plugins ARG with_uda ARG cpus=1 ENV UDA_HOME=/opt/uda # build UDA RUN mkdir /opt/uda \ && if test ${with_uda} = y; then \ mkdir /uda/build \ && cd /uda/build \ && cmake \ -DCMAKE_CXX_FLAGS=-I/usr/include/libxml2 \ -DUDA_HOST=localhost \ -DUDA_PORT=56565 \ -DBUILD_SHARED_LIBS=true \ -DCMAKE_INSTALL_PREFIX:PATH=${UDA_HOME} \ .. \ && make -j ${cpus} \ && make install \ && sed -i 's/^Libs:.*/& -lmemcached/g' ${UDA_HOME}/lib/pkgconfig/uda-* \ && cd ${UDA_HOME}/python_installer \ && python setup.py install --prefix=${UDA_HOME} \ ; \ fi ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${UDA_HOME}/lib/plugins \ LIBRARY_PATH=${LIBRARY_PATH}:${UDA_HOME}/lib/plugins \ MODULEPATH=/etc/modulefiles:${UDA_HOME}/modulefiles \ PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:${UDA_HOME}/lib/pkgconfig \ PYTHONPATH=${UDA_HOME}/lib/python3.6/site-packages \ UDA_DISABLED=0 \ UDA_IMAS_MACHINE_MAP=${UDA_HOME}/etc/udaMachines.txt \ UDA_IMAS_PLUGIN_MAP=${UDA_HOME}/etc/udaMappings.txt \ UDA_LOG=/tmp \ UDA_LOG_LEVEL=ERROR \ UDA_LOG_MODE=a \ UDA_PLUGIN_CONFIG=${UDA_HOME}/etc/udaPlugins.conf \ UDA_PLUGIN_DEBUG_SINGLEFILE=1 \ UDA_PLUGIN_DIR=${UDA_HOME}/lib/plugins \ UDA_SARRAY_CONFIG=${UDA_HOME}/etc/udagenstruct.conf # build UDA plugins RUN if test ${with_uda} = y; then \ mkdir /uda-plugins/build \ && cd /uda-plugins/build \ && cmake \ -DCMAKE_INSTALL_PREFIX:PATH=${UDA_HOME} \ '-DBUILD_PLUGINS:STRING=exp2imas;hl2a;imas_forward;imas_mapping;imas_partial;imas_remote;imas_uda;imasdd;iter_md;jet_equilibrium;jet_magnetics;jet_summary;mast_imas;source;tcv;tcvm;tore_supra;west;west_tunnel' \ .. \ && make -j ${cpus} \ && make install \ && cat ${UDA_HOME}/etc/plugins/udaPlugins* \ | sed 's/, , WEST_TUNNEL::read()/, Use SSH tunnel, WEST_TUNNEL::read()/g' \ | sort -u > ${UDA_PLUGIN_CONFIG} \ && echo 'JET IMAS_MAPPING - -' > ${UDA_IMAS_PLUGIN_MAP} \ && echo 'MAST IMAS_MAPPING - -' >> ${UDA_IMAS_PLUGIN_MAP} \ && echo 'WEST WEST_TUNNEL - -' >> ${UDA_IMAS_PLUGIN_MAP} \ && echo 'JET * - EXP2IMAS' > ${UDA_IMAS_MACHINE_MAP} \ && echo 'MAST * - IMAS_UDA' >> ${UDA_IMAS_MACHINE_MAP} \ && echo 'WEST * - WEST_TUNNEL' >> ${UDA_IMAS_MACHINE_MAP} \ ; \ fi COPY --from=imas-git-puller /installer /installer ARG tag_dd ARG tag_al # compile everything but fortran RUN cd installer \ && if test ${with_uda} = y; then . /etc/profile.d/modules.sh && module load uda && export IMAS_UDA=fat; fi \ && make -j ${cpus} IMAS_FORTRAN=no GIT_OFFLINE=y SITECONFIG=/installer/site-config/Makefile.Docker.Ubuntu TAG_DD=${tag_dd} TAG_AL=${tag_al} # compile fortran RUN cd installer \ && . /opt/intel/oneapi/compiler/latest/env/vars.sh \ && if test ${with_uda} = y; then . /etc/profile.d/modules.sh && module load uda && export IMAS_UDA=fat; fi \ && make -j ${cpus} GIT_OFFLINE=y SITECONFIG=/installer/site-config/Makefile.Docker.Ubuntu TAG_DD=${tag_dd} TAG_AL=${tag_al} # install everything RUN cd installer \ && if test ${with_uda} = y; then . /etc/profile.d/modules.sh && module load uda && export IMAS_UDA=fat; fi \ && make install -j ${cpus} GIT_OFFLINE=y SITECONFIG=/installer/site-config/Makefile.Docker.Ubuntu TAG_DD=${tag_dd} TAG_AL=${tag_al} ENV MODULEPATH=${MODULEPATH}:/opt/imas/etc/modulefiles # build UDA Plugins again, so that now it finds IMAS RUN if test ${with_uda} = y; then \ . /etc/profile.d/modules.sh \ && module load IMAS \ && mkdir /uda-plugins/build-with-imas \ && cd /uda-plugins/build-with-imas \ && cmake \ -DCMAKE_INSTALL_PREFIX:PATH=${UDA_HOME} \ '-DBUILD_PLUGINS:STRING=exp2imas;hl2a;imas_forward;imas_mapping;imas_partial;imas_remote;imas_uda;imasdd;iter_md;jet_equilibrium;jet_magnetics;jet_summary;mast_imas;source;tcv;tcvm;tore_supra;west;west_tunnel' \ .. \ && make -j ${cpus} \ && make install \ ; \ fi COPY files/ual/docker-entrypoint.sh / ENTRYPOINT ["/docker-entrypoint.sh"] CMD ["/bin/bash"] ``` - `kepler-devel` adds Ant, JAXFront and Kepler - JAXFront is a licensed software and IMAS Docker uses the [free edition](https://www.jaxfront.org/pages/pricing/free-edition.html) ``` dockerfile FROM ual-devel AS kepler-devel COPY files/kepler/Makefile.Docker.Ubuntu /kepler-installer/site-config/ COPY files/kepler/jaxfront /etc/modulefiles/ COPY --from=imas-git-puller /kepler-installer /kepler-installer ARG ver_kb ARG tag_kp ARG tag_kca ARG tag_ant RUN curl -L https://archive.apache.org/dist/ant/binaries/apache-ant-${tag_ant}-bin.tar.xz | tar xJ -C /opt \ && curl -L https://www.jaxfront.org/download/JAXFront-Demo-Project.zip | bsdtar xf - \ && mv JAXFront\ Eclipse\ Example\ Project /opt/jaxfront \ && ln -s /opt/apache-ant-${tag_ant}/bin/ant /usr/bin/ \ && cd kepler-installer/ \ && . /etc/profile.d/modules.sh \ && module load IMAS jaxfront \ && make install GIT_OFFLINE=y SITECONFIG=/kepler-installer/site-config/Makefile.Docker.Ubuntu VER_KB=${ver_kb} TAG_KP=${tag_kp} TAG_KCA=${tag_kca} \ && module load Kepler \ && kepler_install -f default COPY files/kepler/docker-entrypoint.sh / ENV USER=root ENV KEPLER=${USER}/.local/Kepler/default/kepler ``` - `fc2k-devel` adds FC2K ``` dockerfile FROM kepler-devel AS fc2k-devel COPY files/fc2k/install_Docker.Ubuntu.xml /fc2k/config/ COPY files/fc2k/settings_Docker.Ubuntu.xml /fc2k/config/ COPY --from=imas-git-puller /fc2k /fc2k RUN cd fc2k \ && . /etc/profile.d/modules.sh \ && module load IMAS \ && ln --symbolic --force /fc2k/config/settings_Docker.Ubuntu.xml /fc2k/config/settings_default.xml \ && ant install -Dcfg.file=/fc2k/config/install_Docker.Ubuntu.xml COPY files/fc2k/docker-entrypoint.sh / ``` - `ual` starts from `base` image and copies from `ual-devel` all things built previously (BLAS, LAPACK, UDA, IMAS) - This way the `ual` image is free of the `*-devel` software (i.e. the Intel compilers) and IMAS source codes ``` dockerfile FROM base AS ual COPY --from=base-devel /opt/blas /opt/blas COPY --from=base-devel /opt/lapack /opt/lapack COPY --from=ual-devel /opt/uda /opt/uda COPY --from=ual-devel /opt/imas /opt/imas ENV PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:/opt/blas/lib/pkgconfig:/opt/lapack/lib/pkgconfig \ UDA_HOME=/opt/uda ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${UDA_HOME}/lib/plugins \ LIBRARY_PATH=${LIBRARY_PATH}:${UDA_HOME}/lib/plugins \ MODULEPATH=/etc/modulefiles:${UDA_HOME}/modulefiles:/opt/imas/etc/modulefiles \ PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:${UDA_HOME}/lib/pkgconfig \ PYTHONPATH=${UDA_HOME}/lib/python3.6/site-packages \ UDA_DISABLED=0 \ UDA_IMAS_MACHINE_MAP=${UDA_HOME}/etc/udaMachines.txt \ UDA_IMAS_PLUGIN_MAP=${UDA_HOME}/etc/udaMappings.txt \ UDA_LOG=/tmp \ UDA_LOG_LEVEL=ERROR \ UDA_LOG_MODE=a \ UDA_PLUGIN_CONFIG=${UDA_HOME}/etc/udaPlugins.conf \ UDA_PLUGIN_DEBUG_SINGLEFILE=1 \ UDA_PLUGIN_DIR=${UDA_HOME}/lib/plugins \ UDA_SARRAY_CONFIG=${UDA_HOME}/etc/udagenstruct.conf COPY files/ual/docker-entrypoint.sh / ENTRYPOINT ["/docker-entrypoint.sh"] CMD ["/bin/bash"] ``` - `kepler` extends it and copies from `kepler-devel` ``` dockerfile FROM ual AS kepler COPY files/kepler/jaxfront /etc/modulefiles/ ARG tag_ant COPY --from=kepler-devel /opt/apache-ant-${tag_ant} /opt/apache-ant-${tag_ant} COPY --from=kepler-devel /opt/imas/etc/modulefiles/Kepler /opt/imas/etc/modulefiles/Kepler COPY --from=kepler-devel /opt/imas/etc/modulefiles/Keplertools /opt/imas/etc/modulefiles/Keplertools COPY --from=kepler-devel /opt/imas/extra/Kepler /opt/imas/extra/Kepler COPY --from=kepler-devel /opt/imas/extra/Keplertools /opt/imas/extra/Keplertools COPY --from=kepler-devel /opt/jaxfront /opt/jaxfront COPY --from=kepler-devel /root/.local/Kepler /root/.local/Kepler COPY --from=kepler-devel /usr/bin/ant /usr/bin/ant ENV USER=root ENV KEPLER=${USER}/.local/Kepler/default/kepler COPY files/kepler/docker-entrypoint.sh / ``` - `fc2k` extends it and copies from `fc2k-devel` ``` dockerfile FROM kepler AS fc2k COPY --from=fc2k-devel /opt/imas/etc/modulefiles/fc2k /opt/imas/etc/modulefiles/fc2k COPY --from=fc2k-devel /opt/imas/extra/fc2k /opt/imas/extra/fc2k COPY files/fc2k/docker-entrypoint.sh / ``` - `gui` extends it and adds XFCE4 and TigerVNC ``` dockerfile FROM fc2k AS gui # the password to VNC is imas COPY files/gui/passwd /root/.vnc/ COPY files/gui/xstartup /root/.vnc/ COPY files/gui/Kepler.desktop /root/Desktop/ ARG tag_tigervnc RUN apt-get update \ && apt-get install -y \ xfce4 \ && rm -rf /var/lib/apt/lists/* RUN curl -L https://sourceforge.net/projects/tigervnc/files/stable/${tag_tigervnc#v}/tigervnc-${tag_tigervnc#v}.x86_64.tar.gz | tar xz -C /opt CMD ["/opt/tigervnc-${tag_tigervnc#v}.x86_64/usr/bin/vncserver", "-fg", ":1"] ``` - There are 7 images produced: - `imas/ual-devel` - `imas/kepler-devel` - `imas/fc2k-devel` - `imas/ual` - `imas/kepler` - `imas/fc2k` - `imas/gui` ## `push.sh` and `save.sh` - The Bash script `push.sh` tags all images with registry prefix and pushes them there - The Bash script `save.sh` saves the non-devel images as `.tar.zst` in `/tmp` directory |
Dockerfile graph
This work has been carried out within the framework of the EUROfusion Consortium, funded by the European Union via the Euratom Research and Training Programme (Grant Agreement No. 101052200—EUROfusion). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the European Commission can be held responsible for them. The scientific work is published for the realization of the international project co-financed by Polish Ministry of Science and Higher Education in 2021 from financial resources of the program entitled "PMW” 5218/HEU - EURATOM/2022/2