WDF has been designed to work independently from the host operational system. This functionality has been achieved through the use of Docker containers. The following manual describes the installation of WDF using Dockers with optional conversion to Singularity images enabling application on the HPC systems.

Installation

In order to install Docker on you machine, follow the official documentation available here.

Once Docker has been installed on you computer, copy our repository containing Dockerfiles of to install and run WDF. You can, then, clone the [WDF gitlab repo] (https://gitlab.com/wdfpipe/wdf) and install as a standard python package. You can also download images directly from Docker Hub.

Please note that our LATEST-TAG is 2.1.1

In the repository under dockers folder, you will find four subfolders (in parenthesis corresponding tag names from the Docker Hub are given):

  • python-env (wdfteam/wdfpipe:ubuntu-3.11)
  • wdf-env (wdfteam/wdfpipe:wdf_env_2.1.1)

Each subfolder contains three files:

  • Dockerfiles
  • build.sh
  • version.txt

To build Docker image run in the terminal ./build.sh. If the file has no permissions to be executed, you can change them by typing chmod +x build.sh (in case of Linux).

The first Docker image that should be built is python-env. This container contains all basic C++ libraries and packages required by WDF and the python environment. Since there are many dependencies to be installed inside the image, it might take a while until the process is finished.

The last container to be built is wdf-env Again run ./build.sh. Once the build is completed, you can use WDF.

The last Docker container wdf-env is the one that you are going to use with WDF. To run the container, type the following in the terminal:

docker run --name wdf_test -it wdfteam/wdfpipe:wdf_env_2.1.1 /bin/bash

The flag -it is used to run container in the interactive mode - you can enter container directly on your terminal. If you want to run Jupyter inside the container, you need to add another flag -p 8887:8887 to specify the port on which you can reach the container from the host machine:

docker run --name wdf_test -it -p 8887:8887 wdfteam/wdfpipe:wdf_env_2.1.1 /bin/bash

We suggest using port 8887 since it is usually free - default port 8888 might be taken by Jupyter running already on the host machine. Furthermore, to run jupyter inside the container, type:

jupyter notebook --ip 0.0.0.0 --port 8887 --no-browser --allow-root

To enter container from the second terminal, type:

docker exec -it -t wdf_test /bin/bash

To exit container type exit. If there are no jobs running in background, then container will stop its work. To restart it type:

docker restart wdf_test
docker exec -it -t wdf_test /bin/bash

The second command will open container in the terminal.

To remove existing Docker containers and images, type respectively:

docker rm CONTAINER_ID
docker rmi -f IMAGE_ID

To get above-mentioned id’s type respectively for containers and images:

docker ps -a
docker images ls

Singularity installation (optional)

In case of HPC systems, installation of Dockers might be impossible (they require admin permissions). As an alternative you can use Singularity. It is a container platform allowing to work with container in particular on HPC systems with limited access.

To use WDF on systems supporting Singularity, you need to convert local Docker images into Singularity images.

Firstly, you need to install Singularity on your machine. Official documentation with installation can be found here.

IMPORTANT NOTE! Install Singularity >= 3.5. For older versions the conversion from Docker images might not work.

Once Singularity is installed on you machine, you can convert Docker images into Singularity versions via docker2singularity.

To convert the image into Singularity .sif image, type the following in your terminal:

docker run -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/test:/output --privileged -t --rm quay.io/singularity/docker2singularity --name wdf_image wdfteam/wdfpipe:wdf_env_2.1.1

This command will create .sif file called wdf_image under directory /tmp/test (directory has to exist). In case of wdf-env version without Jupyter lab, change Docker image name to wdf/wdf-env:2.1.1.

To enter the Singularity image, type inside /tmp/test directory:

singularity shell wdf_image

To exit the image, type exit in the terminal.

Singularity image generated this way is the production version. The version is read-only and you can’t install new software on it. From the Singularity shell you have an access to the host machine file system.