Run containers on HPC with Shifter (and Singularity)


Teaching: 15 min
Exercises: 0 min
  • Learn how to manage and run containers on a HPC cluster with Shifter

Why not Docker on HPC?

There are a few issues preventing Docker from being used as a container engine on HPC systems:

Fortunately, a number of alternatives are available to run containers at HPC facilities, including:

At the moment, Pawsey is using CSCS Shifter on its HPC systems, and therefore this will be the tool of choice in this tutorial.

NCI (the National Computational Infrastructure in Canberra) is using Singularity on its HPC systems, for which examples will be provided as well.

How to login on Pawsey HPC systems?

Pawsey currently has two sytems, Magnus and Zeus. We’re using Zeus for this tutorial. You can login using the ssh command and your Pawsey access credentials (they will be provided for live workshops):

$ ssh <your-pawsey-account-name>

After this, you’ll be prompted to enter your Pawsey account password.

Pulling and managing images with Shifter

To use Shifter on Pawsey HPC systems, we need first to load the corresponding module:

$ module load shifter

In principle, the command to pull container images is very similar to Docker, shifter pull.
However, to avoid disk quota issues on Pawsey HPC systems, the following syntax is recommended, that makes use of the sg linux command, for instance:

$ sg $PAWSEY_PROJECT -c 'shifter pull ubuntu'
# image     :
# cacheDir  : /group/shifterrepos/mdelapierre/.shifter/cache
# tmpDir    : /tmp
# imageDir  : /group/shifterrepos/mdelapierre/.shifter/images
> save image layers ...
> pulling        : sha256:f85999a86bef2603a9e9a4fa488a7c1f82e471cbb76c3b5068e54e1a9320964a
> pulling        : sha256:da1315cffa03c17988ae5c66f56d5f50517652a622afc1611a8bdd6c00b1fde3
> extracting     : /group/shifterrepos/mdelapierre/.shifter/cache/sha256:f85999a86bef2603a9e9a4fa488a7c1f82e471cbb76c3b5068e54e1a9320964a.tar
> make squashfs ...
> create metadata ...
# created: /group/shifterrepos/mdelapierre/.shifter/images/
# created: /group/shifterrepos/mdelapierre/.shifter/images/
$ sg $PAWSEY_PROJECT -c 'shifter pull centos'
$ sg $PAWSEY_PROJECT -c 'shifter pull busybox'

Similar again to Docker, we can list locally pulled images with shifter images:

$ shifter images
library/centos                   latest                       ea4b646d9000   2018-11-27T07:05:23   69.62MB
library/busybox                  latest                       7dc9d60af829   2018-12-19T22:31:48   704.00KB
library/ubuntu                   latest                       d71fc6939e16   2018-12-19T22:30:41   29.94MB

and remove undesired images with shifter rmi:

$ shifter rmi busybox
removed image

Running images with Shifter

Let us change directory to our group directory with:


and then run ls using the Ubuntu image we just pulled, via shifter run:

$ shifter run ubuntu ls

The output will display the content of the current host directory!

A few differences in behaviour can be noticed compared to Docker, such that using Shifter typically requires to specify less options and flags:

As additional examples, you might want to run:

$ shifter run ubuntu ls /
$ shifter run ubuntu whoami

Note how no flag is required to run a container interactively. To launch an interactive shell from within the container, just run it without any commands, for instance:

$ shifter run ubuntu
mdelapierre@zeus-1:/group/pawsey0001/mdelapierre$ exit   # or hit CTRL-D

MPI- and GPU- enabled packages with Shifter

Shifter has support to run containers exploiting MPI parallelism and GPU acceleration (the latter only through CSCS Shifter).

Can Shifter build container images?

Shifter does not allow to build container images. The best way to create an image to be pulled and run through it is to use Docker on a distinct machine (see previous episode).

Using Shifter with a job scheduler

Shifter is compatible with SLURM, the job scheduler installed on Pawsey HPC systems. In particular, SLURM job executor srun is compatible with shifter run, and the two syntaxes can be combined together.

As an example, let us cd in the 10_shifter demo directory:

$ cd <top-level>/demos/10_shifter

and have a look at the script, that uses a Ubuntu container to output the machine hostname:

#!/bin/bash -l
#SBATCH --account=<your-pawsey-project>
#SBATCH --partition=workq
#SBATCH --ntasks=1
#SBATCH --time=00:05:00
#SBATCH --export=NONE
#SBATCH --job-name=blast

module load shifter

srun --export=all shifter run ubuntu hostname

Now use your favourite text editor to specify your Pawsey Project ID in the script, and then submit it using SLURM. If you are running this during a live workshop, use the flag --reservation <your-pawsey-reservation> to use the compute nodes that have been reserved for the event:

$ sbatch --reservation <your-pawsey-reservation>

Running BLAST on HPC with Shifter

As a practical use case, let us re-run the BLAST bioinformatics example from a previous episode using Shifter in a HPC environment.

First, let us pull the BLAST container:

$ module load shifter
$ sg $PAWSEY_PROJECT -c 'shifter pull biocontainers/blast:v2.2.31_cv2'

Then, in the 10_shifter demo directory uncompress the database file:

$ cd <top-level>/demos/10_shifter
$ gunzip zebrafish.1.protein.faa.gz

The script in this directory permits to execute the entire bioinformatics example using Shifter and the SLURM scheduler:

#!/bin/bash -l

#SBATCH --account=<your-pawsey-project>
#SBATCH --partition=workq
#SBATCH --ntasks=1
#SBATCH --time=00:05:00
#SBATCH --export=NONE
#SBATCH --job-name=blast

module load shifter

# prepare database
srun --export=all shifter run biocontainers/blast:v2.2.31_cv2 makeblastdb -in zebrafish.1.protein.faa -dbtype prot

# align with BLAST
srun --export=all shifter run biocontainers/blast:v2.2.31_cv2 blastp -query P04156.fasta -db zebrafish.1.protein.faa -out results.txt

Now use your favourite text editor to specify your Pawsey project ID in the script, and then submit this script using SLURM:

$ sbatch --reservation <your-pawsey-reservation>

HPC containers with Singularity at NCI

Raijin, the NCI HPC system, uses Singularity as the container engine, instead of Shifter. However, much of the user-facing interface is similar, or even the same. The biggest difference is that on Raijin you cannot pull and use your own images; instead, you’ll need to contact the NCI Help Desk at and ask for your image to be added to the library.

This allows the NCI staff to inspect and sanitise the containers before use. For example, to ensure that they allow the use of system MPI libraries, or at least contain a compatible version. The images must be able to be built using a build script, rather than being distributed as just an opaque filesystem image. Mount points for the systems Lustre filesystems will also be included on build so that all of your user data is available at the same location as in the native image (e.g. /home, /short, and /g/data).

First of all, let us load the Singularity module:

module load singularity

You can run an interactive shell inside the container using the singularity shell command. Here, we are using a CentOS image:

$ singularity shell /apps/singularity/images/centos7/centos7-latest.simg 
Singularity: Invoking an interactive shell within container...
Singularity centos7-2018051701.simg:~> cat /etc/centos-release
CentOS Linux release 7.5.1804 (Core) 
Singularity centos7-2018051701.simg:~> 
Singularity centos7-2018051701.simg:~> whoami
Singularity centos7-2018051701.simg:~> exit

You can run a specific command within the container using the singularity exec command:

$ singularity exec /apps/singularity/images/centos7/centos7-latest.simg cat /etc/centos-release
CentOS Linux release 7.5.1804 (Core) 

Note that the CentOS version in the container image is 7.5, whereas Raijin’s native OS is (currently) CentOS 6.10:

$ cat /etc/centos-release
CentOS release 6.10 (Final)

On Raijin, Singularity is also integrated with the PBS batch scheduling system. This allows you to specify the image to use via the directive #PBS -l image in your job script (or similarly on the qsub command line):

#PBS -P <your-nci-project>
#PBS -q normal
#PBS -l ncpus=1
#PBS -l walltime=00:05:00
#PBS -l image=centos7

module load singularity

cat /etc/centos-release

Of course, you can also just use singularity exec within your job script as in the example above:

#PBS -P <your-nci-project>
#PBS -q normal
#PBS -l ncpus=1
#PBS -l walltime=00:05:00

module load singularity

singularity exec /apps/singularity/images/centos7/centos7-latest.simg cat /etc/centos-release

Key Points

  • Shifter has a quite simple syntax that allows to pull, manage and run containers on HPC systems

  • shifter pull and shifter run are the key commands