previous episode

Containers at Pawsey with Singularity

next episode

Basic Use of Containers (with Singularity)

Overview

Teaching: 45 min
Exercises: 30 min
Questions

  • How can I find and run an existing container image with Singularity?

  • How can I save and organise container images as SIF files?

  • What is the difference between singularity run, exec, and shell?

  • How can I execute pipelines and other shell expressions inside a container?

Objectives

  • Identify the components of a Singularity Docker/OCI URI and registry image reference

  • Download and organise an image as a local SIF file

  • Run predefined and user-selected commands from a container

  • Inspect the software environment packaged inside a container

  • Run pipelines and other shell expressions inside a container with bash -c

  • Open an interactive shell inside a container

Request an interactive allocation

If you’re running this tutorial on a shared system (e.g. Setonix at Pawsey), you should use one of the compute nodes rather than the login node. You can do this by requesting an interactive allocation from the scheduler, for instance on Setonix with Slurm (do this if you are not in an salloc interactive session yet):

$ salloc -N 1 -n 1 -c 8 --reservation=ContainersTraining -t 4:00:00

salloc: Granted job allocation 3453895
salloc: Waiting for resource configuration
salloc: Nodes nid000152 are ready for job

Get ready for the hands-on

Before we start, let us ensure we have got the required files to run the tutorials.

If you haven’t done so already, move to a suitable working directory and download the following GitHub repository. On Pawsey systems, use your scratch directory; on other HPC or cloud systems, use the equivalent working directory recommended by the system administrators.

$ cd "$MYSCRATCH"    # On Pawsey systems
$ git clone https://github.com/PawseySC/singularity-containers
$ export TUTO="$PWD/singularity-containers"

Now cd to the working directory. In this case:

$ cd "${TUTO}/demos/basic_use"
$ pwd

The working directory should be something like:

/path/to/your/scratch/singularity-containers/demos/basic_use

Singularity: the container engine used in this training

SingularityCE, referred to as Singularity throughout this training, is a container engine designed for shared HPC environments. It allows users to run containers without requiring elevated privileges and integrates containerised applications with host filesystems, schedulers, networks, and HPC hardware.

Throughout this training, we use Singularity to run containers. Many of the images used with Singularity were originally built and published through the Docker/OCI ecosystem. Singularity can retrieve these images directly from compatible registries, convert them to the Singularity Image Format (SIF), and run them without requiring the Docker engine.

Most of the commands and concepts in this episode also apply to Apptainer, an open-source continuation of the Singularity project.

Exploring scientific images on Docker Hub

As discussed in the introductory episode, container registries store and distribute container images. Before building your own image, check whether the application developers, a software vendor, or another trusted organisation already provides a suitable image.

Docker Hub is a widely used public container registry. It contains images published by software vendors, open-source projects, organisations, and individual users.

Before using our first image, let us explore how scientific software is presented on Docker Hub.

Explore Docker Hub

Open Docker Hub in a web browser and search for each of the following image repositories:

rocker/rstudio
tensorflow/tensorflow
opencfd/openfoam-default

These repositories provide images for different types of scientific work:

  • rocker/rstudio provides an RStudio Server environment for statistical computing and data analysis.
  • tensorflow/tensorflow provides the TensorFlow machine-learning framework, with different image variants for CPU, GPU, development, and interactive environments.
  • opencfd/openfoam-default provides OpenFOAM applications, runtime libraries, source code, development tools, and tutorial cases for computational fluid dynamics.

For each repository, inspect:

  • the organisation or user that published the image
  • the image description and documentation
  • the available tags
  • when the image was last updated
  • the supported CPU architectures
  • the approximate compressed image size
  • whether source files or build instructions are linked
  • whether the repository provides different image variants

Do not download any of these images yet. Some scientific application images are large, and we are only exploring how images and their metadata are presented in Docker Hub.

Were there other repositories with similar names in the search results? What information would help you decide which publisher to trust?

A repository can publish multiple related images through its tags. A tag may indicate:

Consequently, selecting a container image requires more than finding a repository associated with a familiar application. You must also identify the publisher and select an appropriate tag.

Docker Hub contains repositories from many publishers. Finding a repository associated with the desired application does not automatically mean that its images are trustworthy, maintained, compatible with your target system, or suitable for research use. When evaluating an image, consider:

Finding the image used in this episode

The scientific images that we have explored provide realistic examples, but some of them are large or require additional configuration. For our first commands, we will use a smaller and simpler teaching image.

Search Docker Hub for:

sylabsio/lolcow

Open the repository identified on Docker Hub as sylabsio/lolcow.

This repository path contains:

A tagged image reference for this repository is:

sylabsio/lolcow:latest

Here, latest is the tag associated with the published image. We will discuss the meaning and limitations of this tag later in the episode.

A teaching image

We use an image published in the sylabsio/lolcow repository because it produces an immediate and visually distinctive result without requiring input data or application-specific knowledge.

It should not be interpreted as a recommendation for research or production workloads. Its role is to help us learn the basic Singularity commands before working with larger scientific application images.

Running this training on Setonix

The following setup is specific to Setonix. If you are completing this training on another system, use the Singularity or Apptainer installation provided by that system.

Before continuing, confirm that you are working inside an interactive salloc session on a Setonix compute node, as described in the hands-on preparation section above. Do not run the following container commands directly on a login node.

You can confirm the current node with:

$ hostname

A Setonix compute-node hostname begins with nid.

Once you are on a compute node, load the Singularity module:

$ module load singularity/4.1.0-nompi

On Setonix, multiple Singularity module variants are available. These variants configure different levels of integration between the container and the host software environment, including support for MPI, GPUs, and Slurm.

The nompi suffix identifies the module variant intended for applications that do not require MPI communication, including many bioinformatics applications. It avoids injecting the host MPI software environment into the container.

If the container requires stronger isolation from the host software environment, the nohost variant may be more appropriate. This variant avoids additional specialised host integrations. Note that configured host filesystems, such as /scratch, may still be available inside the container.

List the available Singularity module variants with:

$ module avail singularity

For a detailed explanation of the available variants, see the Singularity documentation in the Pawsey User Support Documentation.

Confirm that the expected Singularity version is available after loading the module:

$ module list

The correct module should be listed as loaded:

Currently Loaded Modules:
...
15) singularity/4.1.0-nompi

Check the version directly using the now available singularity command:

$ singularity --version

The output should identify SingularityCE version 4.1.0:

singularity-ce version 4.1.0

You only need to load the module once in each terminal session. If you open another terminal, start a new login session, or submit a batch job, load the module again in that environment before invoking singularity.

Downloading an image as a SIF file

Singularity uses the Singularity Image Format (SIF) for its native container images. A SIF image is a single file that can be copied, renamed, moved, and stored like any other file.

For images that you intend to keep and reuse, we recommend storing named SIF files in an organised personal or project image library.

First, create a directory to use as your local Singularity image library:

$ export MY_LOCAL_LIBRARY="${MYSOFTWARE}/singularity/images"
$ mkdir -p "$MY_LOCAL_LIBRARY"

On Docker Hub, the repository page presents the selected image with a Docker command similar to (do no use it here, as Docker is not installed on Setonix):

docker pull sylabsio/lolcow:latest

This command contains two conceptually different parts:

The components of this registry image reference are:

In everyday language, people commonly call lolcow, sylabsio/lolcow, or sylabsio/lolcow:latest the image name. This is convenient but less precise. In this episode, repository name refers specifically to lolcow, while registry image reference refers to sylabsio/lolcow:latest or its registry-qualified form.

We are using Singularity rather than Docker, so we will not run the docker pull command. Instead, we use the namespace, repository, and tag shown by Docker Hub to construct a Singularity Docker/OCI URI.

The general form of a Singularity URI is:

SCHEME://SOURCE

When using singularity pull, the URI scheme tells Singularity what type of source it must retrieve and how to access it. The scheme must match the source. Common examples include:

In this example, Docker Hub distributes the source image in the Docker/OCI format, so we must use the docker:// scheme.

The general form of a Singularity Docker/OCI URI is:

docker://REGISTRY/NAMESPACE/REPOSITORY:TAG

For the image selected on Docker Hub, the explicit URI is:

docker://docker.io/sylabsio/lolcow:latest

Its components are:

The registry hostname must identify the registry containing the repository. For example:

An image published in Quay would use the same docker:// scheme but a different registry hostname:

docker://quay.io/NAMESPACE/REPOSITORY:TAG

The docker:// scheme does not instruct Singularity to start Docker. It tells Singularity to interpret the source as Docker/OCI image content and retrieve it using the appropriate registry protocol. The registry hostname tells Singularity which registry to contact. Docker does not need to be installed or running on the system.

The general form of singularity pull is:

$ singularity pull [OUTPUT_FILE] URI

For this training, provide the complete output path explicitly:

$ singularity pull "${MY_LOCAL_LIBRARY}/lolcow--latest.sif" \
    docker://docker.io/sylabsio/lolcow:latest

This command connects three related but distinct identifiers:

Docker Hub registry image reference:
sylabsio/lolcow:latest

Singularity Docker/OCI URI:
docker://docker.io/sylabsio/lolcow:latest

Local SIF path:
${MY_LOCAL_LIBRARY}/lolcow--latest.sif

The source URI tells Singularity what image content to retrieve. The output path tells Singularity where to save the converted SIF image and what local filename to use.

Check that the SIF file was created:

$ ls -lh "$MY_LOCAL_LIBRARY"

A filesystem directory containing SIF files is not technically a container registry. It is an organised local collection, or image library, that you manage yourself.

Optional: Letting Singularity choose the SIF filename

The output filename can be omitted:

$ singularity pull docker://docker.io/sylabsio/lolcow:latest

Singularity then derives the filename from the repository name and tag and creates the SIF file in the current working directory:

lolcow_latest.sif

You can use --dir to select another output directory while still allowing Singularity to generate the filename:

$ singularity pull --dir "$MY_LOCAL_LIBRARY" \
    docker://docker.io/sylabsio/lolcow:latest

This creates:

${MY_LOCAL_LIBRARY}/lolcow_latest.sif

The --dir option controls only the output directory. Singularity still generates the filename automatically using an underscore between the repository name and tag.

We do not use this automatic filename in the training because repository names and tags can themselves contain hyphens or underscores. In such cases, it can be difficult to distinguish where the repository name ends and where the tag begins.

Our local naming convention uses two hyphens, --, as a clear separator:

repository--tag.sif

For this image, the resulting filename is:

lolcow--latest.sif

Providing the complete output path explicitly allows us to select both the destination directory and this clearer filename:

$ singularity pull "${MY_LOCAL_LIBRARY}/lolcow--latest.sif" \
    docker://docker.io/sylabsio/lolcow:latest

Optional: Omitting docker.io from Docker Hub URIs

Docker Hub is the default registry for Singularity docker:// URIs. Therefore:

docker://sylabsio/lolcow:latest

is equivalent to:

docker://docker.io/sylabsio/lolcow:latest

This tutorial uses the explicit docker.io hostname to make the registry visible. You will commonly encounter the abbreviated form in documentation and existing workflows.

pull or build?

Use singularity pull when obtaining an existing image from a registry:

$ singularity pull output.sif docker://registry/namespace/repository:tag

singularity build would have also worked for this example, but it is a more general command. It can also create a SIF image from a registry reference, but it is principally introduced later when building or customising images from definition files.

Running a container’s default action

The image is now available as a SIF file in your personal image library. Define a variable containing its path so that it can be referenced more conveniently in subsequent commands:

$ COW_IMAGE="${MY_LOCAL_LIBRARY}/lolcow--latest.sif"

Singularity provides three main commands for running or interacting with a container image:

We will use all three commands in this episode. First, use singularity run to execute the default action provided by the lolcow image:

$ singularity run "$COW_IMAGE"

 ______________________________
< Fri Sep 4 16:54:42 AWST 2026 >
 ------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

The exact message and colours will vary because the image generates the output dynamically.

The image’s default action combines several command-line programs:

Conceptually, the image runs a pipeline similar to:

date | cowsay | lolcat

These programs were not developed specifically for Singularity, and they can be installed directly on many Linux distributions. In this example, however, they are provided by the container image and do not need to be installed on the host system. (BTW, lolcow is just the name of the image but not an existing command or script in the image or anywhere.)

You can check whether the commands are available directly on the host:

$ command -v date cowsay lolcat

If a command is not available, command -v does not print a path for it. Regardless of whether any of these programs happen to be installed on the host, the copies packaged inside the container are available when the container is used.

This illustrates an important benefit of containers: the required applications are supplied by the image instead of depending on software installed separately on each host system.

The run command starts a container from the SIF image and executes the image’s predefined runscript. The runscript is configured by the image publisher when the image is built.

The general form of the singularity command is:

singularity run IMAGE [ARGUMENTS...]

Running an image does not necessarily open an interactive session. The action performed by singularity run depends on the runscript defined in that particular image. For lolcow, the runscript generates a random message and displays it using an ASCII-art cow.

In the following sections, we will use:

Optional: Running an image using a Docker/OCI URI

Creating a named SIF file with singularity pull gives you control over where the image is stored and how it is named. This is the recommended approach for images that you intend to manage and reuse.

For a quick test, Singularity can also retrieve and run an image using its Docker/OCI URI:

$ singularity run docker://docker.io/sylabsio/lolcow:latest

The first time this URI is used, Singularity may display informational messages while it retrieves and prepares the image:

INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
[...]
INFO:    Creating SIF file...

Singularity then starts a container and executes the runscript provided by the image.

When Singularity processes the command, it:

  1. reads the Singularity Docker/OCI URI
  2. retrieves the Docker/OCI image manifest and filesystem layers from Docker Hub
  3. converts the image into the Singularity Image Format
  4. stores the converted image in its internal cache
  5. starts a container from the cached image
  6. executes the runscript defined by the image publisher

Run the same command again:

$ singularity run docker://docker.io/sylabsio/lolcow:latest

The second execution should start sooner because Singularity can reuse the converted image stored in its internal cache. In both executions, the container runs locally. The Docker/OCI URI tells Singularity where to retrieve the image, but it does not mean that the image is executed remotely.

Local SIF file or Docker/OCI URI?

The two commands use the same published image, but they manage it differently.

Run the SIF file stored in your personal image library:

$ singularity run "$COW_IMAGE"

Run the image using its Singularity Docker/OCI URI:

$ singularity run docker://docker.io/sylabsio/lolcow:latest

Using a Docker/OCI URI is convenient for quickly testing an image. Singularity manages the converted image in its internal cache, where cached objects may be identified by content-based hashes rather than recognisable repository names.

For images that you intend to retain and use in research workflows, prefer an explicitly named SIF file stored in your personal or project image library.

Docker/OCI image, SIF image, Docker Hub, Docker, and Singularity

These related concepts should not be confused:

  • A Docker/OCI image is an image packaged according to the Docker/OCI image format. In a registry, it is commonly distributed as a manifest, configuration metadata, and a set of filesystem layers.
  • A SIF image is a container image stored in the Singularity Image Format. It is normally represented by a single .sif file.
  • Docker Hub, identified by docker.io, is an online registry that stores and distributes Docker/OCI images.
  • Docker is a container platform and engine that can build, distribute, and run Docker/OCI images (not used in this episode).
  • Singularity is the container engine used in this episode to run images. It can retrieve a Docker/OCI image from Docker Hub or another compatible registry, convert it into SIF, and run the resulting container without using the Docker engine.

In this example:

Docker Hub
    |
    |  Registry image reference:
    |  docker.io/sylabsio/lolcow:latest
    v
Singularity retrieves and converts the image
    |
    |  Singularity SIF image:
    |  lolcow--latest.sif
    v
Singularity runs the container

Docker Hub is the source registry, docker.io/sylabsio/lolcow:latest is the registry image reference, and lolcow--latest.sif is the local SIF image created by Singularity. Singularity retrieves the Docker/OCI image content, converts it to SIF, and runs the resulting container. The Docker engine is not involved.

Running a command in a container

singularity run executes the default action defined by the image publisher. To execute a command of your choice, use singularity exec:

$ singularity exec "$COW_IMAGE" cowsay "Hello from my local SIF image!"

The general form of the singularity command is:

singularity exec IMAGE COMMAND [ARGUMENTS...]

The image is followed by the command to execute and any arguments to pass to it.

Ask the cowsay application for help:

$ singularity exec "$COW_IMAGE" cowsay -h

List the figures packaged with cowsay:

$ singularity exec "$COW_IMAGE" cowsay -l

Choose a figure

Select one of the figures reported by cowsay -l and print your own message. For example, if the image includes the dragon figure:

$ singularity exec "$COW_IMAGE" cowsay -f dragon "Running from a container!"

Running cowsay without a message

If you run cowsay without providing a message:

$ singularity exec "$COW_IMAGE" cowsay

the program waits for input from the terminal. Type your message, press Enter, and then press Ctrl+D to indicate the end of the input. cowsay will then display the message.

Pressing Ctrl+C interrupts and cancels the command instead.

Providing the message as a command-line argument is usually simpler.

Inspecting the container environment

The image packages both its applications and the user-space environment required by them. Compare the operating-system information visible on the host with that inside the container.

On the host:

$ cat /etc/os-release

Inside the container:

$ singularity exec "$COW_IMAGE" cat /etc/os-release

The outputs may describe different Linux distributions or releases. The information printed inside the container describes the user-space environment packaged in the image. The container does not boot its own kernel. Its processes continue to use the host system’s Linux kernel, as discussed in the introductory episode.

You can also use which to locate the three commands used by the image’s default action:

$ singularity exec "$COW_IMAGE" which date cowsay lolcat

/bin/date
/usr/games/cowsay
/usr/games/lolcat

These paths belong to the container’s filesystem. The commands do not need to be installed on the host. This output is different from that obtained when trying to locate the programs in the host.

Running shell expressions with bash -c

The command passed directly to singularity exec must be an executable that Singularity can start. Some useful shell operations are not separate executable files.

For example, if we invoque command -v directly as another mean to locate the important tools in the image, we would get an error:

$ singularity exec "$COW_IMAGE" command -v date cowsay lolcat

FATAL:   "command": executable file not found in $PATH

This fails because command is a shell built-in that reports how a shell would resolve one or more command names. But it is not a separate executable that Singularity can start.

To use a shell built-in like this one, start Bash inside the container and use its -c option:

$ singularity exec "$COW_IMAGE" bash -c 'command -v date cowsay lolcat'

/bin/date
/usr/games/cowsay
/usr/games/lolcat

The -c option tells Bash to interpret the following quoted string as a shell command. The quotation marks keep the complete command string together so that it can be passed to Bash inside the container.

This pattern is useful whenever the operation to execute inside a container includes shell features such as:

Add colour to the cow

The output produced by cowsay is not colourful, while the image’s default action produces colourful output.

The image also contains lolcat, which adds terminal colours to text. How can you pass the output from cowsay to lolcat to end with a colourful message?

Naive solution

A first attempt may be:

$ singularity exec "$COW_IMAGE" cowsay "Hello from my local SIF image!" | lolcat

bash: lolcat: command not found

This does not run the complete pipeline inside the container. The host shell interprets the pipe before Singularity starts:

singularity exec "$COW_IMAGE" cowsay "Hello..."  |  lolcat
                     runs inside the container      runs on the host

Therefore, cowsay runs inside the container, but the host shell searches for lolcat on the host. The command fails if lolcat is not installed in the host.

Solution

Pass the complete pipeline as a quoted command string to Bash inside the container:

$ singularity exec "$COW_IMAGE" bash -c 'cowsay "Hello from my local SIF image!" | lolcat'

The host shell passes the quoted pipeline as one argument to bash -c. Bash inside the container interprets the pipe, so both cowsay and lolcat run inside the container.

Reproduce the image’s default action using singularity exec

The image’s default action uses:

  • date to print the current date and time
  • cowsay to place that text in an ASCII-art speech bubble
  • lolcat to add terminal colours

Use singularity exec and bash -c to combine these commands into a pipeline that reproduces the image’s default action.

Solution

$ singularity exec "$COW_IMAGE" bash -c 'date | cowsay | lolcat'

Bash runs inside the container and interprets both pipe operators. Therefore, all three commands are resolved and executed using the container environment.

This pipeline produces the same type of output as:

$ singularity run "$COW_IMAGE"

singularity run executes the pipeline already defined in the image’s runscript, while singularity exec with bash -c specifies the pipeline explicitly.

Opening an interactive shell in a container

The singularity exec command runs a specified command and then returns control to the host shell. For interactive inspection and troubleshooting, use singularity shell instead:

$ singularity shell "$COW_IMAGE"

The prompt changes to Singularity> to indicate that you are interacting with a shell inside the container environment:

Singularity>

Commands entered at this prompt are interpreted by the shell running inside the container. For example, inspect the packaged operating-system environment:

Singularity> cat /etc/os-release

Because a shell is already running inside the container, shell built-ins such as command can be used directly without bash -c:

Singularity> command -v date cowsay lolcat

/bin/date
/usr/games/cowsay
/usr/games/lolcat

Shell operators are also interpreted inside the container. Therefore, the pipeline used in the previous challenge can be entered directly:

Singularity> date | cowsay | lolcat

Similarly, you can provide your own colourful message:

Singularity> cowsay "Running interactively" | lolcat

The important difference is that the interactive container shell now interprets the commands and pipe operators. There is no need to start another shell with bash -c.

Exit the container shell when finished:

Singularity> exit

You can also press Ctrl+D to exit.

An interactive shell is useful for:

For repeatable workflows and batch jobs, prefer singularity exec. Commands passed to exec can be recorded directly in scripts, while commands entered interactively are not automatically preserved.

A standard SIF image is read-only during execution. Exploring the image from an interactive shell does not modify the original SIF image. In the next episode, we will see how containerised applications access and write persistent files on the host.

Image tags and reproducibility

A tag is a human-readable label associated with published image content in a repository. A tag can be moved by the publisher to refer to updated content. The latest tag is only a conventional label: it does not guarantee that the referenced image contains the newest application version.

We use the latest tag from the sylabsio/lolcow repository because it is the tag provided for this teaching example. For research and production workflows, prefer a meaningful version tag when one is available:

docker://docker.io/namespace/repository:1.2.3

A versioned tag communicates the intended software version more clearly, although publishers can technically update tags. Retaining the downloaded SIF file preserves the exact image contents that you obtained at that time.

Getting help

Use singularity help to display general help:

$ singularity help

Add a command name for command-specific help:

$ singularity help pull
$ singularity help run
$ singularity help exec
$ singularity help shell

These commands display help for the Singularity container engine. To display help for an application packaged inside an image, execute that application’s help command through the container, for example:

$ singularity exec "$COW_IMAGE" cowsay -h

Review the basic workflow

In this episode, you followed the basic lifecycle of an existing container image:

  1. Find an image in a registry.
  2. Inspect its publisher, tags, documentation, and compatibility.
  3. Pull it into a named SIF file for regular use.
  4. Use run, exec, and shell for different interactions with the image.
  5. Use bash -c when an operation requires shell syntax.
  6. Optionally, use a Docker/OCI URI for a quick test.

For your own workflows, store reusable SIF files in an organised location and record the original Singularity Docker/OCI URI, registry image reference, and tag from which each file was obtained.

Key Points

  • Singularity can retrieve Docker/OCI images from compatible registries without using the Docker engine

  • singularity run executes the action defined by the image publisher

  • singularity exec executes a command selected by the user

  • singularity shell opens an interactive shell for inspection and troubleshooting

  • Use bash -c when commands executed with singularity exec require shell built-ins, pipelines, redirections, or other shell syntax

  • Use singularity pull to create explicitly named SIF files that you can organise and reuse

  • For images to be downloaded, prefer versioned image tags over latest when they are available

previous episode

next episode