Prerequisites
You’ll need a laptop with a web browser and a terminal. See also the Setup page.
You might want to brush up on the fundamentals of the Linux Shell for HPC.
This tutorial has been designed in a modular way. Once you have completed the introductory episodes, there are almost no further dependencies between episodes. You can then choose the topics that interest you and follow the corresponding episodes independently.
This tutorial is an update of previous Pawsey training material, so we apologise in advance if you encounter any remaining references to the original material. We have done our best to bring the content up to date, but the work is still in progress.
A webinar series for the old episodes is available on the Pawsey Centre Youtube channel! Playlist here. You can download the webinar slides here.
Like this tutorial? Star it on Github (active link on top right)!
| Setup | ||
| 00:00 | 1. Setup Docker on your computer | How can I install Docker and verify that it works on my computer? |
| 00:15 | 2. Get ready on Setonix |
How can I connect to Setonix from Linux, macOS, or Windows?
How can I request an interactive allocation for the training? How can I clone the training repository into my scratch directory? How can I start downloading container images needed later in the training? |
| 00:30 | 3. Introduction to containers |
What are containers for?
Who is using containers in HPC ecosystems? Why do shared HPC systems use Singularity instead of Docker to run containers? |
| 00:40 | 4. Basic Use of Containers (with Singularity) |
How can I find and run an existing container image with Singularity?
How can I save and organise container images as SIF files? What is the difference between singularity run, exec, and shell? How can I execute pipelines and other shell expressions inside a container? |
| 01:55 | 5. Share files with the host: BLAST, a bioinformatics demo | |
| 02:15 | 6. Building Docker/OCI images for HPC with Singularity |
Why do we build images with Docker but run them with Singularity on a shared HPC system?
How can I describe and build a reproducible container image with a Dockerfile? How can I test the image locally before using it on a cluster? How can I publish an image to a registry and convert it to a SIF file? |
| 03:05 | 7. Running MPI Applications in Containers: An OpenFOAM Example |
How can an MPI application packaged in a container run across HPC compute nodes?
How are host MPI and interconnect libraries made available inside a container? What performance overhead can MPI containers introduce compared with native execution? |
| 04:30 | 8. "Writable" containers: bind mounts, overlays, Trinity and Conda |
Where can a containerised application store files when its container image is read-only?
How can I make a specific host directory available inside a container? Can files created within the container filesystem persist between container runs? Can I add software to a container without rebuilding its image? How can I avoid storing thousands of individual application or installation files directly on a parallel filesystem? What should I use when changes to the container filesystem are needed only during one container run? |
| 05:10 | 9. Deep learning with GPU containers: a PyTorch example | |
| 05:25 | 10. Building Docker/OCI images for HPC, Part 2: Robust and portable images |
How can I reduce the size of an image?
What practices make a Docker image robust and suitable for HPC use? How can I validate a Docker image on my computer before running it on Setonix? How can private dependencies be accessed during a build without storing credentials in the image? How can I manage local Docker images, containers and build cache, and distinguish them from images stored in a registry? |
| 06:30 | Finish |
The actual schedule may vary slightly depending on the topics and exercises chosen by the instructor.