Making Python not awful with containers


Teaching: 0 min
Exercises: 20 min
  • Embed a Python app in a container and run it

Why can Python be Awful?

Python is a great language for doing all kinds of work.

Python can get messy

Build and run a dockerised Python app

This is a quick Python data science example. We’ll build a Python container, add a simple Python app, and run it.

To begin, let’s clone another repo:

$ git clone
$ cd python-demo

There a few files here:

# Use an official Python runtime as a parent image
FROM python:3.6-slim

# Set the working directory to /app

# Copy the current directory contents into the container at /app
ADD . /app

# Install any needed packages specified in requirements.txt
RUN pip install -r requirements.txt

# Define environment variable

# Run when the container launches
CMD ["python", ""]
from sklearn import tree
X = [[181, 80, 44], [177, 70, 43], [160, 60, 38], [154, 54, 37],[166, 65, 40],
     [190, 90, 47], [175, 64, 39],
     [177, 70, 40], [159, 55, 37], [171, 75, 42], [181, 85, 43]]

Y = ['apple', 'apple', 'orange', 'orange', 'apple', 'apple', 'orange', 'orange',
     'orange', 'apple', 'apple']

#classifier - DecisionTreeClassifier
clf_tree = tree.DecisionTreeClassifier();
clf_tree =,Y);

test_data = [[190,70,42],[172,64,39],[182,80,42]];

prediction_tree = clf_tree.predict(test_data);

print("Prediction of DecisionTreeClassifier:",prediction_tree);

There are some aspects of this Dockerfile that are worth mentioning:

Now to build our container we can simply run:

$ docker build -t python-demo .

After that, we can run it with:

$ docker run python-demo

Run a Python app on HPC with Shifter

a) re-use the image we have just built

On our Docker machine let us push the container image we created above to Docker Hub; you’ll need a free account.

First we must give the image a name that complies with the Hub’s nomenclature (see previous episode on build):

$ docker tag python-demo <your-dockerhub-account>/python-demo

Now let’s push the image:

$ docker push <your-dockerhub-account>/python-demo
The push refers to repository []
862d6710cd15: Pushed 
302ce4960403: Pushed 
latest: digest: sha256:4db5f0f69cc888d47f4c4b4cac33fad6b004a8e333b36a699ebd43f5b44a7241 size: 1999

We are now moving to the Pawsey HPC system. Let’s pull the image, then change directory to either $MYSCRATCH or $MYGROUP:

$ module load shifter
$ sg $PAWSEY_PROJECT -c 'shifter pull <your-dockerhub-account>/python-demo'


Let us write a SLURM script to execute our Python app using this container, we’ll use our favourite text editor to save it as (remember to specify your Pawsey project ID in the script!):

#!/bin/bash -l

#SBATCH --account=<your-pawsey-project>
#SBATCH --partition=workq
#SBATCH --ntasks=1
#SBATCH --time=00:05:00
#SBATCH --export=NONE
#SBATCH --job-name=python

module load shifter

# run Python app
srun --export=all shifter run <your-dockerhub-account>/python-demo

Let’s submit it to the SLURM scheduler:

$ sbatch --reservation <your-pawsey-reservation>

b) use a publicly available image for scientific Python

In this case we are going to use jupyter/scipy-notebook:

$ module load shifter
$ sg $PAWSEY_PROJECT -c 'shifter pull jupyter/scipy-notebook'


Let us write a second SLURM script, we’ll call it Contrary to Docker example above, our Python app is not embedded in the image, so we’ll need to explicitly download it from the Git repo, and then run it through the Python interpreter in the container:

#!/bin/bash -l

#SBATCH --account=<your-pawsey-project>
#SBATCH --partition=workq
#SBATCH --ntasks=1
#SBATCH --time=00:05:00
#SBATCH --export=NONE
#SBATCH --job-name=python2

module load shifter

# clone Git repo with the app
git clone
cd python-demo

# run Python app
srun --export=all shifter run jupyter/scipy-notebook python

Finally we are submitting the script with SLURM:

$ sbatch --reservation <your-pawsey-reservation>

Key Points

  • Containers are great way to manage Python workflows