previous episode

Intro to Docker (Pawsey Centre)

next episode

Making Python not awful with containers

Overview

Teaching: 0 min
Exercises: 20 min
Questions
Objectives

  • Run Python applications using container

  • Use JupyterHub in a container

Why can Python be Awful?

Python is a great language for doing all kinds of work, but sometimes it can present issues…

Python can get messy

Build and run a basic Python app

Let’s start by running a very simple hello world example with a basic Python container. To start, cd to the /demos/07_python directory and create a basic helloworld.py:

#!/usr/bin/env python

print("Hello World")

And we can run this without even building a new Docker image:

chmod +x helloworld.py

docker run --rm -v $(pwd):/app -w /app python:3.6-slim python ./helloworld.py

Unable to find image 'python:3.6-slim' locally
3.6-slim: Pulling from library/python
fc7181108d40: Pull complete
01f3b6f74410: Pull complete
d8e6069e053d: Pull complete
e6bd64ad4fe3: Pull complete
a034e22b86de: Pull complete
Digest: sha256:0d095570901e7cf0dac93ba4d0ee75ee2364715338b0396874589c9cc23343b6
Status: Downloaded newer image for python:3.6-slim

Hello World

Adding Python modules to a container

We want to be able leverate other Python modules to do some acutal work, so we need to be able to build Python containers that use tools like pip and conda.

For this example, cd to the demos/07_python/logistic-regression and we’ll trying installing some ML packages to run a logistic regression model.

We have both a Jupyter notebook (LogisticRegression.ipynb) and source code (logreg.py) that we’ll use. An example Dockerfile is also provided.

Build a custom Jupyter/Python container For this example you should try building your own Docker container. Some hints:

  • Decide if you should use a base image (the Jupyter Docker Stacks might be a good place to start)
  • You’ll need to install the Plotly pacakge

Solution

FROM jupyter/datascience-notebook:latest

RUN conda install plotly

You can build it with

docker build -t logreg-py .

After you’ve built your image we’ll start up the container and log into our Jupyter notebook server:

docker run --name=jupyter-logreg -d -p 80:8888 -v $(pwd):/home/jovyan -w /home/jovyan jupyter-logreg:latest

Then you’ll need to use docker logs to find out the access key for your Jupyter server:

docker logs jupyter-logreg

Executing the command: jupyter notebook
[I 02:55:14.895 NotebookApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
[I 02:55:16.361 NotebookApp] JupyterLab extension loaded from /opt/conda/lib/python3.7/site-packages/jupyterlab
[I 02:55:16.361 NotebookApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 02:55:16.363 NotebookApp] Serving notebooks from local directory: /home/jovyan
[I 02:55:16.363 NotebookApp] The Jupyter Notebook is running at:
[I 02:55:16.363 NotebookApp] http://(54bdc6f5c9cd or 127.0.0.1):8888/?token=82be2f08380f138a92ad9c6323f7fcfd124c453144492d3f
[I 02:55:16.363 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 02:55:16.387 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///home/jovyan/.local/share/jupyter/runtime/nbserver-6-open.html
    Or copy and paste one of these URLs:
        http://(54bdc6f5c9cd or 127.0.0.1)/?token=82be2f08380f138a92ad9c6323f7fcfd124c453144492d3f

Point your web browser to **http://<nimbus-ip/?token=** and we'll go through the notebook.

Running Python code without Jupyter

Because our Jupyter container has a functioning version of Python installed, we can also run our container and Python code directly from the command line:

docker run --name=python-logreg -v $(pwd):/home/jovyan -w /home/jovyan jupyter-logreg:latest python ./logreg.py

You should have plot now saved as a .png file.

Machine Learning Example

Let’s try another ML example. We’ll build another Docker image that uses Jupyter. For this this example, cd to the demos/07_python/image-classification.

In here we have some image data (dataset/), a Jupyter notebook, and a Dockerfile.

Just like before, we’ll have you build your own Jupyter container.

Build a custom Jupyter/Python container For this example you should try building your own Docker container. Some hints:

  • Decide if you should use a base image (the Jupyter Docker Stacks might be a good place to start)
  • You’ll need to install 2 modules: mahotas annd opencv-python

Solution

FROM jupyter/datascience-notebook:latest

RUN conda install mahotas
RUN pip install opencv-python

You can build it with

docker build -t jupyter-image .

After you’ve built your image we’ll start up the container and log into our Jupyter notebook server:

docker run --name=jupyter-image -d -p 80:8888 -v $(pwd):/home/jovyan -w /home/jovyan jupyter-image:latest

Then you’ll need to use docker logs to find out the access key for your Jupyter server:

docker logs jupyter-images

Executing the command: jupyter notebook
[I 02:55:14.895 NotebookApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
[I 02:55:16.361 NotebookApp] JupyterLab extension loaded from /opt/conda/lib/python3.7/site-packages/jupyterlab
[I 02:55:16.361 NotebookApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 02:55:16.363 NotebookApp] Serving notebooks from local directory: /home/jovyan
[I 02:55:16.363 NotebookApp] The Jupyter Notebook is running at:
[I 02:55:16.363 NotebookApp] http://(54bdc6f5c9cd or 127.0.0.1):8888/?token=82be2f08380f138a92ad9c6323f7fcfd124c453144492d3f
[I 02:55:16.363 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 02:55:16.387 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///home/jovyan/.local/share/jupyter/runtime/nbserver-6-open.html
    Or copy and paste one of these URLs:
        http://(54bdc6f5c9cd or 127.0.0.1)/?token=82be2f08380f138a92ad9c6323f7fcfd124c453144492d3f

Point your web browser to **http://<nimbus-ip/?token=** and we'll go through the notebook.

Key Points

  • Containers are great way to manage Python workflows

previous episode

next episode