Build and share your own container image
Overview
Teaching: 10 min
Exercises: 10 minQuestions
Objectives
Build a container image
Share an image with others
Building a basic container
Singularity can build container images in different formats. Let’s focus on the Singularity Image Format, SIF, which is the one typically adopted to ship production-ready containers.
This example is adapted from this well crafted Singularity Tutorial.
Let us cd into the appropriate directory:
$ cd $TUTO/demos/lolcow
To build an image we need a recipe, or definition file, in the Singularity language. You’ll learn more on how to write one in a dedicated episode.
Here, let’s use one, lolcow.def, to build our first image. To this end we’re using sudo singularity build, followed by the filename (and path) we decide to attribute to the container image, and then by the filename of the def file to be used:
$ sudo singularity build lolcow.sif lolcow.def
This build should take approximately 5 minutes to complete:
INFO: Starting build...
[..]
INFO: Running post scriptlet
[..]
INFO: Adding help info
INFO: Adding labels
INFO: Adding environment to container
INFO: Adding runscript
INFO: Creating SIF file...
INFO: Build complete: lolcow.sif
At the end, you’ll find the final image file, lolcow.sif, in your directory:
$ ls
lolcow.def lolcow.sif
Sudo privileges with Singularity
As the image builds, let’s discuss this important matter.
Singularity does not allow for privileges escalation.
In other words, if you are a standard user and you run singularity, any command inside the container will be run with the privileges of the standard user, i.e. without admin powers. If you try and sudo from inside the container you will get an error.
On the other hand, if your user can run with sudo, and if you then decide to run Singularity as sudo singularity, then you will run any command from inside the container with admin powers.
This design contributes to making Singularity safe to run on HPC: users without admin rights are unable to escalate their privileges from inside the containers.
However, when building a container image you might need to install software using commands that require admin rights, e.g. apt get in Ubuntu/Debian or yum in Centos. To achieve this, you need to run sudo singularity build, as we have just done above.
This requirement implies that you must carry out your build in a machine where you DO have admin rights. Ruling out HPC systems, suitable platforms to build container images can be your laptop, an office workstation, or a virtual machine on the cloud.
Remote build
What if you need to build an image from a system where you don’t have admin privileges, i.e. you can’t run commands with sudo?
Singularity offers the option to run a build remotely, using the Sylabs Remote Builder; once again (see below) you will need a Sylabs account and an access token to use this feature. If this is the case, just use singularity build -r to proceed with the remote build. Once finished, the image will be downloaded so that it’s ready to use:
$ singularity build -r lolcow_remote.sif lolcow.def
INFO: Remote "default" added.
INFO: Authenticating with remote: default
INFO: API Key Verified!
INFO: Remote "default" now in use.
INFO: Starting build...
[..]
INFO: Running post scriptlet
[..]
INFO: Adding help info
INFO: Adding labels
INFO: Adding environment to container
INFO: Adding runscript
INFO: Creating SIF file...
INFO: Build complete: /tmp/image-699539270
WARNING: Skipping container verifying
67.07 MiB / 67.07 MiB 100.00% 14.18 MiB/s 4s
At the time of writing, when using the Remote Builder you won’t be able to use the %files header in the def file, to copy host files into the image.
Use the newly created container
Once the image has finished building, how would you run the command
fortunefrom inside this container?Solution
$ singularity exec lolcow.sif fortuneWhenever you find that you are on the side of the majority, it is time to reform. -- Mark TwainNow, try and run this pipe of commands:
bash -c 'fortune | cowsay | lolcat'.Solution
$ singularity exec lolcow.sif bash -c 'fortune | cowsay | lolcat'You will get something similar to this, hopefully just more colourful:
_______________________________________ / Have a place for everything and keep \ | the thing somewhere else; this is not | | advice, it is merely custom. | | | \ -- Mark Twain / --------------------------------------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||Great, we’ve just containerised a cow that cites novelists!
Share your container image
Now that you’ve built your container image, you might want to run it on other systems, or share it with collaborators.
The simplest way to achieve this is to remember that a SIF image is just a file, so… you can transfer it across systems using Linux command line utilities like scp or rsync, or even graphical applications such as Filezilla.
Just remember that images can be quite large, typically ranging from tens of MBs up to several GBs. The lolcow image we created is about 70 MB, but for instance a typical RStudio image is well above 1 GB.
If you want to keep the images publicly available, as they are just files you can store them in a server that is accessible through HTTP or FTP, or design a setup based upon open source image registry solutions such as Harbor.
Sylabs offer their own image hosting platform, the Sylabs Cloud Library, which is currently free upon signup. Let’s see how this works. If you don’t want to signup, just skip the hands-on and follow the demo.
Once you create an account, you’ll need to click on your account name on the top right of the page, select Access Tokens, then create a token, and copy it to the clipboard.
Then you can configure the machine you’re using for building container images, so that you can also push them to the Cloud Library:
$ singularity remote login
Generate an API Key at https://cloud.sylabs.io/auth/tokens, and paste here:
API Key:
Now paste the token you had copied to the clipboard end press Enter:
INFO: API Key Verified!
You are now ready to push your image to the Cloud Library, e.g. via singularity push:
$ export MY_SYLABS_USER="my_sylabs_user"
$ singularity push -U lolcow.sif library://$MY_SYLABS_USER/default/lolcow:30oct19
WARNING: Skipping container verifying
67.08 MiB / 67.08 MiB [==================================================================================================================================] 100.00% 6.37 MiB/s 10s
Note the use of the flag -U to allow pushing unsigned containers (for more information see this Singularity documentation page).
Also note once again the format for the registry: <user>/<project>/<name>:<tag>.
Finally, you (or other peers) are now able to pull your image from the Cloud Library:
$ singularity pull -U library://$MY_SYLABS_USER/default/lolcow:30oct19
INFO: Downloading library image
67.07 MiB / 67.07 MiB [===================================================================================================================================] 100.00% 8.10 MiB/s 8s
WARNING: Skipping container verification
INFO: Download complete: lolcow_30oct19.sif
Key Points
Build images using
sudo singularity buildUse the remote builder with the flag
-r, if you need to build images from a machine where you don’t have sudo rightsYou can share your Singularity Image File with others, as you would do with any other (big) file
Upload images to a web registry with
singularity push(Sylabs account required)