Apptainer tutorial

What you’ll learn

• Understand why containers are useful for HPC workloads
• Pull prebuilt images from Docker Hub and NVIDIA NGC
• Build custom container images from definition files
• Run containerized applications on DAIC with GPU support
• Manage bind mounts and cache directories

Prerequisites: Slurm Basics (submitting jobs, requesting GPUs)

Time: 45 minutes

What and Why containerization?

Containerization packages your software, libraries, and dependencies into a single portable unit: a container. This makes your application behave the same way everywhere: on your laptop, in the cloud, or on DAIC. This means:

• Consistency: The application runs the same way regardless of where it’s executed. You can develop on one machine, test on another, and deploy on a cluster without worrying about dependency differences.
• Isolation: Each container is independent from others, preventing conflicts and enhancing security and reliability.
• Portability: Containers can run on different systems without modification, simplifying movement between servers, clusters, or clouds.
• Efficiency: Containers share the host system’s resources like the operating system, making them lightweight and fast to start compared to virtual machines.

On DAIC specifically, users often encounter issues with limited home directory space or Windows-based /tudelft.net mounts (see Storage), which can complicate the use of conda/mamba and/or pip. Containers offer a solution by encapsulating all software and dependencies in a self-contained environment. You can, for instance, store containers on staff-umbrella with all required dependencies, including those installed via pip, and run them reliably and reproducibly without being limited by home directory size or mount compatibility.

Containerization on DAIC: Apptainer

DAIC supports Apptainer (formerly known as Singularity), an open-source container platform designed for high-performance computing environments. Apptainer runs container images securely on shared clusters and allows you to use Docker images directly, without needing Docker itself.

A typical Apptainer workflow revolves around three key components:

Because Apptainer integrates well with Slurm, containers can be launched directly within batch jobs or interactive sessions on DAIC.
The following sections show how to obtain, build, and run images.

Workflow overview

The typical lifecycle for containers on DAIC is:

1. Build the image locally from a .def file.
2. Transfer or pull the resulting .sif file onto DAIC.
3. Test interactively using salloc to get a compute node.
4. Run in a batch job with sbatch or srun using apptainer exec or apptainer run.
5. Provision bind mounts, GPU flags, and cache locations as needed.
6. Clean up and manage storage (e.g., APPTAINER_CACHEDIR).

How to run commands/programs inside a container?

Once you have a container image (e.g., myimage.sif), you can launch it in different ways depending on how you want to interact with it:

where:

• <image> is the path to a container image, typically, a *.sif file.

Tips:

• Use shell for exploration or debugging inside the container.
• Use exec or run for automation, workflows, or Slurm batch jobs.
• Add -C or -c to isolate the container filesystem (see Exposing host directories).

$ hostname
daic01.hpc.tudelft.nl

$ salloc --account=<your-account> --partition=all --gres=gpu:1 --mem=8G --time=01:00:00
salloc: Granted job allocation 12345

$ srun apptainer exec --nv myimage.sif python script.py

How to get container files?

You can obtain container images in two main ways:

1. Pull prebuilt images by pulling from a container registry/repository (see Using prebuilt images).
2. Build your own image locally using a definition file (*.def), then transfer the resulting .sif file to DAIC (see Building images).

1. Using prebuilt images

Apptainer allows pulling and using images directly from repositories like DockerHub, BioContainers, NVIDIA GPU Cloud (NGC), and others.

Example: Pulling from DockerHub

$ mkdir ~/containers && cd ~/containers

$ apptainer pull docker://ubuntu:latest
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 837dd4791cdc done
Copying config 1f6ddc1b25 done
Writing manifest to image destination
...
INFO:    Creating SIF file...

Now, to check the obtained image file:

$ ls
ubuntu_latest.sif

$ apptainer exec ubuntu_latest.sif cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
...

$ ls /.apptainer.d/
ls: cannot access /.apptainer.d/: No such file or directory

$ apptainer shell ubuntu_latest.sif
Apptainer> hostname
daic01.hpc.tudelft.nl
Apptainer> ls /.apptainer.d/
Apptainer  actions  env  labels.json  libs  runscript  startscript
Apptainer> exit

Notes:

• Inside the container, the command prompt changes to Apptainer>
• The container inherits your environment (e.g., $HOME, hostname) but has its own internal filesystem (e.g. /.apptainer.d)

$ apptainer shell -C ubuntu_latest.sif

Example: Pulling from NVIDIA GPU cloud (NGC)

NGC provides pre-built images for GPU accelerated applications. These images are large, so download them locally on your machine and then transfer to DAIC. To install Apptainer locally, follow the official Installing Apptainer instructions.

export APPTAINER_CACHEDIR=/tudelft.net/staff-umbrella/<project>/apptainer/cache

On your local machine:

$ apptainer pull docker://nvcr.io/nvidia/pytorch:24.01-py3
$ scp pytorch_24.01-py3.sif daic01.hpc.tudelft.nl:/tudelft.net/staff-umbrella/<project>/apptainer

Test the image on DAIC:

$ cd /tudelft.net/staff-umbrella/<project>/apptainer

$ salloc --account=<your-account> --partition=all --gres=gpu:1 --time=00:10:00
salloc: Granted job allocation 12345

$ srun apptainer shell -C --nv pytorch_24.01-py3.sif
Apptainer> python -c "import torch; print(torch.cuda.is_available())"
True

2. Building images

If you prefer (or need) a custom container image, you can build one from a definition file (*.def), that specifies your dependencies and setup steps.

On DAIC, you can build images directly if your current directory allows writes and sufficient quota (e.g., under staff-umbrella).
For large or complex builds, it can be more convenient to build locally on your workstation and then transfer the resulting .sif file to DAIC.

$ apptainer build myimage.sif myimage.def

Example: CUDA-enabled container

An example definion file, cuda_based.def, for a cuda-enabled container may look as follows:

# Header
Bootstrap: docker
From: nvidia/cuda:12.1.1-devel-ubuntu22.04

# (Optional) Sections/ data blobs
%post
    apt-get update # update system
    apt-get install -y git   # install git
    git clone https://github.com/NVIDIA/cuda-samples.git  # clone target repository
    cd cuda-samples
    git fetch origin --tags && git checkout v12.1 # fetch certain repository version
    cd Samples/1_Utilities/deviceQuery && make # install certain tool

%runscript
    /cuda-samples/Samples/1_Utilities/deviceQuery/deviceQuery  

where:

• The header, specifies the source (eg, Bootstrap: docker) and the base image (From: nvidia/cuda:12.1.1-devel-ubuntu22.04). Here, the container builds on Ubuntu 22.04 with CUDA 12.1 pre-installed.
• The rest of the file are optional data blobs or sections. In this example, the following blobs are used:
  • %post: the steps to download, configure and install needed custom software and libraries on the base image. In this example, the steps install git, clone a repo, and install a package via make
  • %runscript: the entry point to the container with the apptainer run command. In this example, the deviceQuery is executed once the container is run.
  • Other blobs may be present in the def file. See Definition files documentation for more details and examples.

Build this image locally and transfer it to DAIC:

$ apptainer build cuda_based_image.sif cuda_based.def
INFO:    Starting build...
Getting image source signatures
...
INFO:    Adding runscript
INFO:    Creating SIF file...
INFO:    Build complete: cuda_based_image.sif

$ scp cuda_based_image.sif daic01.hpc.tudelft.nl:/tudelft.net/staff-umbrella/<project>/apptainer

On DAIC, test the image:

$ cd /tudelft.net/staff-umbrella/<project>/apptainer

$ salloc --account=<your-account> --partition=all --cpus-per-task=2 --mem=1G --gres=gpu:1 --time=00:10:00
salloc: Granted job allocation 12345

$ srun apptainer run --nv -C cuda_based_image.sif
/cuda-samples/Samples/1_Utilities/deviceQuery/deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "NVIDIA L40"
  CUDA Driver Version / Runtime Version          12.9 / 12.1
  CUDA Capability Major/Minor version number:    8.9
  Total amount of global memory:                 46068 MBytes
  ...
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.9, CUDA Runtime Version = 12.1, NumDevs = 1
Result = PASS

$ apptainer shell --nv -C cuda_based_image.sif

Example: Extending existing images

During software development, it is common to incrementally build code and go through many iterations of debugging and testing. To save time, you can base a new image on an existing one using the Bootstrap: localimage and From:<path/to/local/image> header. This avoids re-installing the same dependencies with every iteration.

As an example, assume it is desirable to develop some code on the basis of the cuda_based.sif image created in the Example: CUDA-enabled container. Building from the original cuda_based.def file can take ~ 4 minutes. However, if the *.sif file is already available, building on top of it, via a dev_on_cuda_based.def file as below, takes ~ 2 minutes. This is already a time saving factor of 2.

# Header
Bootstrap: localimage
From: cuda_based.sif

# (Optional) Sections/ data blobs
%runscript
    echo "Arguments received: $*"
    exec echo "$@"

Now, build and test:

$ apptainer build dev_image.sif dev_on_cuda_based.def
INFO:    Starting build...
INFO:    Verifying bootstrap image cuda_based.sif
INFO:    Adding runscript
INFO:    Creating SIF file...
INFO:    Build complete: dev_image.sif

$ apptainer run dev_image.sif "hello world"
Arguments received: hello world
hello world

$ apptainer shell dev_image.sif
Apptainer> ls /cuda-samples/Samples/1_Utilities/deviceQuery/deviceQuery
/cuda-samples/Samples/1_Utilities/deviceQuery/deviceQuery

Apptainer> cat /.apptainer.d/bootstrap_history/Apptainer0
bootstrap: docker
from: nvidia/cuda:12.1.1-devel-ubuntu22.04
...

As can be seen in this example, the new def file not only preserves the dependencies of the original image, but it also preserves a complete history of all build processes while giving flexible environment that can be customized as need arises.

Example: Deploying conda and pip in a container

There might be situations where you have a certain conda environment in your local machine that you need to set up in DAIC to commence your analysis. In such cases, deploying your conda environment in a container and sending this container to DAIC does the job for you.

As an example, let’s create a simple demo environment, environment.yml in our local machine,

name: apptainer
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.9
  - matplotlib
  - pip
  - pip:
    - -r requirements.txt

And everything that should be installed with pip in requirement.txt file:

--extra-index-url https://download.pytorch.org/whl/cu123
torch
annoy

Now, it is time to create the container definition file Apptainer.def. One option is to base the image on condaforge/miniforge, which is a minimal Ubuntu installation with conda preinstalled at /opt/conda:

Bootstrap: docker
From: condaforge/miniforge3:latest

%files
    environment.yml /environment.yml
    requirements.txt /requirements.txt

%post
    # Update and install necessary packages
    apt-get update && apt-get install -y tree time vim ncdu speedtest-cli build-essential

    # Create a new Conda environment using the environment files.
    mamba env create --quiet --file /environment.yml
    
    # Clean up
    apt-get clean && rm -rf /var/lib/apt/lists/*
    mamba clean --all -y

    # Now add the script to activate the Conda environment
    echo '. "/opt/conda/etc/profile.d/conda.sh"' >> $APPTAINER_ENVIRONMENT
    echo 'conda activate apptainer' >> $APPTAINER_ENVIRONMENT

Now, build and check the image:

$ apptainer build demo-env-image.sif Apptainer.def
INFO:    Starting build...
Getting image source signatures
...
INFO:    Creating SIF file...
INFO:    Build complete: demo-env-image.sif

Verify the container setup:

$ apptainer exec demo-env-image.sif which python
/opt/conda/envs/apptainer/bin/python

Perfect! This confirms that our container image built successfully and the Conda environment is automatically activated. The Python executable is correctly pointing to our custom environment path, indicating that all our dependencies should be available.

We are going to use the environment inside a container together with a Python script that we store outside the container. Create the file analysis.py, which generate a plot:

#!/usr/bin/env python3

import matplotlib.pyplot as plt
import numpy as np

x = np.linspace(0, 2 * np.pi, 100)
y = np.sin(x)

plt.plot(x, y)
plt.title('Sine Wave')
plt.savefig('sine_wave.png')

Now, run the analysis:

$ apptainer exec demo-env-image.sif python analysis.py
$ ls
sine_wave.png

Exposing host directories

Depending on use case, it may be necessary for the container to read or write data from or to the host system. For example, to expose only files in a host directory called ProjectDataDir to the container image’s /mnt directory, add the --bind directive with appropriate <hostDir>:<containerDir> mapping to the commands you use to launch the container, in conjunction with the -C flag eg, shell or exec as below:

$ ls ProjectDataDir
raw_data.txt

$ apptainer shell -C --bind ProjectDataDir:/mnt ubuntu_latest.sif
Apptainer> ls /mnt
raw_data.txt
Apptainer> echo "Date: $(date)" >> /mnt/raw_data.txt
Apptainer> exit

$ tail -n1 ProjectDataDir/raw_data.txt
Date: Fri Mar 20 10:30:00 CET 2026

If the desire is to expose this directory as read-only inside the container, the --mount directive should be used instead of --bind, with rodesignation as follows:

$ apptainer shell -C --mount type=bind,source=ProjectDataDir,destination=/mnt,ro ubuntu_latest.sif
Apptainer> ls /mnt
raw_data.txt
Apptainer> echo "Date: $(date)" >> /mnt/raw_data.txt
bash: /mnt/raw_data.txt: Read-only file system

Advanced: containers and (fake) native installation

It’s possible to use Apptainer to install and then use software as if it were installed natively in the host system. For example, if you are a bioinformatician, you may be using software like samtools or bcftools for many of your analyses, and it may be advantageous to call it directly. Let’s take this as an illustrative example:

1. Create a directory structure: an exec directory for container images and a bin directory for symlinks:

$ mkdir -p software/bin/ software/exec

2. Create a definition file and build the image:

$ cd software/exec

$ cat bio-recipe.def
Bootstrap: docker
From: ubuntu:latest
%post
    apt-get update
    apt-get install -y samtools bcftools
    apt-get clean

$ apptainer build bio-container.sif bio-recipe.def

3. Create a wrapper script:

$ cat wrapper_bio-container.sh
#!/bin/bash
containerdir="$(dirname $(readlink -f ${BASH_SOURCE[0]}))"
cmd="$(basename $0)"
apptainer exec "${containerdir}/bio-container.sif" "$cmd" "$@"

$ chmod +x wrapper_bio-container.sh

4. Create symlinks:

$ cd ../bin
$ ln -s ../exec/wrapper_bio-container.sh samtools
$ ln -s ../exec/wrapper_bio-container.sh bcftools

5. Add the directory to your $PATH and use the tools:

$ export PATH=$PATH:$PWD

$ bcftools -v
bcftools 1.13
Using htslib 1.13+ds
...

$ samtools version
samtools 1.13
Using htslib 1.13+ds
...

software/
├── bin
│   ├── bcftools -> ../exec/wrapper_bio-container.sh
│   └── samtools -> ../exec/wrapper_bio-container.sh
└── exec
    ├── bio-container.sif
    └── wrapper_bio-container.sh

echo 'export PATH=$PATH:/path/to/software/bin' >> ~/.bashrc

Exercises

Practice what you’ve learned with these hands-on exercises.

Exercise 1: Pull and explore an image

Pull the python:3.11-slim image from Docker Hub and explore it:

1. Use apptainer pull to download the image
2. Use apptainer shell to open an interactive session
3. Check the Python version inside the container
4. List the contents of /usr/local/lib/python3.11/
5. Exit the container

Apptainer> python --version
Python 3.11.x
Apptainer> ls /usr/local/lib/python3.11/
...  site-packages  ...

Exercise 2: Run a command in a container

Using the Python image from Exercise 1:

1. Create a simple Python script hello.py that prints “Hello from Apptainer!”
2. Use apptainer exec to run the script inside the container
3. Try running it with the -C flag - what happens to your script?

$ apptainer exec python_3.11-slim.sif python hello.py
Hello from Apptainer!

$ apptainer exec -C python_3.11-slim.sif python hello.py
python: can't open file 'hello.py': [Errno 2] No such file or directory

Exercise 3: Build a custom image

Create a definition file for a container with your favorite tools:

1. Start from ubuntu:22.04
2. Install at least two packages (e.g., curl and jq)
3. Add a %runscript that displays a welcome message
4. Build the image and test it with apptainer run

$ apptainer run mytools.sif
Welcome to my custom container!
$ apptainer exec mytools.sif which curl jq
/usr/bin/curl
/usr/bin/jq

Exercise 4: GPU container on DAIC

Test GPU access with a prebuilt image:

1. Request an interactive GPU session with salloc
2. Pull or use an existing PyTorch NGC image
3. Run a Python command that checks torch.cuda.is_available()
4. Verify the GPU is detected with nvidia-smi inside the container

$ srun apptainer exec --nv pytorch.sif python -c "import torch; print(torch.cuda.is_available())"
True
$ srun apptainer exec --nv pytorch.sif nvidia-smi
... (GPU info displayed) ...

Exercise 5: Bind mounts

Practice data isolation:

1. Create a directory with a test file
2. Run a container with -C (isolated) and --bind to mount only that directory
3. Inside the container, verify you can access the test file but not your home directory
4. Try mounting the directory as read-only with --mount

$ mkdir testdir && echo "test" > testdir/data.txt
$ apptainer shell -C --bind testdir:/mnt ubuntu_latest.sif
Apptainer> cat /mnt/data.txt
test
Apptainer> ls /home/$USER
ls: cannot access '/home/...': No such file or directory

$ apptainer shell -C --mount type=bind,source=testdir,destination=/mnt,ro ubuntu_latest.sif
Apptainer> echo "new" >> /mnt/data.txt
bash: /mnt/data.txt: Read-only file system

Troubleshooting

Build fails with “no space left on device”

Apptainer uses your home directory for temporary files during builds. Since /home on DAIC is limited to 5 MB, builds often fail.

Solution: Set a different cache directory before building:

$ export APPTAINER_CACHEDIR=/tudelft.net/staff-umbrella/<project>/apptainer/cache
$ export APPTAINER_TMPDIR=/tudelft.net/staff-umbrella/<project>/apptainer/tmp
$ mkdir -p $APPTAINER_CACHEDIR $APPTAINER_TMPDIR

Add these to your ~/.bashrc to make them permanent.

GPU not visible inside container

Your container runs but torch.cuda.is_available() returns False or nvidia-smi fails.

Possible causes and solutions:

1. Missing --nv flag: Always pass --nv to enable GPU access:

   $ apptainer exec --nv myimage.sif python -c "import torch; print(torch.cuda.is_available())"
   

2. Not running on a GPU node: Check that you requested a GPU and are using srun:

   $ salloc --gres=gpu:1 ...
   $ srun apptainer exec --nv myimage.sif nvidia-smi
   

3. CUDA version mismatch: The container’s CUDA version must be compatible with the host driver. Check host driver version:

   $ nvidia-smi | grep "Driver Version"
   

Cache filling up disk space

Apptainer caches pulled images and build layers. This can consume significant space over time.

Solution: Periodically clean the cache:

$ apptainer cache clean

To see cache usage:

$ apptainer cache list

Container can’t access my files

By default, Apptainer mounts your home directory and current working directory. With -C (contain), the container is isolated.

Solution: Explicitly bind the directories you need:

$ apptainer exec -C --bind /tudelft.net/staff-umbrella/myproject:/data myimage.sif ls /data

Summary

You learned how to:

• Pull images from Docker Hub and NVIDIA NGC
• Build images from definition files with %post and %runscript sections
• Run containers with shell, exec, and run commands
• Enable GPU access with the --nv flag
• Isolate filesystems with -C and selectively expose directories with --bind
• Manage cache by setting APPTAINER_CACHEDIR

Key commands

What’s next?

• Learn Vim for editing files directly on the cluster
• See Container GPU Jobs for batch job examples
• Explore Apptainer documentation for advanced features