Tutorial: Running LLMs on DAIC

This guide shows how to run inference with LLMs on DAIC using Ollama. It uses the REIT LLM Serving Template, which submits the Slurm jobs needed to start an Ollama server on a GPU node and run a sample inference request.

The template starts two Slurm jobs for one inference workflow:

1. A server job that runs Ollama on a GPU node.
2. A client job that waits for the server, pulls a model, and sends a sample inference request.

1. Clone the Template Repository

Clone the template repository wherever you normally keep code. The cloned repository can live in your home directory, a Git workspace, or project storage.

git clone https://gitlab.ewi.tudelft.nl/reit/reit-llm-serving-template.git
cd reit-llm-serving-template

2. Run the Standard Batch Workflow

For most users, the easiest path is the launcher script:

bash start-serve-client.sh \
  --backend ollama \
  --project </path/to/your/project/in/umbrella/or/bulk/storage>

This starts the Ollama server first, then submits the client job with a Slurm dependency. The client job waits until Ollama is reachable before running the sample request.

The runtime directory passed with --project should be on umbrella or bulk storage. That directory stores generated containers, model files, and host/port files, so it can become too large for your home directory. It does not need to be the same directory as the cloned repository.

If you run the launcher from outside the cloned repository, pass --template so the Slurm jobs can find the template files:

bash /path/to/reit-llm-serving-template/start-serve-client.sh \
  --backend ollama \
  --project </path/to/your/project/in/umbrella/or/bulk/storage> \
  --template /path/to/reit-llm-serving-template

To use a different Ollama model, set MODEL_NAME before the launcher command:

MODEL_NAME=llama3.1:8b \
  bash start-serve-client.sh \
    --backend ollama \
    --project </path/to/your/project/in/umbrella/or/bulk/storage>

The default model in the template is qwen3.5:2b. You can use any model tag available in the Ollama library.

3. Check Job Progress

The launcher prints the submitted server and client job IDs. You can monitor them with:

squeue -j <server-job-id>,<client-job-id>

The Slurm output files are written in the directory where you submit the launcher command:

cat log-ollama-server-<server-job-id>.out
cat log-ollama-client-<client-job-id>.out

The client log should show that it waited for the server, pulled the configured model, ran test.py, and printed a model response.

4. What the Template Does

The launcher submits scripts from the Ollama backend directory:

start-serve-client.sh
  ├── backends/ollama/server.sbatch
  └── backends/ollama/client.sbatch

The server job:

• allocates a GPU node through Slurm
• builds or reuses an Ollama Apptainer image
• starts ollama serve on a random high port
• writes connection details under ${PROJECT_DIR}/ollama/
• cleans up host.txt and port.txt when Ollama exits

The client job:

• reads ${PROJECT_DIR}/ollama/host.txt
• reads ${PROJECT_DIR}/ollama/port.txt
• waits until the Ollama server is reachable
• pulls MODEL_NAME
• runs the sample OpenAI-compatible client in test.py

Using the launcher also keeps the model cache under ${PROJECT_DIR}/ollama/models/, prevents accidental serving from a login node, builds missing containers automatically, and cleans up the discovery files when the server exits.

5. Common Customizations

Most users only need to change a few settings.

For example, to test another model:

MODEL_NAME=llama3.1:8b \
  bash start-serve-client.sh \
    --backend ollama \
    --project </path/to/your/project/in/umbrella/or/bulk/storage>

6. What You Should Normally Not Edit

These values are usually set by the launcher or by the backend scripts:

Only edit backend helper functions if you understand the server/client workflow and need to change the template behavior.

7. Advanced: Interactive Debugging Workflow

For debugging, you can start an interactive GPU allocation and run the Ollama server manually.

srun --cpus-per-task=2 --mem=8G --time=00:30:00 --gres=gpu:1 --pty bash -il

Inside the allocation:

export PROJECT_DIR=</path/to/your/project/in/umbrella/or/bulk/storage>
export TEMPLATE_ROOT_DIR=/path/to/reit-llm-serving-template
source "${TEMPLATE_ROOT_DIR}/backends/ollama/ollama-function.sh"
ollama serve

Keep that terminal open. In a second terminal, connect to DAIC and run:

export PROJECT_DIR=</path/to/your/project/in/umbrella/or/bulk/storage>
export TEMPLATE_ROOT_DIR=/path/to/reit-llm-serving-template
source "${TEMPLATE_ROOT_DIR}/backends/ollama/ollama-function.sh"
ollama run qwen3.5:2b

You can check the Ollama endpoint directly with:

curl "http://$(cat "${PROJECT_DIR}/ollama/host.txt"):$(cat "${PROJECT_DIR}/ollama/port.txt")"

Stop the server with Ctrl-C in the first terminal. The helper removes host.txt and port.txt when the server exits.

8. Troubleshooting

9. When to Contact REIT

Contact REIT if:

• the template scripts fail after you have verified the Slurm job is running
• the Ollama server starts but the generated host/port files are wrong
• the client can reach the server but test.py consistently fails
• you need help adapting the workflow for a larger evaluation or a production-facing service

For ordinary model choice, prompt design, or GPU memory sizing, first try a smaller model and inspect the server/client logs.

Acknowledgment

This tutorial was inspired by the Stanford ollama_helper project. The DAIC template adapts similar ideas to TU Delft’s Slurm environment.