DAIC Experimental — Documentation for the experimental environment. (Go to stable docs)

Bash basics

Essential command-line skills for working on DAIC.

14 minute read

What you’ll learn

By the end of this tutorial, you’ll be able to:

Navigate the DAIC filesystem confidently
Create, copy, move, and delete files and directories
Redirect command output (stdout/stderr) to files
Chain commands together with pipes
Find files and search their contents
Write simple shell scripts to automate tasks

Time: About 45 minutes

Prerequisites: You should have logged into DAIC at least once.

The scenario

You’re a researcher who just got access to DAIC. You need to:

Set up a project directory
Organize your files
Find things when you forget where you put them
Automate repetitive tasks with scripts

Let’s learn the commands you need by actually doing these tasks.

Part 1: Finding your way around

When you log into DAIC, you arrive at your home directory. But where exactly are you, and what’s here?

Where am I?

The pwd command (print working directory) shows your current location:

$ pwd
/home/netid01

You’re in your home directory. On DAIC, this is a small space (5 MB) meant only for configuration files - not for your actual work.

What’s here?

The ls command lists what’s in the current directory:

$ ls
linuxhome

Not much! Let’s see more detail with ls -la:

$ ls -la
total 12
drwxr-xr-x   3 netid01 netid01 4096 Mar 20 09:00 .
drwxr-xr-x 100 root    root    4096 Mar 20 08:00 ..
-rw-r--r--   1 netid01 netid01  220 Mar 20 09:00 .bashrc
lrwxrwxrwx   1 netid01 netid01   45 Mar 20 09:00 linuxhome -> /tudelft.net/staff-homes-linux/n/netid01

Now we see hidden files (starting with .) and details about each file. The linuxhome entry has an arrow - it’s a symbolic link pointing to your larger personal storage.

Permission denied accessing linuxhome?

If you get “Permission denied” when accessing linuxhome, your Kerberos ticket has expired. Renew it with:

$ kinit
Password for netid01@TUDELFT.NET:

See Troubleshooting for more details.

Moving around

The cd command (change directory) moves you to a different location:

$ cd linuxhome
$ pwd
/home/netid01/linuxhome

Some useful shortcuts:

$ cd ..        # Go up one level
$ cd ~         # Go to home directory
$ cd -         # Go back to previous directory
$ cd           # Also goes to home directory

Exercise 1: Explore the filesystem

Try these commands and observe what happens:

$ cd /tudelft.net/staff-umbrella
$ ls
$ cd ~
$ pwd

Check your work

You should see project directories when listing /tudelft.net/staff-umbrella. After cd ~ and pwd, you should see your home directory path (e.g., /home/netid01).

Part 2: Understanding DAIC storage

Before we create files, let’s understand where to put them. DAIC has several storage locations:

Location	Purpose	Size
`/home/<netid>`	Config files only	5 MB
`~/linuxhome`	Personal files, code	~30 GB
`/tudelft.net/staff-umbrella/<project>`	Project data and datasets	Varies

Rule of thumb:

Code and small files → linuxhome or umbrella
Large datasets → umbrella
Never put large files in /home

Let’s navigate to where you’ll do most of your work:

$ cd /tudelft.net/staff-umbrella
$ ls

You should see one or more project directories. For this tutorial, let’s assume you have access to a project called myproject:

$ cd myproject
$ pwd
/tudelft.net/staff-umbrella/myproject

Part 3: Creating a project structure

Now let’s set up a workspace for a machine learning project.

Creating directories

The mkdir command creates directories:

$ mkdir ml-experiment
$ cd ml-experiment
$ pwd
/tudelft.net/staff-umbrella/myproject/ml-experiment

Create multiple directories at once with -p (which also creates parent directories if needed):

$ mkdir -p data/raw data/processed models results logs
$ ls
data  logs  models  results
$ ls data
processed  raw

We’ve created this structure:

ml-experiment/
├── data/
│   ├── raw/
│   └── processed/
├── models/
├── results/
└── logs/

Creating files

Create a simple file with echo and redirection:

$ echo "# ML Experiment" > README.md
$ cat README.md
# ML Experiment

The > operator writes output to a file, overwriting any existing content.

Standard output and standard error

Every command has two separate output streams:

Standard output (stdout) - where commands send their normal results
Standard error (stderr) - where commands send error messages and warnings

By default, both streams display in your terminal, so they appear mixed together. But they are separate channels that can be redirected independently.

See both streams in action:

$ ls /home /nonexistent
ls: cannot access '/nonexistent': No such file or directory   # stderr
/home:                                                         # stdout
netid01  netid02  netid03

The error message and the directory listing both appear in your terminal, but they came from different streams. This matters when you want to save output to files or filter out errors.

Redirecting output

The > operator redirects stdout to a file:

$ echo "Hello" > logs/output.txt       # Write to file (overwrites)
$ echo "World" >> logs/output.txt      # Append to file
$ cat logs/output.txt
Hello
World

Redirect stderr with 2>:

$ ls /home /nonexistent 2> logs/errors.txt
/home:
netid01  netid02  netid03
$ cat logs/errors.txt
ls: cannot access '/nonexistent': No such file or directory

The error went to the file; only the normal output appeared on screen.

Redirect stdout and stderr to separate files:

$ ls /home /nonexistent > logs/listing.txt 2> logs/errors.txt
$ cat logs/listing.txt
/home:
netid01  netid02  netid03
$ cat logs/errors.txt
ls: cannot access '/nonexistent': No such file or directory

Redirect both to the same file:

$ ls /home /nonexistent > logs/all_output.txt 2>&1
$ ls /home /nonexistent &> logs/all_output.txt    # Shorthand (bash only)

The 2>&1 syntax means “send stderr (2) to wherever stdout (1) is going.” The &> shorthand is a bash extension: it is not POSIX, so use > file 2>&1 in scripts that must run under sh.

To send output to another command rather than to a file, use a pipe (see Combining commands with pipes).

Discard output entirely:

$ mycommand > /dev/null 2>&1      # Discard everything
$ mycommand 2> /dev/null          # Discard only errors

Why this matters for HPC

By default, Slurm merges both streams into a single file named slurm-<jobid>.out. Adding #SBATCH --error=<file> to your job script sends stderr to a separate file instead. Understanding these streams helps you:

Find error messages when jobs fail
Keep logs readable by separating normal output from errors
Debug issues by examining what went to each stream

Exercise 2: Build your own structure

Create a directory structure for a different project:

$ cd /tudelft.net/staff-umbrella/myproject
$ mkdir -p nlp-project/{data,src,notebooks,outputs}
$ ls nlp-project

Then create a README:

$ echo "# NLP Project" > nlp-project/README.md
$ echo "Author: $(whoami)" >> nlp-project/README.md
$ cat nlp-project/README.md

Check your work

ls nlp-project should show:

data  notebooks  outputs  src

cat nlp-project/README.md should show:

# NLP Project
Author: <your-netid>

Part 4: Working with files

Let’s create some actual code to work with.

Creating a Python script

We’ll use cat with a “here document” to create a multi-line file:

$ cd /tudelft.net/staff-umbrella/myproject/ml-experiment
$ cat > train.py << 'EOF'
#!/usr/bin/env python3
"""Simple training script."""

import argparse

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--epochs', type=int, default=10)
    parser.add_argument('--lr', type=float, default=0.001)
    args = parser.parse_args()

    print(f"Training for {args.epochs} epochs with lr={args.lr}")
    for epoch in range(args.epochs):
        print(f"Epoch {epoch+1}/{args.epochs}")
    print("Done!")

if __name__ == '__main__':
    main()
EOF

Verify the file was created:

$ cat train.py
$ ls -l train.py
-rw-r--r-- 1 netid01 netid01 480 Mar 20 10:35 train.py

Copying files

The cp command copies files:

$ cp train.py train_backup.py
$ ls
data  logs  models  README.md  results  train_backup.py  train.py

Copy entire directories with -r (recursive):

$ cp -r data data_backup
$ ls
data  data_backup  logs  models  README.md  results  train_backup.py  train.py

Moving and renaming

The mv command moves files. It’s also how you rename:

$ mv train_backup.py old_train.py      # Rename
$ mv old_train.py models/              # Move to models directory
$ ls models
old_train.py

Deleting files

The rm command removes files:

$ rm models/old_train.py
$ ls models

Delete directories with -r:

$ rm -r data_backup
$ ls
data  logs  models  README.md  results  train.py

Warning

rm permanently deletes files - there’s no recycle bin! Use rm -i for confirmation prompts, especially when using wildcards.

Exercise 3: File operations

Practice by doing the following:

Copy train.py to evaluate.py
Create a src directory
Move both Python files into src
Verify with ls src

$ cp train.py evaluate.py
$ mkdir src
$ mv train.py evaluate.py src/
$ ls src
evaluate.py  train.py

Check your work

ls src should show both files:

evaluate.py  train.py

If train.py is missing, you may have forgotten to copy before moving.

Part 5: Viewing and editing files

Viewing file contents

Several commands let you view files:

$ cat src/train.py              # Print entire file
$ head -n 5 src/train.py        # First 5 lines
$ tail -n 5 src/train.py        # Last 5 lines
$ less src/train.py             # Scrollable viewer (q to quit)

For log files that are being written, tail -f shows new lines as they appear:

$ tail -f logs/training.log     # Watch live (Ctrl+C to stop)

Counting lines

$ wc -l src/train.py
18 src/train.py

Editing files

For quick edits, use nano (beginner-friendly):

$ nano src/train.py

Type to insert text
Ctrl+O to save
Ctrl+X to exit

For more power, use vim (see our Vim tutorial):

$ vim src/train.py

Part 6: Finding things

As your project grows, you’ll need to find files and search their contents.

Finding files by name

The find command searches for files:

$ find . -name "*.py"
./src/train.py
./src/evaluate.py

The . means “start from current directory”. Common options:

$ find . -name "*.py"                    # Files matching pattern
$ find . -type d -name "data*"           # Directories only
$ find . -type f -mtime -7               # Files modified in last 7 days
$ find . -size +100M                     # Files larger than 100MB

Searching inside files

The grep command searches file contents:

$ grep "epochs" src/train.py
    parser.add_argument('--epochs', type=int, default=10)
    print(f"Training for {args.epochs} epochs with lr={args.lr}")
    for epoch in range(args.epochs):
        print(f"Epoch {epoch+1}/{args.epochs}")

Search all Python files recursively:

$ grep -r "import" src/
src/train.py:import argparse
src/evaluate.py:import argparse

The -r option searched every file under src/, finding the same line in both copies.

Useful options:

$ grep -n "epochs" src/train.py    # Show line numbers
$ grep -i "EPOCH" src/train.py     # Case-insensitive
$ grep -l "import" src/*.py        # Just show filenames

Combining commands with pipes

Redirection sends a command’s output to a file. A pipe (|) sends it to another command instead, which reads it as input:

$ ls src | wc -l
2

ls src produced a list of filenames and, instead of printing them, the shell handed them to wc -l, which counted the lines. No temporary file was involved.

Pipes chain as far as you need, with each stage reading the previous stage’s stdout:

$ grep -r "epochs" src/ | wc -l
8
$ grep -r "epochs" src/ | grep -v "range" | wc -l
6
$ find . -name "*.py" | sort | head -n 2
./src/evaluate.py
./src/train.py

The -v option inverts a match, so the second command counts the lines that contain “epochs” but not “range”.

A pipe carries stdout only. Errors still go to your terminal:

$ grep -r "import" src/ nonexistent/ | wc -l
grep: nonexistent/: No such file or directory
2

The error message bypassed wc and appeared on screen; only the two matching lines were counted. To send stderr through the pipe as well, redirect it into stdout first:

$ grep -r "import" src/ nonexistent/ 2>&1 | wc -l
3

Now the error counts as a line like any other.

Why this matters for HPC

Cluster commands print more than you want to read. Pipes let you narrow the output on the spot - counting your queued jobs, or filtering a long node list down to the ones you care about - without writing anything to disk. See the Slurm tutorial for the commands worth filtering.

Exercise 4: Find and search

Find all files modified in the last day:
```
$ find . -mtime -1
```
Search for all occurrences of “print” in your Python files:
```
$ grep -n "print" src/*.py
```
Find all directories named “data”:
```
$ find . -type d -name "data"
```
Count how many Python files your project contains:
```
$ find . -name "*.py" | wc -l
```

Check your work

The find . -mtime -1 command should list files you recently created. The grep -n command shows line numbers where “print” appears. The directory search should show ./data (and any other data directories you created). The count should be 2: train.py and evaluate.py.

Part 7: Automating with scripts

When you find yourself typing the same commands repeatedly, it’s time to write a script.

Your first script

Create a script that sets up a new experiment:

$ cat > setup_experiment.sh << 'EOF'
#!/bin/bash
# Setup script for new experiments

# Check if experiment name was provided
if [ -z "$1" ]; then
    echo "Usage: ./setup_experiment.sh <experiment_name>"
    exit 1
fi

EXPERIMENT_NAME=$1
BASE_DIR="/tudelft.net/staff-umbrella/myproject"

echo "Creating experiment: $EXPERIMENT_NAME"

# Create directory structure
mkdir -p "$BASE_DIR/$EXPERIMENT_NAME"/{data,models,results,logs}

# Create a README
cat > "$BASE_DIR/$EXPERIMENT_NAME/README.md" << README
# $EXPERIMENT_NAME

Created: $(date)
Author: $(whoami)

## Description
TODO: Add description

## Results
TODO: Add results
README

echo "Done! Experiment created at $BASE_DIR/$EXPERIMENT_NAME"
ls -la "$BASE_DIR/$EXPERIMENT_NAME"
EOF

Make it executable

Before you can run a script, you need to make it executable:

$ chmod +x setup_experiment.sh
$ ls -l setup_experiment.sh
-rwxr-xr-x 1 netid01 netid01 694 Mar 20 11:00 setup_experiment.sh

The x in the permissions means “executable”.

Run the script

$ ./setup_experiment.sh bert-finetuning
Creating experiment: bert-finetuning
Done! Experiment created at /tudelft.net/staff-umbrella/myproject/bert-finetuning
total 24
drwxr-xr-x 6 netid01 netid01 4096 Mar 20 11:00 .
drwxr-xr-x 8 netid01 netid01 4096 Mar 20 11:00 ..
drwxr-xr-x 2 netid01 netid01 4096 Mar 20 11:00 data
drwxr-xr-x 2 netid01 netid01 4096 Mar 20 11:00 logs
drwxr-xr-x 2 netid01 netid01 4096 Mar 20 11:00 models
-rw-r--r-- 1 netid01 netid01  142 Mar 20 11:00 README.md
drwxr-xr-x 2 netid01 netid01 4096 Mar 20 11:00 results

Script building blocks

Here are patterns you’ll use often in scripts.

Variables:

NAME="experiment1"
echo "Working on $NAME"

Assign with = (no spaces around it). Reference with $NAME or ${NAME}.

Conditionals (if statements):

The if statement runs commands based on a condition:

if [ -f "data.csv" ]; then
    echo "Data file exists"
else
    echo "Data file not found!"
    exit 1
fi

The [ ... ] is a test command. Common test operators:

Test	Meaning
`-f file`	File exists and is a regular file
`-d dir`	Directory exists
`-e path`	Path exists (file or directory)
`-z "$var"`	Variable is empty
`-n "$var"`	Variable is not empty
`"$a" = "$b"`	Strings are equal
`"$a" != "$b"`	Strings are not equal
`"$n" -eq "$m"`	Numbers are equal
`"$n" -lt "$m"`	n is less than m
`"$n" -gt "$m"`	n is greater than m

Always quote variables inside [ ... ]. Without quotes, an empty or unset variable leaves the test with too few arguments and bash reports a syntax error.

Loops (for statements):

The for loop iterates over a list of items:

for file in data/*.csv; do
    echo "Processing $file"
    python process.py "$file"
done

The shell expands data/*.csv to all matching files, then runs the commands between do and done for each one, with $file set to the current filename.

Other loop patterns:

for i in 1 2 3 4 5; do          # Loop over explicit list
    echo "Run $i"
done

for i in {1..10}; do            # Loop over range
    echo "Epoch $i"
done

for arg in "$@"; do             # Loop over script arguments
    echo "Argument: $arg"
done

Command substitution:

TODAY=$(date +%Y-%m-%d)
echo "Running on $TODAY"

$(command) runs the command and substitutes its output.

Exercise 5: Write a cleanup script

Create a script that removes old log files:

$ cat > cleanup_logs.sh << 'EOF'
#!/bin/bash
# Remove log files older than 7 days

LOG_DIR="${1:-.}"  # Use first argument, or current directory

echo "Cleaning logs in $LOG_DIR"

# Find and remove old logs
find "$LOG_DIR" -name "*.log" -mtime +7 -exec rm -v {} \;

echo "Done!"
EOF

$ chmod +x cleanup_logs.sh
$ ./cleanup_logs.sh logs/

Check your work

Verify the script is executable:

$ ls -l cleanup_logs.sh
-rwxr-xr-x 1 netid01 netid01 ... cleanup_logs.sh

The x in the permissions confirms it’s executable. When run, it prints “Cleaning logs in logs/” and “Done!” (plus any files it removes).

Part 8: Useful shortcuts and tips

Tab completion

Press Tab to autocomplete:

Filenames
Directory names
Commands

$ cd /tudelft.net/staff-umb<TAB>
$ cd /tudelft.net/staff-umbrella/

Command history

$ history              # Show recent commands
$ !42                  # Run command number 42
$ !!                   # Run the last command
$ !grep                # Run the last command starting with "grep"

Press Ctrl+R to search history interactively.

Keyboard shortcuts

Shortcut	Action
`Ctrl+C`	Cancel current command
`Ctrl+D`	Exit shell / end input
`Ctrl+L`	Clear screen
`Ctrl+A`	Move to start of line
`Ctrl+E`	Move to end of line
`Ctrl+U`	Delete to start of line
`Ctrl+K`	Delete to end of line

Aliases

Create shortcuts for common commands. Add to ~/.bashrc:

alias ll='ls -lah'
alias umbrella='cd /tudelft.net/staff-umbrella/myproject'
alias myjobs='squeue -u $USER'

Then reload:

$ source ~/.bashrc
$ umbrella    # Now this works!

Summary

You’ve learned to:

Task	Command
See current location	`pwd`
List files	`ls -la`
Change directory	`cd path`
Create directory	`mkdir -p path`
Create/overwrite file	`echo "text" > file`
Append to file	`echo "text" >> file`
Redirect stderr	`command 2> errors.txt`
Redirect both	`command > out.txt 2>&1`
View file	`cat file` or `less file`
Copy	`cp source dest`
Move/rename	`mv source dest`
Delete	`rm file` or `rm -r dir`
Find files	`find . -name "*.py"`
Search contents	`grep "pattern" file`
Pipe into another command	`command \| wc -l`
Make script executable	`chmod +x script.sh`

What’s next?

Now that you’re comfortable with the command line:

Data Transfer - Move data to and from DAIC
Slurm Tutorial - Learn to submit jobs to the cluster
Vim Tutorial - Edit files more efficiently
Shell Setup - Configure your environment

Last modified August 27, 2026: teach pipes: the tutorial covered redirection but never chaining commands (55cfad1)