Data Transfer
2 minute read
Overview
There are several ways to transfer data to and from DAIC:
| Method | Best for | Notes |
|---|---|---|
rclone | Cloud storage, large transfers | Supports many backends |
rsync | Large directories, incremental sync | Efficient for updates |
scp | Individual files | Simple one-time transfers |
| SFTP | Direct transfer to staff-umbrella | Use webdata or SFTP client |
Rclone
Rclone supports transfers to/from many cloud providers and remote systems. Run rclone on your local machine to transfer data to DAIC.
Install rclone locally:
See rclone install guide for your operating system.
Configure DAIC as an SFTP remote (one-time setup):
rclone config
# Choose: n (new remote)
# Name: daic
# Type: sftp
# Host: daic01.hpc.tudelft.nl
# User: <NetID>
# Use SSH key authentication
Copy from local to DAIC:
rclone copy /local/data/ daic:/tudelft.net/staff-umbrella/<project>/
Sync local directory to DAIC:
rclone sync /local/data/ daic:/tudelft.net/staff-umbrella/<project>/data/
See rclone documentation for more options and cloud backends.
Git clone
Clone repositories directly on DAIC:
git clone git@gitlab.tudelft.nl:your-group/your-repo.git
SSH keys required
Set up SSH keys on DAIC and add the public key to your GitLab/GitHub account.Rsync
Rsync is efficient for transferring large directories and synchronizing changes.
From local to DAIC:
rsync -avz --progress /local/path/ <NetID>@daic01.hpc.tudelft.nl:/tudelft.net/staff-umbrella/<project>/
From DAIC to local:
rsync -avz --progress <NetID>@daic01.hpc.tudelft.nl:/tudelft.net/staff-umbrella/<project>/ /local/path/
Common options:
-aarchive mode (preserves permissions, timestamps)-vverbose output-zcompress during transfer--progressshow transfer progress--dry-runtest without transferring
SCP
For simple one-time file transfers:
Copy file to DAIC:
scp /local/file.tar.gz <NetID>@daic01.hpc.tudelft.nl:/tudelft.net/staff-umbrella/<project>/
Copy file from DAIC:
scp <NetID>@daic01.hpc.tudelft.nl:/tudelft.net/staff-umbrella/<project>/file.tar.gz /local/path/
Copy directory:
scp -r /local/directory/ <NetID>@daic01.hpc.tudelft.nl:/tudelft.net/staff-umbrella/<project>/
SFTP to staff-umbrella
You can transfer data directly to project storage without going through DAIC.
Using SFTP
Connect to sftp.tudelft.nl with your NetID credentials using the command line or clients like FileZilla, WinSCP, or Cyberduck.
Host: sftp.tudelft.nl
Username: <NetID>
Port: 22
Command line example:
sftp sftp.tudelft.nl
sftp> cd staff-umbrella/<project>
sftp> put localfile.txt
sftp> get remotefile.txt
sftp> put -r localfolder/
sftp> get -r remotefolder/
sftp> bye
Navigate to /staff-umbrella/<project>/ to access your project storage.