Data Transfer
What can slow down my data transfer?
To move data quickly, all of the stages in the process have to be fast; the file system you are reading data from, the CPU reading the data, the network connection between the sender and the reciever, and the recipient CPU and disk. The slowest element in that chain will slow down the entire transfer.
On SciNet's side, our underlying filesystem is the high-performance GPFS system, and the node we recommend you use (below) has a high-speed connection to the network and fast CPUs.
Data Mover Node
Serious moves of data to or from SciNet should be done from the datamover1 node. From any of the interactive SciNet nodes, one should be able to ssh datamover1 to log in. This is the machine that has the fastest network connection to the outside world (by a factor of 10; a 10Gb/s link as vs 1Gb/s).
Transfers must be originated from datamover1; that is, one can not copy files from the outside world directly to or from the data mover node; one has to log in to the data mover node and copy the data to or from the outside network.
hpn-ssh
All traffic to and from the data centre has to go via SSH, or secure shell. This is a protocol which sets up a secure connection between between two sites. On top of this protocol, there are many ways to copy files.
The usual ssh protocols were not designed for speed. On the datamover1 node, we have installed hpn-ssh, or High-Performance-enabled ssh. This is backwards compatable with the `usual' ssh, but is capable of significantly higher speeds. If you routinely have large data transfers to do, we recommend having your system administrator look into installing hpn-ssh on your system.
Everything we discuss below, unless otherwise stated, will work regardless of whether you have hpn-ssh installed on your remote system.
scp
scp, or secure copy, is the easiest way to copy files, although we generally find rsync below to be faster.
scp works like cp to copy files:
$ scp original_file copy_file
except that either the original or the copy can be on another system:
$ scp jonsdatafile.bin jon@remote.system.com:/home/jon/bigdatadir/
will copy the data file into the directory /home/jon/bigdatadir/ on remote.system.com after logging in as jon; you will be prompted for a password (unless you've set up ssh keys).
Copying from remote systems works the same way:
$ scp jon@remote.system.com:/home/jon/bigdatadir/newdata.bin .
And wildcards work as you'd expect (except you have to quote the wildcards on the remote system, as it can't expand properly here.)
$ scp *.bin jon@remote.system.com:/home/jon/bigdatadir/ $ scp jon@remote.system.com:"/home/jon/inputdata/*" .
There are few options worth knowing about:
- scp -C compresses the file before transmitting it; if the file compresses well, this can significantly increase the effective data transfer rate (though usually not as much as compressing the data, then sending it, then uncompressing). If the file doesn't compress well, than this adds CPU overhead without accomplishing much, and can slow down your data transfer.
- scp -oNoneEnabled=yes -oNoneSwitch=yes -- if CPU overhead is a significant bottleneck in the
rsync
rsync -av -e ssh 45-tangle2/*0025.bin ljdursi@nuexport00.cita.utoronto.ca:/mnt/raid-cita/ljdursi/athDrape/45-tangle2
If the files compress well, compressing first, or transmitting with compression on (scp -C, rsync -z) can significantly enhance effective data transfer rates
What transfer speeds should I expect?
| Mode | With hpn-ssh | Without |
|---|---|---|
| rsync | 60-80 MB/s | 30-40 MB/s |
| scp | 50 MB/s | 25 MB/s |
Why are my transfer so much slower?
Numbers << less than that: could be a number of things.
- network connection between scinet and your machine - how busy is the server - how busy is the disk