User Ramdisk

Ram Disk

On the GPC nodes, a `ram disk' is available. Up to half of the memory on the node may be used as a temporary file system. This is particularly useful for use in the early stages of migrating desktop-computing codes to a High Performance Computing platform such as the GPC, especially those that use a lot of I/O, such as Blast. Using a lot if I/O becomes a bottleneck in large scale computing. One especially suffers a performance penalty on parallel file systems (such as the GPFS used on SciNet), since the files are synchronized across the whole network.

Ramdisk is much faster than real disk, and is especially beneficial for codes which perform a lot of small I/O work, since the ramdisk does not require network traffic. However, each node sees its own ramdisk and cannot see files on that of other nodes.

To use the ramdisk, create and read to / write from files in /dev/shm/.. just as one would to (eg) /scratch/USER/. Only the amount of RAM needed to store the files will be taken up by the temporary file system. Thus if you have 8 serial jobs each requiring 1 GB of RAM, and 1GB is taken up by various OS services, you would still have approximately 7GB available to use as ramdisk on a 16GB node. However, if you were to write 8 GB of data to the RAM disk, this would exceed available memory and your job would likely crash.

Note that when using the ramdisk:

• At the start of your job, you can copy frequently accessed files to ramdisk. If there are many such files, it is beneficial to put them in a tar file.
• One would periodically copy the output files to files on /scratch or /project so that they are available after the job has completed.
• It is very important to delete your files from ram disk at the end of your job. If you do not do this, the next user to use that node will have less RAM available than they might expect, and this might kill their jobs.

A simple script using the ramdisk in a 1 day openMP job might look like this:

#!/bin/bash
#MOAB/Torque submission script for SciNet GPC (OpenMP)
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N ramdisk-test
trap 'rm -rf /dev/shm/*' TERM EXIT INT
cd $PBS_O_WORKDIR
cp jobfiles.tar /dev/shm/
cd /dev/shm
mkdir -p inputdir outputdir
tar xf jobfiles.tar
./job inputdir outputdir &
jobpid=$!
cd outputdir
while [ `ps -u $USER | grep -e "^$jobpid" | wc -l` ]; do
  sleep 3600
  rm -rf outputfiles.tar
  tar cf outputfiles.tar *
  cp outputfiles.tar $PBS_O_WORKDIR
done
rm -rf /dev/shm/*

Notes:

• The script assumes that the tar file jobfiles.tar contains the executable job and the input files in a subdirectory called inputdir.
• The exectable is supposed to take the locations of the input and output directory as arguments.
• The trap comment makes sure that the ramdisk gets flushed even when the jobs gets killed before the end of the script is reached. trap is bash script construction that executes the given command when the job is given, in this case, a TERM, EXIT or INT signal. The TERM signal is given by the scheduler 30 seconds before you time is up.