oldwiki.scinet.utoronto.ca - User contributions [en-gb]

Compiling Gromacs

2009-12-05T20:25:42Z

Dacaplan: /* fftw single precision compilation */

==Available Compilations==

Chris Neale has compiled gromacs on GPC, with assistance from Scott Northrup, and
on the power6 cluster with assistance from Ching-Hsing Yu. Users are welcome to utilize
these binary executables, but only at their own peril since compiling and testing your own
executable is safer and more stable.

''Gromacs executables:''

'''GPC:''' /scratch/cneale/exe/intel/gromacs-4.0.5/exec/bin

'''TCS:''' /scratch/cneale/exe/gromacs-4.0.4_aix/exec/bin

==Compiling your own GROMACS executables==

Below you will find scripts for the different compilations that you can follow to make your own binaries.

'''NOTE:''' ''the steps are not listed in order! You must compile fftw before compiling gromacs, and if you are going to use mvapich2-1.4rc1 then you must compile it also before compiling parallel gromacs.''

# Compiling serial single precision gromacs on GPC
# Compiling openmpi parallel gromacs on GPC
# Compiling serial gromacs on the power6 (submitted to the queue):
# Compiling parallel gromacs on the power6 (submitted to the queue):
# fftw single precision compilation
# Change to get mvapich2-1.4rc1 to compile gromacs
# Compiling mvapich2-1.4rc1
# Compiling gromacs on GPC using mvapich2-1.4rc1

-- [[User:Cneale|cneale]] 18 August 2009

=====Compiling serial single precision gromacs on GPC=====

<source lang="sh">
cd /scratch/cneale/exe/intel/gromacs-4.0.5
mkdir exec
module purge
module load intel
export FFTW_LOCATION=/scratch/cneale/GPC/exe/intel/fftw-3.1.2/exec
export GROMACS_LOCATION=/scratch/cneale/GPC/exe/intel/gromacs-4.0.5/exec
export CPPFLAGS=-I$FFTW_LOCATION/include
export LDFLAGS=-L$FFTW_LOCATION/lib
./configure --prefix=$GROMACS_LOCATION --without-motif-includes
--without-motif-libraries --without-x --without-xml >output.configure
2>&1
make >output.make 2>&1
make install >output.make_install 2>&1
make distclean
</source>

-- [[User:Cneale|cneale]] 18 August 2009

=====Compiling openmpi parallel gromacs on GPC:=====

<source lang="sh">
cd /scratch/cneale/exe/intel/gromacs-4.0.5
mkdir exec
module purge
module load openmpi intel
export FFTW_LOCATION=/scratch/cneale/exe/intel/fftw-3.1.2/exec
export GROMACS_LOCATION=/scratch/cneale/exe/intel/gromacs-4.0.5/exec
export CPPFLAGS="-I$FFTW_LOCATION/include
-I/scinet/gpc/mpi/openmpi/1.3.2-intel-v11.0-ofed/include
-I/scinet/gpc/mpi/openmpi/1.3.2-intel-v11.0-ofed/lib"
export LDFLAGS=-L$FFTW_LOCATION/lib
/gpc/mpi/openmpi/1.3.2-intel-v11.0-ofed/lib/openmpi
-I/scinet/gpc/x1/intel/Compiler/11.0/081/lib/intel64
-I/scinet/gpc/x1/intel/Compiler/11.0/081/mkl/lib/em64t/"
./configure --prefix=$GROMACS_LOCATION --without-motif-includes
--without-motif-libraries --without-x --without-xml --enable-mpi
--program-suffix="_openmpi" >output.configure.mpi 2>&1
make >output.make.mpi 2>&1
make install-mdrun >output.make_install.mpi 2>&1
make distclean
</source>

-- [[User:Cneale|cneale]] 18 August 2009
=====Compiling serial gromacs on the power6 (submitted to the queue)=====

Note that the -O5 flag for the power6 compilation makes it take about
20 hours to compile. You can drop that if you want, but it does give
you a few more percent performance.

<source lang="sh">
#======================================================================
# Specifies the name of the shell to use for the job
# @ shell = /usr/bin/ksh
# @ job_type = serial
# @ class = verylong
## # @ node = 1
## # @ tasks_per_node = 1
# @ output = $(jobid).out
# @ error = $(jobid).err
# @ wall_clock_limit = 40:00:00
#=====================================
## this is necessary in order to avoid core dumps for batch files
## which can cause the system to be overloaded
# ulimits
# @ core_limit = 0
#=====================================
## necessary to force use of infiniband network for MPI traffic
### TURN IT OFF # @ network.MPI = csss,not_shared,US,HIGH
#=====================================
# @ environment=COPY_ALL
# @ queue
export
PATH=/usr/lpp/ppe.hpct/bin:/usr/vacpp/bin:.:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java14/jre/bin:/usr/java14/bin:/usr/lpp/LoadL/full/bin:/usr/local/bin
export F77=xlf_r
export CC=xlc_r
export CXX=xlc++_r
export FFLAGS="-O5 -qarch=pwr6 -qtune=pwr6"
export CFLAGS="-O5 -qarch=pwr6 -qtune=pwr6"
export CXXFLAGS="-O5 -qarch=pwr6 -qtune=pwr6"
export FFTW_LOCATION=/scratch/cneale/exe/fftw-3.1.2_aix/exec
export GROMACS_LOCATION=/scratch/cneale/exe/gromacs-4.0.4_aix/exec
export CPPFLAGS=-I$FFTW_LOCATION/include
export LDFLAGS=-L$FFTW_LOCATION/lib
cd /scratch/cneale/exe/gromacs-4.0.4_aix
mkdir exec
./configure --prefix=$GROMACS_LOCATION --without-motif-includes
--without-motif-libraries --without-x --without-xml >output.configure
2>&1
make >output.make 2>&1
make install >output.make_install 2>&1
make distclean
</source>

-- [[User:Cneale|cneale]] 18 August 2009
=====Compiling parallel gromacs on the power6 (submitted to the queue)=====

<source lang="sh">
#===============================================================================
# Specifies the name of the shell to use for the job
# @ shell = /usr/bin/ksh
##### @ job_type = serial
# @ job_type = parallel
# @ class = verylong
# @ node = 1
# @ tasks_per_node = 1
# @ output = $(jobid).out
# @ error = $(jobid).err
# @ wall_clock_limit = 40:00:00
#=====================================
## this is necessary in order to avoid core dumps for batch files
## which can cause the system to be overloaded
# ulimits
# @ core_limit = 0
#=====================================
## necessary to force use of infiniband network for MPI traffic
### TURN IT OFF # @ network.MPI = csss,not_shared,US,HIGH
#=====================================
# @ environment=COPY_ALL
# @ queue

export F77=xlf_r
export CC=xlc_r
export CXX=xlc++_r
export FFLAGS="-O5 -qarch=pwr6 -qtune=pwr6"
export CFLAGS="-O5 -qarch=pwr6 -qtune=pwr6"
export CXXFLAGS="-O5 -qarch=pwr6 -qtune=pwr6"
export FFTW_LOCATION=/scratch/cneale/exe/fftw-3.1.2_aix/exec
export GROMACS_LOCATION=/scratch/cneale/exe/gromacs-4.0.4_aix/exec
export CPPFLAGS=-I$FFTW_LOCATION/include
export LDFLAGS=-L$FFTW_LOCATION/lib
cd /scratch/cneale/exe/gromacs-4.0.4_aix
echo "cn-r0-10" > ~/.rhosts
echo localhost > ~/host.list
for((i=2;i<=16;i++)); do
echo localhost >> ~/host.list
done
export MP_HOSTFILE=~/host.list
./configure --prefix=$GROMACS_LOCATION --without-motif-includes
--without-motif-libraries --without-x --without-xml --enable-mpi
--disable-nice --program-suffix="_mpi" CC=mpcc_r F77=mpxlf_r >
output.configure_mpi 2>&1
make mdrun > output.make_mpi 2>&1
make install-mdrun > output.make_install_mpi 2>&1
make distclean
</source>

-- [[User:Cneale|cneale]] 18 August 2009
=====fftw single precision compilation=====

FFTW is required by GROMACS. This compilation must be completed before compiling GROMACS.

<source lang="sh">
mkdir exec
export FFTW_LOCATION=/scratch/cneale/GPC/exe/intel/fftw-3.1.2/exec
module purge
module load openmpi intel
./configure --enable-float --enable-threads --prefix=${FFTW_LOCATION}
make
make install
make distclean
</source>

-- [[User:Cneale|cneale]] 18 August 2009

=====Change to get mvapich2-1.4rc1 to compile gromacs=====

This change is required to the mvapich2-1.4rc1 source code in order to compile GROMACS with it.

<source lang="sh">
src/mpid/ch3/channels/mrail/src/gen2/ibv_channel_manager.c
line 503
unsigned long debug = 0;
to
static unsigned long debug = 0;
</source>

-- [[User:Cneale|cneale]] 18 August 2009
=====Compiling mvapich2-1.4rc1=====

<source lang="sh">
cd /scratch/cneale/exe/mvapich2-1.4rc1
mkdir exec
module purge
module load intel
./configure --prefix=/scratch/cneale/exe/mvapich2-1.4rc1/exec CC=icc
CXX=icpc F90=ifort F77=ifort >output.configure 2>&1
make >output.make 2>&1
make install >output.make_install 2>&1
make distclean
</source>

-- [[User:Cneale|cneale]] 18 August 2009
=====Compiling gromacs on GPC using mvapich2-1.4rc1=====

<source lang="sh">
#!/bin/bash
cd /scratch/cneale/exe/intel/gromacs-4.0.5
mkdir exec
PATH=/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/xcat/bin:/opt/xcat/sbin:/root/bin:/opt/torque/bin:/opt/xcat/bin:/opt/xcat/sbin:/usr/lpp/mmfs/bin:/scratch/cneale/exe/mvapich2-1.4rc1/exec/bin/:/scinet/gpc/x1/intel/Compiler/11.0/081/bin/intel64
LD_LIBRARY_PATH=/scratch/cneale/exe/mvapich2-1.4rc1/exec/lib/:/scinet/gpc/x1/intel/Compiler/11.0/081/lib/intel64:/scinet/gpc/x1/intel/Compiler/11.0/081/mkl/lib/em64t/
export FFTW_LOCATION=/scratch/cneale/exe/intel/fftw-3.1.2/exec
export GROMACS_LOCATION=/scratch/cneale/exe/intel/gromacs-4.0.5/exec
export CPPFLAGS="-I$FFTW_LOCATION/include
-I/scratch/cneale/exe/mvapich2-1.4rc1/exec/include
-I/scratch/cneale/exe/mvapich2-1.4rc1/exec/lib"
export LDFLAGS=-L$FFTW_LOCATION/lib
./configure --prefix=$GROMACS_LOCATION --without-motif-includes
--without-motif-libraries --without-x --without-xml --enable-mpi
--program-suffix="_mvapich2" >output.configure.mpi.mvapich2 2>&1
make >output.make.mpi.mvapich2 2>&1
make install-mdrun >output.make_install.mpi.mvapich2 2>&1
make distclean
</source>

-- [[User:Cneale|cneale]] 18 August 2009

=====Using Intel Math Kernel Library for FFT=====
Gromacs provides the option to use the Intel Math Kernel Library (MKL) for the fast Fourier implementation. To use the MKL, make the following changes to your chosen compilation method:

* Do not specify the FFTW_LOCATION environment variable, and remove references to it from CPPFLAGS and LDFLAGS.
* Add -I/opt/intel/Compiler/11.0/081/mkl/include/fftw/ to the CPPFLAGS environment variable.
* Add "-L/opt/intel/Compiler/11.0/081/lib/intel64/ -L/opt/intel/Compiler/11.0/081/mkl/lib/em64t/ -lguide -lpthread -lfftw3xc_intel" to the LDFLAGS environment variable.
* Add --with-fft=mkl to your ./configure commandline.
* Continue your build as usual.

--[[User:Cmadill|Cmadill]] 20:33, 26 September 2009 (UTC)

==TODO list==

Compiling Gromacs

2009-12-05T20:24:35Z

Dacaplan: /* Compiling serial single precision gromacs on GPC */

==Available Compilations==

Chris Neale has compiled gromacs on GPC, with assistance from Scott Northrup, and
on the power6 cluster with assistance from Ching-Hsing Yu. Users are welcome to utilize
these binary executables, but only at their own peril since compiling and testing your own
executable is safer and more stable.

''Gromacs executables:''

'''GPC:''' /scratch/cneale/exe/intel/gromacs-4.0.5/exec/bin

'''TCS:''' /scratch/cneale/exe/gromacs-4.0.4_aix/exec/bin

==Compiling your own GROMACS executables==

Below you will find scripts for the different compilations that you can follow to make your own binaries.

'''NOTE:''' ''the steps are not listed in order! You must compile fftw before compiling gromacs, and if you are going to use mvapich2-1.4rc1 then you must compile it also before compiling parallel gromacs.''

# Compiling serial single precision gromacs on GPC
# Compiling openmpi parallel gromacs on GPC
# Compiling serial gromacs on the power6 (submitted to the queue):
# Compiling parallel gromacs on the power6 (submitted to the queue):
# fftw single precision compilation
# Change to get mvapich2-1.4rc1 to compile gromacs
# Compiling mvapich2-1.4rc1
# Compiling gromacs on GPC using mvapich2-1.4rc1

-- [[User:Cneale|cneale]] 18 August 2009

=====Compiling serial single precision gromacs on GPC=====

<source lang="sh">
cd /scratch/cneale/exe/intel/gromacs-4.0.5
mkdir exec
module purge
module load intel
export FFTW_LOCATION=/scratch/cneale/GPC/exe/intel/fftw-3.1.2/exec
export GROMACS_LOCATION=/scratch/cneale/GPC/exe/intel/gromacs-4.0.5/exec
export CPPFLAGS=-I$FFTW_LOCATION/include
export LDFLAGS=-L$FFTW_LOCATION/lib
./configure --prefix=$GROMACS_LOCATION --without-motif-includes
--without-motif-libraries --without-x --without-xml >output.configure
2>&1
make >output.make 2>&1
make install >output.make_install 2>&1
make distclean
</source>

-- [[User:Cneale|cneale]] 18 August 2009

=====Compiling openmpi parallel gromacs on GPC:=====

<source lang="sh">
cd /scratch/cneale/exe/intel/gromacs-4.0.5
mkdir exec
module purge
module load openmpi intel
export FFTW_LOCATION=/scratch/cneale/exe/intel/fftw-3.1.2/exec
export GROMACS_LOCATION=/scratch/cneale/exe/intel/gromacs-4.0.5/exec
export CPPFLAGS="-I$FFTW_LOCATION/include
-I/scinet/gpc/mpi/openmpi/1.3.2-intel-v11.0-ofed/include
-I/scinet/gpc/mpi/openmpi/1.3.2-intel-v11.0-ofed/lib"
export LDFLAGS=-L$FFTW_LOCATION/lib
/gpc/mpi/openmpi/1.3.2-intel-v11.0-ofed/lib/openmpi
-I/scinet/gpc/x1/intel/Compiler/11.0/081/lib/intel64
-I/scinet/gpc/x1/intel/Compiler/11.0/081/mkl/lib/em64t/"
./configure --prefix=$GROMACS_LOCATION --without-motif-includes
--without-motif-libraries --without-x --without-xml --enable-mpi
--program-suffix="_openmpi" >output.configure.mpi 2>&1
make >output.make.mpi 2>&1
make install-mdrun >output.make_install.mpi 2>&1
make distclean
</source>

-- [[User:Cneale|cneale]] 18 August 2009
=====Compiling serial gromacs on the power6 (submitted to the queue)=====

Note that the -O5 flag for the power6 compilation makes it take about
20 hours to compile. You can drop that if you want, but it does give
you a few more percent performance.

<source lang="sh">
#======================================================================
# Specifies the name of the shell to use for the job
# @ shell = /usr/bin/ksh
# @ job_type = serial
# @ class = verylong
## # @ node = 1
## # @ tasks_per_node = 1
# @ output = $(jobid).out
# @ error = $(jobid).err
# @ wall_clock_limit = 40:00:00
#=====================================
## this is necessary in order to avoid core dumps for batch files
## which can cause the system to be overloaded
# ulimits
# @ core_limit = 0
#=====================================
## necessary to force use of infiniband network for MPI traffic
### TURN IT OFF # @ network.MPI = csss,not_shared,US,HIGH
#=====================================
# @ environment=COPY_ALL
# @ queue
export
PATH=/usr/lpp/ppe.hpct/bin:/usr/vacpp/bin:.:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java14/jre/bin:/usr/java14/bin:/usr/lpp/LoadL/full/bin:/usr/local/bin
export F77=xlf_r
export CC=xlc_r
export CXX=xlc++_r
export FFLAGS="-O5 -qarch=pwr6 -qtune=pwr6"
export CFLAGS="-O5 -qarch=pwr6 -qtune=pwr6"
export CXXFLAGS="-O5 -qarch=pwr6 -qtune=pwr6"
export FFTW_LOCATION=/scratch/cneale/exe/fftw-3.1.2_aix/exec
export GROMACS_LOCATION=/scratch/cneale/exe/gromacs-4.0.4_aix/exec
export CPPFLAGS=-I$FFTW_LOCATION/include
export LDFLAGS=-L$FFTW_LOCATION/lib
cd /scratch/cneale/exe/gromacs-4.0.4_aix
mkdir exec
./configure --prefix=$GROMACS_LOCATION --without-motif-includes
--without-motif-libraries --without-x --without-xml >output.configure
2>&1
make >output.make 2>&1
make install >output.make_install 2>&1
make distclean
</source>

-- [[User:Cneale|cneale]] 18 August 2009
=====Compiling parallel gromacs on the power6 (submitted to the queue)=====

<source lang="sh">
#===============================================================================
# Specifies the name of the shell to use for the job
# @ shell = /usr/bin/ksh
##### @ job_type = serial
# @ job_type = parallel
# @ class = verylong
# @ node = 1
# @ tasks_per_node = 1
# @ output = $(jobid).out
# @ error = $(jobid).err
# @ wall_clock_limit = 40:00:00
#=====================================
## this is necessary in order to avoid core dumps for batch files
## which can cause the system to be overloaded
# ulimits
# @ core_limit = 0
#=====================================
## necessary to force use of infiniband network for MPI traffic
### TURN IT OFF # @ network.MPI = csss,not_shared,US,HIGH
#=====================================
# @ environment=COPY_ALL
# @ queue

export F77=xlf_r
export CC=xlc_r
export CXX=xlc++_r
export FFLAGS="-O5 -qarch=pwr6 -qtune=pwr6"
export CFLAGS="-O5 -qarch=pwr6 -qtune=pwr6"
export CXXFLAGS="-O5 -qarch=pwr6 -qtune=pwr6"
export FFTW_LOCATION=/scratch/cneale/exe/fftw-3.1.2_aix/exec
export GROMACS_LOCATION=/scratch/cneale/exe/gromacs-4.0.4_aix/exec
export CPPFLAGS=-I$FFTW_LOCATION/include
export LDFLAGS=-L$FFTW_LOCATION/lib
cd /scratch/cneale/exe/gromacs-4.0.4_aix
echo "cn-r0-10" > ~/.rhosts
echo localhost > ~/host.list
for((i=2;i<=16;i++)); do
echo localhost >> ~/host.list
done
export MP_HOSTFILE=~/host.list
./configure --prefix=$GROMACS_LOCATION --without-motif-includes
--without-motif-libraries --without-x --without-xml --enable-mpi
--disable-nice --program-suffix="_mpi" CC=mpcc_r F77=mpxlf_r >
output.configure_mpi 2>&1
make mdrun > output.make_mpi 2>&1
make install-mdrun > output.make_install_mpi 2>&1
make distclean
</source>

-- [[User:Cneale|cneale]] 18 August 2009
=====fftw single precision compilation=====

FFTW is required by GROMACS. This compilation must be completed before compiling GROMACS.

<source lang="sh">
mkdir exec
export FFTW_LOCATION=/scratch/cneale/exe/intel/fftw-3.1.2/exec
module purge
module load openmpi intel
./configure --enable-float --enable-threads --prefix=${FFTW_LOCATION}
make
make install
make distclean
</source>

-- [[User:Cneale|cneale]] 18 August 2009
=====Change to get mvapich2-1.4rc1 to compile gromacs=====

This change is required to the mvapich2-1.4rc1 source code in order to compile GROMACS with it.

<source lang="sh">
src/mpid/ch3/channels/mrail/src/gen2/ibv_channel_manager.c
line 503
unsigned long debug = 0;
to
static unsigned long debug = 0;
</source>

-- [[User:Cneale|cneale]] 18 August 2009
=====Compiling mvapich2-1.4rc1=====

<source lang="sh">
cd /scratch/cneale/exe/mvapich2-1.4rc1
mkdir exec
module purge
module load intel
./configure --prefix=/scratch/cneale/exe/mvapich2-1.4rc1/exec CC=icc
CXX=icpc F90=ifort F77=ifort >output.configure 2>&1
make >output.make 2>&1
make install >output.make_install 2>&1
make distclean
</source>

-- [[User:Cneale|cneale]] 18 August 2009
=====Compiling gromacs on GPC using mvapich2-1.4rc1=====

<source lang="sh">
#!/bin/bash
cd /scratch/cneale/exe/intel/gromacs-4.0.5
mkdir exec
PATH=/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/xcat/bin:/opt/xcat/sbin:/root/bin:/opt/torque/bin:/opt/xcat/bin:/opt/xcat/sbin:/usr/lpp/mmfs/bin:/scratch/cneale/exe/mvapich2-1.4rc1/exec/bin/:/scinet/gpc/x1/intel/Compiler/11.0/081/bin/intel64
LD_LIBRARY_PATH=/scratch/cneale/exe/mvapich2-1.4rc1/exec/lib/:/scinet/gpc/x1/intel/Compiler/11.0/081/lib/intel64:/scinet/gpc/x1/intel/Compiler/11.0/081/mkl/lib/em64t/
export FFTW_LOCATION=/scratch/cneale/exe/intel/fftw-3.1.2/exec
export GROMACS_LOCATION=/scratch/cneale/exe/intel/gromacs-4.0.5/exec
export CPPFLAGS="-I$FFTW_LOCATION/include
-I/scratch/cneale/exe/mvapich2-1.4rc1/exec/include
-I/scratch/cneale/exe/mvapich2-1.4rc1/exec/lib"
export LDFLAGS=-L$FFTW_LOCATION/lib
./configure --prefix=$GROMACS_LOCATION --without-motif-includes
--without-motif-libraries --without-x --without-xml --enable-mpi
--program-suffix="_mvapich2" >output.configure.mpi.mvapich2 2>&1
make >output.make.mpi.mvapich2 2>&1
make install-mdrun >output.make_install.mpi.mvapich2 2>&1
make distclean
</source>

-- [[User:Cneale|cneale]] 18 August 2009

=====Using Intel Math Kernel Library for FFT=====
Gromacs provides the option to use the Intel Math Kernel Library (MKL) for the fast Fourier implementation. To use the MKL, make the following changes to your chosen compilation method:

* Do not specify the FFTW_LOCATION environment variable, and remove references to it from CPPFLAGS and LDFLAGS.
* Add -I/opt/intel/Compiler/11.0/081/mkl/include/fftw/ to the CPPFLAGS environment variable.
* Add "-L/opt/intel/Compiler/11.0/081/lib/intel64/ -L/opt/intel/Compiler/11.0/081/mkl/lib/em64t/ -lguide -lpthread -lfftw3xc_intel" to the LDFLAGS environment variable.
* Add --with-fft=mkl to your ./configure commandline.
* Continue your build as usual.

--[[User:Cmadill|Cmadill]] 20:33, 26 September 2009 (UTC)

==TODO list==

Data Management

2009-08-14T15:05:05Z

Dacaplan:

SciNet's storage system is based on IBM's [http://en.wikipedia.org/wiki/IBM_General_Parallel_File_System GPFS] (General Parallel File System). There are two main systems for user data: <tt>/home</tt>, a small, backed-up space where user home directories are located, and <tt>/scratch</tt>, a large system for input or output data for jobs; data on <tt>/scratch</tt> is not only not backed up, data placed there will be deleted after a two weeks. SciNet does not provide long-term storage for large data sets.

==Home Disk Space==

Every SciNet user gets a 10GB directory on <tt>/home</tt> which is regularly backed-up. Home is visible from <tt>login.scinet</tt> nodes, and from the development nodes on [[GPC_Quickstart | GPC]] and the [[TCS_Quickstart | TCS]]. However, on the compute nodes of the GPC clusters -- as when jobs are running -- <tt>/home</tt> is mounted read-only; thus GPC jobs can read files in /home but cannot write to files there. <tt>/home</tt> is a good place to put code, input files for runs, and anything else that needs to be kept to reproduce runs.

==Scratch Disk Space==

Every SciNet user also gets a directory in <tt>/scratch</tt>. Scratch is visible from the <tt>login.scinet</tt> nodes, the development nodes on [[GPC_Quickstart | GPC]] and the [[TCS_Quickstart | TCS]], and on the compute nodes of the clusters, mounted as read-write. Thus jobs must write their output somewhere in <tt>/scratch</tt>. There are '''NO''' backups of anything on /scratch.

There is a large amount of space available on <tt>/scratch</tt> but it is purged every two weeks so that all users running jobs and generating large outputs will have room to store their data temporarily. Computational results which you want to keep longer than this must be copied (using <tt>scp</tt>) off of SciNet entirely and to your local system. SciNet does not provide long-term storage for large data sets.

==Performance==

[http://en.wikipedia.org/wiki/IBM_General_Parallel_File_System GPFS] is a high-performance filesystem which provides rapid reads and writes to large datasets in parallel from many nodes. As a consequence of this design, however, it performs quite ''poorly'' at accessing data sets which consist of many, small files. For instance, you will find that reading data in from one 4MB file is enormously faster than from 100 40KB files. Such small files are also quite wasteful of space, as the blocksize for the filesystem is 4MB. This is something you should keep in mind when planning your input/output strategy for runs on SciNet.