Difference between revisions of "P7 Linux Cluster"
(Created page with "{{Infobox Computer |name=P7 Cluster (P7) |installed=May 2011 |operatingsystem= Linux (RHEL 6.0) |loginnode= p7n01 (from <tt>login.scinet</tt>) |numberofnodes=5 |rampernode=128 Gb...") |
|||
| Line 1: | Line 1: | ||
{{Infobox Computer | {{Infobox Computer | ||
| + | |image=[[Image:Tesla_S2070_3qtr.gif|center|300px|thumb]] | ||
|name=P7 Cluster (P7) | |name=P7 Cluster (P7) | ||
|installed=May 2011 | |installed=May 2011 | ||
| Line 8: | Line 9: | ||
|corespernode=32 (128 Threads) | |corespernode=32 (128 Threads) | ||
|interconnect=Infiniband | |interconnect=Infiniband | ||
| − | |vendorcompilers= | + | |vendorcompilers=xlc/xlf |
|queuetype=Torque | |queuetype=Torque | ||
}} | }} | ||
| + | |||
| + | === Specifications=== | ||
The P7 Cluster consists of 5 IBMPower 755 Servers each with 4x 8core 3.3GHz Power7 CPUs and 128GB Ram. | The P7 Cluster consists of 5 IBMPower 755 Servers each with 4x 8core 3.3GHz Power7 CPUs and 128GB Ram. | ||
| Line 19: | Line 22: | ||
is currently the gateway/devel node for this cluster. | is currently the gateway/devel node for this cluster. | ||
| − | == | + | === Compiler/Devel Node === |
| + | |||
| + | Two compilers are available, gcc/g++/gfortran version 4.4.4 is the default with RHEL 6.0 and is available by default. | ||
| + | |||
| + | To use the IBM Power specific compilers xlc/xlc++/xlf you need to load the following modules | ||
| + | <pre> | ||
| + | $ module load vacpp xlf | ||
| + | </pre> | ||
| + | |||
| + | NOTE: Be sure to use "-q64" if using the IBM compilers. | ||
| + | |||
| + | === MPI === | ||
| + | |||
| + | OpenMPI is available for both compilers | ||
| + | |||
| + | <pre> | ||
| + | $ module openmpi/1.5.3-gcc-v4.4.4 | ||
| + | </pre> | ||
| + | |||
| + | <pre> | ||
| + | $ module openmpi/1.5.3-ibm-11.1+13.1 | ||
| + | </pre> | ||
| + | |||
| + | |||
| + | IBM's POE is installed but due to current problems with loadleveler/lapi/poe it is not recommended for use. | ||
| + | |||
| + | === Submit a Job === | ||
| + | |||
| + | Currently a vary basic torque queuing system has been setup and as such a job can be submitted in the regular | ||
| + | torque way as discussed in the GPC | ||
| + | |||
| + | Create a script | ||
| + | <pre> | ||
| + | #!/bin/bash | ||
| + | # P7 script | ||
| + | #PBS -l nodes=1:ppn=128,walltime=1:00:00 | ||
| + | #PBS -N P7test | ||
| + | |||
| + | cd $PBS_O_WORKDIR | ||
| + | |||
| + | mpirun -np 128 ./a.out | ||
| + | </pre> | ||
| + | |||
| + | Then submit | ||
| + | <pre> | ||
| + | qsub example.sh | ||
| + | </pre> | ||
| + | |||
| + | Interactive sessions can also be facilitated with | ||
| + | |||
| + | <pre> | ||
| + | qsub -I -l nodes=1:ppn=128,walltime=1:00:00 | ||
| + | </pre> | ||
| + | |||
| + | To see running jobs use | ||
| + | <pre> | ||
| + | qstat | ||
| + | </pre> | ||
| + | |||
| + | and to cancel a running or queued job | ||
| + | <pre> | ||
| + | qdel JOBID | ||
| + | </pre> | ||
Revision as of 13:09, 19 May 2011
| P7 Cluster (P7) | |
|---|---|
| Installed | May 2011 |
| Operating System | Linux (RHEL 6.0) |
| Interconnect | Infiniband |
| Ram/Node | 128 Gb |
| Cores/Node | 32 (128 Threads) |
| Login/Devel Node | p7n01 (from login.scinet) |
| Vendor Compilers | xlc/xlf |
| Queue Submission | Torque |
Specifications
The P7 Cluster consists of 5 IBMPower 755 Servers each with 4x 8core 3.3GHz Power7 CPUs and 128GB Ram.
Login
First login via ssh with your scinet account at login.scinet.utoronto.ca, and from there you can proceed to p7n01 which is currently the gateway/devel node for this cluster.
Compiler/Devel Node
Two compilers are available, gcc/g++/gfortran version 4.4.4 is the default with RHEL 6.0 and is available by default.
To use the IBM Power specific compilers xlc/xlc++/xlf you need to load the following modules
$ module load vacpp xlf
NOTE: Be sure to use "-q64" if using the IBM compilers.
MPI
OpenMPI is available for both compilers
$ module openmpi/1.5.3-gcc-v4.4.4
$ module openmpi/1.5.3-ibm-11.1+13.1
IBM's POE is installed but due to current problems with loadleveler/lapi/poe it is not recommended for use.
Submit a Job
Currently a vary basic torque queuing system has been setup and as such a job can be submitted in the regular torque way as discussed in the GPC
Create a script
#!/bin/bash # P7 script #PBS -l nodes=1:ppn=128,walltime=1:00:00 #PBS -N P7test cd $PBS_O_WORKDIR mpirun -np 128 ./a.out
Then submit
qsub example.sh
Interactive sessions can also be facilitated with
qsub -I -l nodes=1:ppn=128,walltime=1:00:00
To see running jobs use
qstat
and to cancel a running or queued job
qdel JOBID