Difference between revisions of "P7 Linux Cluster"
Jump to navigation
Jump to search
| Line 13: | Line 13: | ||
}} | }} | ||
| − | + | == Specifications== | |
The P7 Cluster consists of 5 IBM Power 755 Servers each with 4x 8core 3.3GHz Power7 CPUs and 128GB Ram. Similar to the Power 6, the Power 7 utilizes | The P7 Cluster consists of 5 IBM Power 755 Servers each with 4x 8core 3.3GHz Power7 CPUs and 128GB Ram. Similar to the Power 6, the Power 7 utilizes | ||
| Line 19: | Line 19: | ||
threads which in many cases can lead to significant speedups. | threads which in many cases can lead to significant speedups. | ||
| − | + | == Login == | |
First login via ssh with your scinet account at '''<tt>login.scinet.utoronto.ca</tt>''', and from there you can proceed to '''<tt>p701</tt>''' which | First login via ssh with your scinet account at '''<tt>login.scinet.utoronto.ca</tt>''', and from there you can proceed to '''<tt>p701</tt>''' which | ||
is currently the gateway/devel node for this cluster. | is currently the gateway/devel node for this cluster. | ||
| − | + | == Compiler/Devel Node == | |
From '''<tt>p701</tt>''' you can compile, do short tests, and submit your jobs to the queue. | From '''<tt>p701</tt>''' you can compile, do short tests, and submit your jobs to the queue. | ||
| − | + | === GNU === | |
gcc/g++/gfortran version 4.4.4 is the default with RHEL 6.0 and is available by default. | gcc/g++/gfortran version 4.4.4 is the default with RHEL 6.0 and is available by default. | ||
| − | + | === IBM === | |
To use the IBM Power specific compilers xlc/xlc++/xlf you need to load the following modules | To use the IBM Power specific compilers xlc/xlc++/xlf you need to load the following modules | ||
<pre> | <pre> | ||
| Line 39: | Line 39: | ||
NOTE: Be sure to use "-q64" when using the IBM compilers. | NOTE: Be sure to use "-q64" when using the IBM compilers. | ||
| − | + | === MPI === | |
| − | + | IBM's POE based on MPICH2 is available by default, however you need to load the IBM compiler | |
| + | modules for it to work properly. | ||
| + | |||
| + | <!-- | ||
| + | ==== OpenMPI ==== | ||
<pre> | <pre> | ||
| Line 50: | Line 54: | ||
$ module openmpi/1.5.3-ibm-11.1+13.1 | $ module openmpi/1.5.3-ibm-11.1+13.1 | ||
</pre> | </pre> | ||
| + | --> | ||
| + | === Submit a Job === | ||
| − | IBM's | + | The current Scheduler is IBM's LoadLeveler similar to the implementation on the TCS. |
| − | + | Below is a sample script. | |
| − | |||
<pre> | <pre> | ||
#!/bin/bash | #!/bin/bash | ||
| − | # | + | #=================================== |
# P7 Load Leveler Submission Script | # P7 Load Leveler Submission Script | ||
#=================================== | #=================================== | ||
# | # | ||
| − | ## | + | # Don't change these parameters unless you really know what you are doing |
| − | #@ environment = MP_INFOLEVEL= | + | # |
| + | #@ environment = MP_INFOLEVEL=0; MP_USE_BULK_XFER=yes; MP_BULK_MIN_MSG_SIZE=64K; \ | ||
# MP_EAGER_LIMIT=64K; LAPI_DEBUG_ENABLE_AFFINITY=no | # MP_EAGER_LIMIT=64K; LAPI_DEBUG_ENABLE_AFFINITY=no | ||
| − | |||
| − | |||
# | # | ||
#=================================== | #=================================== | ||
| − | # | + | # Avoid core dumps |
# @ core_limit = 0 | # @ core_limit = 0 | ||
#=================================== | #=================================== | ||
| − | # Job | + | # Job specific |
#=================================== | #=================================== | ||
# | # | ||
| − | # @ job_name | + | # @ job_name = myjob |
| − | |||
# @ job_type = parallel | # @ job_type = parallel | ||
# @ class = verylong | # @ class = verylong | ||
| Line 83: | Line 86: | ||
# @ wall_clock_limit = 2:00:00 | # @ wall_clock_limit = 2:00:00 | ||
# | # | ||
| − | + | # @ node = 2 | |
| − | |||
| − | |||
| − | # @ node = | ||
# @ tasks_per_node = 128 | # @ tasks_per_node = 128 | ||
# | # | ||
| − | #@ queue | + | # @ queue |
# | # | ||
#=================================== | #=================================== | ||
| − | |||
| − | |||
| − | + | ./my_code | |
| − | |||
| − | |||
| − | + | </pre> | |
| + | To submit a job | ||
| + | <pre> | ||
| + | llsubmit myjob.ll | ||
</pre> | </pre> | ||
| + | |||
| + | To show running jobs use | ||
| + | <pre> | ||
| + | llq | ||
| + | </pre> | ||
| + | |||
| + | To cancel a job use | ||
| + | <pre> | ||
| + | llcancel JOBID | ||
| + | </pre> | ||
| + | |||
| + | |||
| + | |||
Revision as of 10:29, 15 June 2011
| P7 Cluster (P7) | |
|---|---|
| Installed | May 2011 |
| Operating System | Linux (RHEL 6.0) |
| Interconnect | Infiniband |
| Ram/Node | 128 Gb |
| Cores/Node | 32 (128 Threads) |
| Login/Devel Node | p701 (from login.scinet) |
| Vendor Compilers | xlc/xlf |
| Queue Submission | LoadLeveler |
Specifications
The P7 Cluster consists of 5 IBM Power 755 Servers each with 4x 8core 3.3GHz Power7 CPUs and 128GB Ram. Similar to the Power 6, the Power 7 utilizes Simultaneous Multi Threading (SMT), but extends the design from 2 threads per core to 4. This allows the 32 physical cores to support up to 128 threads which in many cases can lead to significant speedups.
Login
First login via ssh with your scinet account at login.scinet.utoronto.ca, and from there you can proceed to p701 which is currently the gateway/devel node for this cluster.
Compiler/Devel Node
From p701 you can compile, do short tests, and submit your jobs to the queue.
GNU
gcc/g++/gfortran version 4.4.4 is the default with RHEL 6.0 and is available by default.
IBM
To use the IBM Power specific compilers xlc/xlc++/xlf you need to load the following modules
$ module load vacpp xlf
NOTE: Be sure to use "-q64" when using the IBM compilers.
MPI
IBM's POE based on MPICH2 is available by default, however you need to load the IBM compiler modules for it to work properly.
Submit a Job
The current Scheduler is IBM's LoadLeveler similar to the implementation on the TCS. Below is a sample script.
#!/bin/bash #=================================== # P7 Load Leveler Submission Script #=================================== # # Don't change these parameters unless you really know what you are doing # #@ environment = MP_INFOLEVEL=0; MP_USE_BULK_XFER=yes; MP_BULK_MIN_MSG_SIZE=64K; \ # MP_EAGER_LIMIT=64K; LAPI_DEBUG_ENABLE_AFFINITY=no # #=================================== # Avoid core dumps # @ core_limit = 0 #=================================== # Job specific #=================================== # # @ job_name = myjob # @ job_type = parallel # @ class = verylong # @ output = $(jobid).out # @ error = $(jobid).err # @ wall_clock_limit = 2:00:00 # # @ node = 2 # @ tasks_per_node = 128 # # @ queue # #=================================== ./my_code
To submit a job
llsubmit myjob.ll
To show running jobs use
llq
To cancel a job use
llcancel JOBID