Difference between revisions of "Knights Landing"
| Line 16: | Line 16: | ||
The nodes are interconnected to the rest of the clusters with QDR Infiniband and shares the regular SciNet GPFS filesystems. | The nodes are interconnected to the rest of the clusters with QDR Infiniband and shares the regular SciNet GPFS filesystems. | ||
| − | + | == Login == | |
First login via ssh with your SciNet account at '''<tt>login.scinet.utoronto.ca</tt>''', and from there you can proceed to '''<tt>knl01,knl02,knl03,knl04</tt>'''. | First login via ssh with your SciNet account at '''<tt>login.scinet.utoronto.ca</tt>''', and from there you can proceed to '''<tt>knl01,knl02,knl03,knl04</tt>'''. | ||
| − | + | == Memory Modes == | |
The four nodes all have identical hardware, however there are multiple options that control how the MCDRAM is accessed and changing the modes require the nodes be rebooted. | The four nodes all have identical hardware, however there are multiple options that control how the MCDRAM is accessed and changing the modes require the nodes be rebooted. | ||
| − | + | === Clustering === | |
Currently all KNL nodes have the Cluster Mode configured to "Quadrant". See [http://colfaxresearch.com/knl-numa/ this] article for more details about the clustering options that contorl how memory is accessed on the KNL. | Currently all KNL nodes have the Cluster Mode configured to "Quadrant". See [http://colfaxresearch.com/knl-numa/ this] article for more details about the clustering options that contorl how memory is accessed on the KNL. | ||
| − | + | === MCDRAM === | |
Two nodes, '''<tt>knl01,kn02</tt>''' have the MCDRAM configured as "Cache" mode and the other two '''<tt>knl03,kn04</tt>''' are configured with the "Flat" memory mode. | Two nodes, '''<tt>knl01,kn02</tt>''' have the MCDRAM configured as "Cache" mode and the other two '''<tt>knl03,kn04</tt>''' are configured with the "Flat" memory mode. | ||
| Line 35: | Line 35: | ||
Initially when you first compile/port your code, use the Cache mode nodes, however if you wish to try and optimize memory performance by directly using the HBM memory with the [http://colfaxresearch.com/knl-mcdram/#sec-3-2 memkind] library, use the Flat nodes. | Initially when you first compile/port your code, use the Cache mode nodes, however if you wish to try and optimize memory performance by directly using the HBM memory with the [http://colfaxresearch.com/knl-mcdram/#sec-3-2 memkind] library, use the Flat nodes. | ||
| − | |||
| − | + | == Queue == | |
| − | |||
| − | |||
| − | |||
Currently there is no queue, be nice. | Currently there is no queue, be nice. | ||
| Line 63: | Line 59: | ||
</pre> | </pre> | ||
| − | == | + | == References == |
http://dap.xeonphi.com/#implinks | http://dap.xeonphi.com/#implinks | ||
Revision as of 19:12, 19 September 2016
| Intel Xeon Phi (Knights Landing ) | |
|---|---|
| Installed | August 2016 |
| Operating System | Linux Centos 7.2 |
| Number of Nodes | 4 |
| Interconnect | QDR Infiniband |
| Ram/Node | 96GB DDR4 + 16GB MCDRAM |
| Cores/Node | 64 |
| Login/Devel Node | knl01 |
| Vendor Compilers | icc,ifort |
| Queue Submission | none |
This is develop/test system of four x86_64 self-hosted 2nd Generation Intel Xeon Phi (Knights Landing, KNL) nodes, aka an Intel "Ninja" platform. Each node has one 64-core Intel(R) Xeon Phi(TM) CPU 7210 @ 1.30GHz with 4 threads per core. These systems are not add-on accelerators, but instead act as full-fledged processors running a regular linux operating system. They are configured with 96GB of DDR4 system RAM along with 16GB of very fast MCDRAM, see here for details. The nodes are interconnected to the rest of the clusters with QDR Infiniband and shares the regular SciNet GPFS filesystems.
Login
First login via ssh with your SciNet account at login.scinet.utoronto.ca, and from there you can proceed to knl01,knl02,knl03,knl04.
Memory Modes
The four nodes all have identical hardware, however there are multiple options that control how the MCDRAM is accessed and changing the modes require the nodes be rebooted.
Clustering
Currently all KNL nodes have the Cluster Mode configured to "Quadrant". See this article for more details about the clustering options that contorl how memory is accessed on the KNL.
MCDRAM
Two nodes, knl01,kn02 have the MCDRAM configured as "Cache" mode and the other two knl03,kn04 are configured with the "Flat" memory mode. See this article for more details about the MCDRAM memory modes.
Initially when you first compile/port your code, use the Cache mode nodes, however if you wish to try and optimize memory performance by directly using the HBM memory with the memkind library, use the Flat nodes.
Queue
Currently there is no queue, be nice.
Software
Software is available using the standard modules framework used on other SciNet systems, however is separate from the GPC modules as the KNL has a newer Centos7 based operating system.
Compilers
The Xeon Phi uses the standard intel compilers.
module load intel/16.0.3
MPI
IntelMPI is currently the default MPI
module load intelmpi/5.1.3.219