Difference between revisions of "GPU Devel Nodes"

Latest revision as of 19:26, 31 August 2018

WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to https://docs.scinet.utoronto.ca

The ARC GPU have been decommisioned. The head node, arc01, is still up, however for GPU computations, users are encouraged to move to the Gravity cluster. For visualization, new Visualization Nodes are being setup.

Difference between revisions of "GPU Devel Nodes"

Latest revision as of 19:26, 31 August 2018

Navigation menu

Search

@@ Line 1: / Line 1: @@
-{{Infobox Computer
+{| style="border-spacing: 8px; width:100%"
-|image=[[Image:Tesla_S2070_3qtr.gif|center|300px|thumb]]
+| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|
-|name=GPU Development Cluster
+'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''
-|installed=April 2011
+|}
-|operatingsystem= Linux Centos 6.0
-|loginnode= arc01 (from <tt>login.scinet</tt>)
-|nnodes=8
-|rampernode=48 Gb
-|corespernode=8 with 2xGPUs
-|interconnect=DDR Infiniband
-|vendorcompilers=nvcc,pgcc,icc,gcc
-|queuetype=Torque
-}}
-The GPU cluster, part of the [[Accelerator Research Cluster]], consists of 8 x86_64 nodes each with two quad core Intel Xeon X5550 2.67GHz CPUs with 48GB of RAM per node.  Each node has two NVIDIA Tesla M2070 GPUs
+<span style="color:#772222">The ARC GPU have been decommisioned. The head node, arc01, is still up, however for GPU computations, users are encouraged to move to the [[Gravity]] cluster.  For visualization, new [[Visualization Nodes]] are being setup.</span>
-with CUDA Capability 2.0 (Fermi) each with 448 CUDA Cores @ 1.15GHz and 6 GB of RAM.  The nodes are interconnected with DDR Infiniband for MPI communications
-and disk I/O to the SciNet GPFS filesystems.  In total this cluster contains 64 x86_64 cores with 384 GB of system RAM and 16 GPUs with 96 GB GPU RAM total.
-Note that SciNet has a mailing lists for people interested in GPGPU computing. To receive information on courses, workshop and other GPGPU related events, sign up at https://support.scinet.utoronto.ca/mailman/listinfo/scinet-gpgpu.
-== Nodes ==
-=== Login ===
-First login via ssh with your scinet account at <tt>login.scinet.utoronto.ca</tt>, and from there you can proceed to '''<tt>arc01</tt>''' which
-is the GPU development node.
-Access to these machines is currently controlled. Please email support@scinet.utoronto.ca for access.
-=== Devel ===
-As mentioned '''<tt>arc01</tt>''' is the head/develop node for interactive use.  This node is for compiling, short testing, and submitting
-batch jobs to the compute nodes.  It is a shared resource so treat it accordingly and use the queue and compute nodes for long are large
-computations.
-=== Xeon Phi/ Tesla K20 ===
-A separate devel node, '''<tt>arc09</tt>''', with a single Intel Xeon Phi and a NVIDIA Tesla K20 is also available for testing these newer technologies.
-For full details see the '''[[phi| Xeon Phi / Tesla K20 ]]''' wiki page.
-=== Compute ===
-To access the other 7 compute nodes with GPU's you need to use the queue, similar to the standard GPC compute nodes.
-Currently the nodes are scheduled by complete node, 8 cores and 2 GPUs, and a maximum walltime of 48 hours.
-For an interactive job use
-<pre>
-qsub -l nodes=1:ppn=8:gpus=2,walltime=48:00:00 -q arc -I
-</pre>
-or for a batch job use
-<pre>
-qsub script.sh
-</pre>
-where <tt>script.sh</tt> is
-<source lang="bash">
-#!/bin/bash
-# Torque submission script for SciNet ARC
-#
-#PBS -l nodes=2:ppn=8:gpus=2,walltime=1:00:00
-#PBS -N GPUtest
-#PBS -q arc
-cd $PBS_O_WORKDIR
-# EXECUTION COMMAND; -np = nodes*ppn
-mpirun -np 16 ./a.out
-</source>
-To check running jobs on the gpu nodes only use
-<pre>
-showq -w class=arc
-</pre>
-== Software ==
-The same software installed on the GPC is available on ARC using the same modules framework.
-See [[GPC_Quickstart#Modules_and_Environment_Variables | here]] for full details.
-==Programming Frameworks==
-Currently there are four programming frameworks to use: NVIDIA's CUDA framework, PGI's CUDA Fortran, PGI's implementation of OpenACC, or OpenCL.
-=== NVIDIA toolkit ===
-==== CUDA ====
-The current installed CUDA Toolkits are 3.2, 4.0, 4.1 (default), and 4.2.  To use 4.0 just add the following module
-<pre>
-module load cuda/4.0
-</pre>
-Note that to use the full 6GB or memory per GPU, CUDA 3.2 or newer must be used.
-The CUDA driver is installed locally, however the CUDA Toolkits are installed in.
-<pre>
-/scinet/arc/cuda-$VERSION/
-</pre>
-The environment variable $SCINET_CUDA_INSTALL is set when a cuda module is loaded and it points to the
-install location.  This is useful when setting up makefiles and if you use the NVIDIA_SDK
-build evironment, modify the NVIDIA_SDK/C/common/common.mk file accordingly.
-<pre>
-CUDA_INSTALL_PATH = $SCINET_CUDA_INSTALL
-</pre>
-The Nvidia cuda compiler (which uses gcc/4.4.6 by default for CUDA < 4.1, while cuda/4.2 uses gcc/4.6.1), is called <tt>nvcc</tt>,
-You'll have to let the cuda compiler know about the capabilities of the Fermi graphics card by supplying the flag
-<pre>-arch=sm_13</pre> or <pre>-arch=sm_20</pre>
-==== NVIDIA Toolkit ====
-For cuda versions 4.0, 4.1, and 4.2, the CUDA SDK can be copied into your home directory from:
-<pre>
-/scinet/arc/src/gpucomputingsdk_4.0.13_linux.run
-/scinet/arc/src/gpucomputingsdk_4.1.28_linux.run
-/scinet/arc/src/gpucomputingsdk_4.2.9_linux.run
-</pre>
-respectively.
-However, for cuda 5.0, the sdk code samples can be copied from the directory
-<pre>
-$SCINET_CUDA_INSTALL/samples/
-</pre>
-NOTE: Not all of the CUDA and OpenCL examples will compile as many require OpenGL graphic libraries not installed on the nodes.
-==== OpenCL ====
-As of 3.0, OpenCL is included in the CUDA Toolkit so loading the CUDA module is all that is required.
-===PGI compilers===
-As of July 2012, The PGI suite of compilers is installed on the ARC.  These can be accessed by
-<pre>$  module load gcc/4.6.1 pgi/12.6</pre>
-(if you use the older pgi/12.5, gcc/4.4.6 is a requirement, and is used, for instance, in the CUDA parts of the PGI compilers). These compilers use their own cuda installation, so you do not need to load an additional cuda module. By default, they use a cuda 4.1 installation, but you can request cuda 4.2 as well using the <tt>-Mcuda=4.2</tt> option.
-The compilation commands are pgcc, pgcpp and pgfortran for c, c++ and fortran, respectively. As usual, we advice to compiler with optimization using the flags
-<pre>
--O4 -fastsse
-</pre>
-The compilers will then optimize for the specific machine that you are compiling on.
-The PGI compilers support OpenMP as well through the compile and link flags
-<pre>
--mp
-</pre>
-==== CUDA Fortran ====
-The PGI fortran compiler (<tt>pgfortran</tt>, also <tt>pgf77</tt> and <tt>pgf90</tt>) understands CUDA extensions to fortran.
-This compiler will automatically understand these extension for source files with the file extension <tt>.cuf</tt>  Otherwise, you have to specify
-<pre>
--Mcuda=4.1
-</pre>
-==== OpenACC ====
-OpenACC is a compiler-directive approach to GPGPU programming. The PGI compilers (c, c++ and fortran) have a partial implementation of this open specification. To switch this on, use the options
-<pre>-acc -ta=nvidia -Mcuda=4.1</pre>
-====More documentation====
-Manuals are on the [[Knowledge_Base:_Tutorials_and_Manuals| Tutorials and Manuals]] page.
-===Other compilers===
-* '''gcc,g++,gfortran''' - GNU compiler (nvcc need to have either gcc-4.4 or gcc-4.6 module loaded to work correctly)
-* '''icc,icpc,ifort''' - Intel compiler
-===Debuggers===
-* '''ddt''' - Allinea's graphical DDT debugger, in the <tt>ddt</tt> module. The most recent version, ddt 4.0 supports cuda 4.0, 4.1, 4.2 and 5.0.
-* '''cuda-gdb''' - Nvidia text based gdb variant, part of the cuda module.
-Note that to debug both host and cuda device code, you have to give the <pre>-g -G</pre> pair of flags to nvcc.
-===MPI===
-The GPC MPI packages can be used on this system. See the GPC section on [[ GPC_Quickstart#MPI |MPI ]] for more details.
-While these mpi packages should work with the PGI compilers as well, this has not been tested and standard wrappers like mpif90 may not work.
-Alternatively, for mpi compilations with the PGI compilers, you can load the mpich1 mpi implementation with <pre>module load mpich1/pgi</pre>after which you can use the option <pre>-Mmpi</pre> or the wrapper scripts <tt>mpicc, mpiCC and mpif90</tt>, as well as <tt>mpirun</tt>.
-=== Driver Version ===
-The current NVIDIA driver version installed is 295.41.
-== Documentation ==
-* CUDA
-** google "CUDA"
-* OpenCL
-** see above
-== Further Info ==
-== User Codes ==
-Please discuss and put any relevant information/problems/best practices you have encountered when using/developing for CUDA and/or OpenCL