oldwiki.scinet.utoronto.ca - User contributions [en-gb]

SOSCIP GPU

2018-03-14T19:21:29Z

Wagnerse: /* Job Submission */

{{Infobox Computer
|image=[[Image:S882lc.png|center|300px|thumb]]
|name=SOSCIP GPU
|installed=September 2017
|operatingsystem= Ubuntu 16.04 le
|loginnode= sgc01
|nnodes= 14x Power 8 with 4x NVIDIA P100
|rampernode=512 GB
|corespernode= 2 x 10core (20 physical, 160 SMT)
|interconnect=Infiniband EDR
|vendorcompilers=xlc/xlf, nvcc
}}

== SOSCIP ==

The SOSCIP GPU Cluster is a Southern Ontario Smart Computing Innovation Platform ([http://soscip.org/ SOSCIP]) resource located at theUniversity of Toronto's SciNet HPC facility. The SOSCIP multi-university/industry consortium is funded by the Ontario Government and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].

== Support Email ==

Please use [mailto:soscip-support@scinet.utoronto.ca <soscip-support@scinet.utoronto.ca>] for SOSCIP GPU specific inquiries.

== Specifications==

The SOSCIP GPU Cluster consists of of 14 IBM Power 822LC "Minsky" Servers each with 2x10core 3.25GHz Power8 CPUs and 512GB Ram. Similar to Power 7, the Power 8 utilizes Simultaneous MultiThreading (SMT), but extends the design to 8 threads per core allowing the 20 physical cores to support up to 160 threads. Each node has 4x NVIDIA Tesla P100 GPUs each with 16GB of RAM with CUDA Capability 6.0 (Pascal) connected using NVlink.

== Access and Login ==

In order to obtain access to the system, you must request access to the SOSCIP GPU Platform. Instructions will have been sent to your sponsoring faculty member via E-mail at the beginning of your SOSCIP project.

Access to the SOSCIP GPU Platform is provided through the BGQ login node, '''<tt> bgqdev.scinet.utoronto.ca </tt>''' using ssh, and from there you can proceed to the GPU development node '''<tt>sgc01-ib0</tt>''' via ssh. Your user name and password is the same as it is for SciNet systems.

== Filesystem ==

The filesystem is shared with the BGQ system. See [https://wiki.scinet.utoronto.ca/wiki/index.php/BGQ#Filesystem here ] for details.

== Job Submission ==

The SOSCIP GPU cluster uses [https://slurm.schedmd.com/ SLURM ] as a job scheduler and jobs are scheduled by node, ie 20 cores and 4 GPUs each. Jobs are submitted from the development node '''<tt>sgc01</tt>'''. The maximum walltime per job is 12 hours (except in the 'long' queue, see below) with up to 8 nodes.

<pre>
$ sbatch myjob.script
</pre>

Where myjob.script is

<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=20 # MPI tasks (needed for srun)
#SBATCH --time=00:10:00 # H:M:S
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node

cd $SLURM_SUBMIT_DIR

hostname
nvidia-smi
</pre>

More information about the <tt>sbatch</tt> command is found [https://slurm.schedmd.com/sbatch.html here].

You can query job information using

<pre>
squeue
</pre>

To see only your own jobs, run

<pre>
squeue -u <userid>
</pre>

Once your job is running, SLURM creates a file usually named <tt>slurm<jobid>.out</tt> in the directory from where you issued the <tt>sbatch</tt> command. This contains the console output from your job. You can monitor the output of your job by using the <tt>tail -f <file></tt> command.

To cancel a job use

<pre>
scancel $JOBID
</pre>

=== Longer jobs ===

If your job takes more than 12 hours, the sbatch command will not let you submit your job. There is, however, a way to have jobs up to 24 hours long, by specifying "-p long" as an option (i.e., add <tt>#SBATCH -p long</tt> to your job script). The priority of such jobs may be throttled in the future if we see that the 'long' queue is having a negative efffect on turnover time in the queue.

=== Interactive ===

For an interactive session use

<pre>
salloc --gres=gpu:4
</pre>

After executing this command, you may have to wait in the queue until a system is available.

More information about the <tt>salloc</tt> command is [https://slurm.schedmd.com/salloc.html here].

=== Automatic Re-submission and Job Dependencies ===

Commonly you may have a job that you know will take longer to run than what is permissible in the queue. As long as your program contains checkpoint or restart capability, you can have one job automatically submit the next. In the following example it is assumed that the program finishes before the time limit requested and then resubmits itself by logging into the development nodes. Job dependencies and a maximum number of job re-submissions are used to ensure sequential operation.

<pre>
#!/bin/bash

#SBATCH --nodes=1
#SBATCH --ntasks=20 # MPI tasks (needed for srun)
#SBATCH --time=00:10:00 # H:M:S
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node

cd $SLURM_SUBMIT_DIR

: ${job_number:="1"} # set job_nubmer to 1 if it is undefined
job_number_max=3

echo "hi from ${SLURM_JOB_ID}"

#RUN JOB HERE

# SUBMIT NEXT JOB
if [[ ${job_number} -lt ${job_number_max} ]]
then
(( job_number++ ))
next_jobid=$(ssh sgc01-ib0 "cd $SLURM_SUBMIT_DIR; /opt/slurm/bin/sbatch --export=job_number=${job_number} -d afterok:${SLURM_JOB_ID} thisscript.sh | awk '{print $4}'")
echo "submitted ${next_jobid}"
fi

sleep 15

echo "${SLURM_JOB_ID} done"

</pre>

== Software Installed ==

=== IBM PowerAI ===

The PowerAI platform contains popular open machine learning frameworks such as '''Caffe, TensorFlow, and Torch'''. Run the <tt>module avail</tt> command for a complete listing. More information is available at this link: https://developer.ibm.com/linuxonpower/deep-learning-powerai/releases/. Release 4.0 is currently installed.

==== GNU Compilers ====

More recent versions of the GNU Compiler Collection (C/C++/Fortran) are provided in the IBM Advanced Toolchain with enhancements for the POWER8 CPU. To load the newer advance toolchain version use:

Advanced Toolchain V10.0
<pre>
module load gcc/6.3.1
</pre>

Advanced Toolchain V11.0
<pre>
module load gcc/7.2.1
</pre>

More information about the IBM Advanced Toolchain can be found here: [https://developer.ibm.com/linuxonpower/advance-toolchain/ https://developer.ibm.com/linuxonpower/advance-toolchain/]

==== IBM XL Compilers ====

To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run

<pre>
module load xlc/13.1.5
module load xlf/15.1.5
</pre>

IBM XL Compilers are enabled for use with NVIDIA GPUs, including support for OpenMP 4.5 GPU offloading and integration with NVIDIA's nvcc command to compile host-side code for the POWER8 CPU.

Information about the IBM XL Compilers can be found at the following links:

[https://www.ibm.com/support/knowledgecenter/SSXVZZ_13.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++]

[https://www.ibm.com/support/knowledgecenter/SSAT4T_15.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]

==== NVIDIA GPU Driver Version ====

The current NVIDIA driver version is 384.66

==== CUDA ====

The current installed CUDA Tookits is are version 8.0 and version 9.0

<pre>
module load cuda/8.0
</pre>

or

<pre>
module load cuda/9.0
</pre>

The CUDA driver is installed locally, however the CUDA Toolkit is installed in:

<pre>
/usr/local/cuda-8.0
/usr/local/cuda-9.0
</pre>

Note that the <tt>/usr/local/cuda</tt> directory is linked to the <tt>/usr/local/cuda-9.0</tt> directory.

Documentation and API reference information for the CUDA Toolkit can be found here: [http://docs.nvidia.com/cuda/index.html http://docs.nvidia.com/cuda/index.html]

==== OpenMPI ====

Currently OpenMPI has been setup on the 14 nodes connected over EDR Infiniband.

<pre>
$ module load openmpi/2.1.1-gcc-5.4.0
$ module load openmpi/2.1.1-XL-13_15.1.5
</pre>

== Other Software ==

Other software packages can be installed onto the SOSCIP GPU Platform. It is best to try installing new software in your own home directory, which will give you control of the software (e.g. exact version, configuration, installing sub-packages, etc.).

In the following subsections are instructions for installing several common software packages.

=== Anaconda (Python) ===

Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation.

Anaconda can be downloaded from here: [https://www.anaconda.com/download/#linux https://www.anaconda.com/download/#linux]

NOTE: Be sure to download the '''Power8''' installer.

TIP: If you plan to use Tensorflow within Anaconda, download the Python 2.7 version of Anaconda

=== Keras ===

Keras ([https://keras.io/ https://keras.io/]) is a popular high-level deep learning software development framework. It runs on top of other deep-learning frameworks such as TensorFlow.

The easiest way to install Keras is to install Anaconda first, then install Keras by using using the pip command.

Keras uses TensorFlow underneath to run neural network models. Before running code using Keras, be sure to load the PowerAI TensorFlow module and the cuda module.

=== PyTorch ===

PyTorch is the Python implementation of the Torch framework for deep learning.

It is suggested that you use PyTorch within Anaconda.

There is currently no build of PyTorch for POWER8-based systems. You will need to compile it from source.

Obtain the source code from here: [http://pytorch.org/ http://pytorch.org/]

Before building PyTorch, make sure to load cuda by running

<pre>
module load cuda/8.0
</pre>

NOTE: Do not have the gcc modules loaded when building PyTorch. Use the default version of gcc (currently v5.4.0) included with the operating system. Build will fail with later versions of gcc.

== LINKS ==

[https://www.olcf.ornl.gov/kb_articles/summitdev-quickstart/#System_Overview Summit Dev System at ORNL]

== DOCUMENTATION ==

# GPU Cluster Introduction: [[Media:GPU_Training_01.pdf‎|SOSCIP GPU Platform]]

SOSCIP GPU

2018-02-23T15:07:03Z

Wagnerse: /* DOCUMENTATION */

{{Infobox Computer
|image=[[Image:S882lc.png|center|300px|thumb]]
|name=SOSCIP GPU
|installed=September 2017
|operatingsystem= Ubuntu 16.04 le
|loginnode= sgc01
|nnodes= 14x Power 8 with 4x NVIDIA P100
|rampernode=512 GB
|corespernode= 2 x 10core (20 physical, 160 SMT)
|interconnect=Infiniband EDR
|vendorcompilers=xlc/xlf, nvcc
}}

== SOSCIP ==

The SOSCIP GPU Cluster is a Southern Ontario Smart Computing Innovation Platform ([http://soscip.org/ SOSCIP]) resource located at theUniversity of Toronto's SciNet HPC facility. The SOSCIP multi-university/industry consortium is funded by the Ontario Government and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].

== Support Email ==

Please use [mailto:soscip-support@scinet.utoronto.ca <soscip-support@scinet.utoronto.ca>] for SOSCIP GPU specific inquiries.

== Specifications==

The SOSCIP GPU Cluster consists of of 14 IBM Power 822LC "Minsky" Servers each with 2x10core 3.25GHz Power8 CPUs and 512GB Ram. Similar to Power 7, the Power 8 utilizes Simultaneous MultiThreading (SMT), but extends the design to 8 threads per core allowing the 20 physical cores to support up to 160 threads. Each node has 4x NVIDIA Tesla P100 GPUs each with 16GB of RAM with CUDA Capability 6.0 (Pascal) connected using NVlink.

== Access and Login ==

In order to obtain access to the system, you must request access to the SOSCIP GPU Platform. Instructions will have been sent to your sponsoring faculty member via E-mail at the beginning of your SOSCIP project.

Access to the SOSCIP GPU Platform is provided through the BGQ login node, '''<tt> bgqdev.scinet.utoronto.ca </tt>''' using ssh, and from there you can proceed to the GPU development node '''<tt>sgc01-ib0</tt>''' via ssh. Your user name and password is the same as it is for SciNet systems.

== Filesystem ==

The filesystem is shared with the BGQ system. See [https://wiki.scinet.utoronto.ca/wiki/index.php/BGQ#Filesystem here ] for details.

== Job Submission ==

The SOSCIP GPU cluster uses [https://slurm.schedmd.com/ SLURM ] as a job scheduler and jobs are scheduled by node, ie 20 cores and 4 GPUs each. Jobs are submitted from the development node '''<tt>sgc01</tt>'''. The maximum walltime per job is 12 hours (except in the 'long' queue, see below) with up to 8 nodes.

<pre>
$ sbatch myjob.script
</pre>

Where myjob.script is

<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=20 # MPI tasks (needed for srun)
#SBATCH --time=00:10:00 # H:M:S
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node

cd $SLURM_SUBMIT_DIR

hostname
nvidia-smi
</pre>

You can queury job information using

<pre>
squeue
</pre>

To cancel a job use

<pre>
scancel $JOBID
</pre>

=== Longer jobs ===

If your job takes more than 12 hours, the sbatch command will not let you submit your job. There is, however, a way to have jobs up to 24 hours long, by specifying "-p long" as an option (i.e., add <tt>#SBATCH -p long</tt> to your job script). The priority of such jobs may be throttled in the future if we see that the 'long' queue is having a negative efffect on turnover time in the queue.

=== Interactive ===

For an interactive session use

<pre>
salloc --gres=gpu:4
</pre>

=== Automatic Re-submission and Job Dependencies ===

Commonly you may have a job that you know will take longer to run than what is permissible in the queue. As long as your program contains checkpoint or restart capability, you can have one job automatically submit the next. In the following example it is assumed that the program finishes before the time limit requested and then resubmits itself by logging into the development nodes. Job dependencies and a maximum number of job re-submissions are used to ensure sequential operation.

<pre>
#!/bin/bash

#SBATCH --nodes=1
#SBATCH --ntasks=20 # MPI tasks (needed for srun)
#SBATCH --time=00:10:00 # H:M:S
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node

cd $SLURM_SUBMIT_DIR

: ${job_number:="1"} # set job_nubmer to 1 if it is undefined
job_number_max=3

echo "hi from ${SLURM_JOB_ID}"

#RUN JOB HERE

# SUBMIT NEXT JOB
if [[ ${job_number} -lt ${job_number_max} ]]
then
(( job_number++ ))
next_jobid=$(ssh sgc01-ib0 "cd $SLURM_SUBMIT_DIR; /opt/slurm/bin/sbatch --export=job_number=${job_number} -d afterok:${SLURM_JOB_ID} thisscript.sh | awk '{print $4}'")
echo "submitted ${next_jobid}"
fi

sleep 15

echo "${SLURM_JOB_ID} done"

</pre>

== Software Installed ==

=== IBM PowerAI ===

The PowerAI platform contains popular open machine learning frameworks such as '''Caffe, TensorFlow, and Torch'''. Run the <tt>module avail</tt> command for a complete listing. More information is available at this link: https://developer.ibm.com/linuxonpower/deep-learning-powerai/releases/. Release 4.0 is currently installed.

==== GNU Compilers ====

More recent versions of the GNU Compiler Collection (C/C++/Fortran) are provided in the IBM Advanced Toolchain with enhancements for the POWER8 CPU. To load the newer advance toolchain version use:

Advanced Toolchain V10.0
<pre>
module load gcc/6.3.1
</pre>

Advanced Toolchain V11.0
<pre>
module load gcc/7.2.1
</pre>

More information about the IBM Advanced Toolchain can be found here: [https://developer.ibm.com/linuxonpower/advance-toolchain/ https://developer.ibm.com/linuxonpower/advance-toolchain/]

==== IBM XL Compilers ====

To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run

<pre>
module load xlc/13.1.5
module load xlf/15.1.5
</pre>

IBM XL Compilers are enabled for use with NVIDIA GPUs, including support for OpenMP 4.5 GPU offloading and integration with NVIDIA's nvcc command to compile host-side code for the POWER8 CPU.

Information about the IBM XL Compilers can be found at the following links:

[https://www.ibm.com/support/knowledgecenter/SSXVZZ_13.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++]

[https://www.ibm.com/support/knowledgecenter/SSAT4T_15.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]

==== NVIDIA GPU Driver Version ====

The current NVIDIA driver version is 384.66

==== CUDA ====

The current installed CUDA Tookits is are version 8.0 and version 9.0

<pre>
module load cuda/8.0
</pre>

or

<pre>
module load cuda/9.0
</pre>

The CUDA driver is installed locally, however the CUDA Toolkit is installed in:

<pre>
/usr/local/cuda-8.0
/usr/local/cuda-9.0
</pre>

Note that the <tt>/usr/local/cuda</tt> directory is linked to the <tt>/usr/local/cuda-9.0</tt> directory.

Documentation and API reference information for the CUDA Toolkit can be found here: [http://docs.nvidia.com/cuda/index.html http://docs.nvidia.com/cuda/index.html]

==== OpenMPI ====

Currently OpenMPI has been setup on the 14 nodes connected over EDR Infiniband.

<pre>
$ module load openmpi/2.1.1-gcc-5.4.0
$ module load openmpi/2.1.1-XL-13_15.1.5
</pre>

== Other Software ==

Other software packages can be installed onto the SOSCIP GPU Platform. It is best to try installing new software in your own home directory, which will give you control of the software (e.g. exact version, configuration, installing sub-packages, etc.).

In the following subsections are instructions for installing several common software packages.

=== Anaconda (Python) ===

Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation.

Anaconda can be downloaded from here: [https://www.anaconda.com/download/#linux https://www.anaconda.com/download/#linux]

NOTE: Be sure to download the '''Power8''' installer.

TIP: If you plan to use Tensorflow within Anaconda, download the Python 2.7 version of Anaconda

=== Keras ===

Keras ([https://keras.io/ https://keras.io/]) is a popular high-level deep learning software development framework. It runs on top of other deep-learning frameworks such as TensorFlow.

The easiest way to install Keras is to install Anaconda first, then install Keras by using using the pip command.

Keras uses TensorFlow underneath to run neural network models. Before running code using Keras, be sure to load the PowerAI TensorFlow module and the cuda module.

=== PyTorch ===

PyTorch is the Python implementation of the Torch framework for deep learning.

It is suggested that you use PyTorch within Anaconda.

There is currently no build of PyTorch for POWER8-based systems. You will need to compile it from source.

Obtain the source code from here: [http://pytorch.org/ http://pytorch.org/]

Before building PyTorch, make sure to load cuda by running

<pre>
module load cuda/8.0
</pre>

NOTE: Do not have the gcc modules loaded when building PyTorch. Use the default version of gcc (currently v5.4.0) included with the operating system. Build will fail with later versions of gcc.

== LINKS ==

[https://www.olcf.ornl.gov/kb_articles/summitdev-quickstart/#System_Overview Summit Dev System at ORNL]

== DOCUMENTATION ==

# GPU Cluster Introduction: [[Media:GPU_Training_01.pdf‎|SOSCIP GPU Platform]]

File:GPU Training 01.pdf

2018-02-23T15:06:38Z

Wagnerse:

SOSCIP GPU

2018-01-18T14:23:30Z

Wagnerse: /* Compile/Devel/Test */

{{Infobox Computer
|image=[[Image:S882lc.png|center|300px|thumb]]
|name=SOSCIP GPU
|installed=September 2017
|operatingsystem= Ubuntu 16.04 le
|loginnode= sgc01
|nnodes= 14x Power 8 with 4x NVIDIA P100
|rampernode=512 GB
|corespernode= 2 x 10core (20 physical, 160 SMT)
|interconnect=Infiniband EDR
|vendorcompilers=xlc/xlf, nvcc
}}

== SOSCIP ==

The SOSCIP GPU Cluster is a Southern Ontario Smart Computing Innovation Platform ([http://soscip.org/ SOSCIP]) resource located at theUniversity of Toronto's SciNet HPC facility. The SOSCIP multi-university/industry consortium is funded by the Ontario Government and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].

== Support Email ==

Please use [mailto:soscip-support@scinet.utoronto.ca <soscip-support@scinet.utoronto.ca>] for SOSCIP GPU specific inquiries.

== Specifications==

The SOSCIP GPU Cluster consists of of 14 IBM Power 822LC "Minsky" Servers each with 2x10core 3.25GHz Power8 CPUs and 512GB Ram. Similar to Power 7, the Power 8 utilizes Simultaneous MultiThreading (SMT), but extends the design to 8 threads per core allowing the 20 physical cores to support up to 160 threads. Each node has 4x NVIDIA Tesla P100 GPUs each with 16GB of RAM with CUDA Capability 6.0 (Pascal) connected using NVlink.

== Access and Login ==

In order to obtain access to the system, you must request access to the SOSCIP GPU Platform. Instructions will have been sent to your sponsoring faculty member via E-mail at the beginning of your SOSCIP project.

Access to the SOSCIP GPU Platform is provided through the BGQ login node, '''<tt> bgqdev.scinet.utoronto.ca </tt>''' using ssh, and from there you can proceed to the GPU development node '''<tt>sgc01-ib0</tt>''' via ssh. Your user name and password is the same as it is for SciNet systems.

== Filesystem ==

The filesystem is shared with the BGQ system. See [https://wiki.scinet.utoronto.ca/wiki/index.php/BGQ#Filesystem here ] for details.

== Job Submission ==

The SOSCIP GPU cluster uses [https://slurm.schedmd.com/ SLURM ] as a job scheduler and jobs are scheduled by node, ie 20 cores and 4 GPUs each. Jobs are submitted from the development node '''<tt>sgc01</tt>'''. The maximum walltime per job is 12 hours (except in the 'long' queue, see below) with up to 8 nodes.

<pre>
$ sbatch myjob.script
</pre>

Where myjob.script is

<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=20 # MPI tasks (needed for srun)
#SBATCH --time=00:10:00 # H:M:S
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node

cd $SLURM_SUBMIT_DIR

hostname
nvidia-smi
</pre>

You can queury job information using

<pre>
squeue
</pre>

To cancel a job use

<pre>
scancel $JOBID
</pre>

=== Longer jobs ===

If your job takes more than 12 hours, the sbatch command will not let you submit your job. There is, however, a way to have jobs up to 24 hours long, by specifying "-p long" as an option (i.e., add <tt>#SBATCH -p long</tt> to your job script). The priority of such jobs may be throttled in the future if we see that the 'long' queue is having a negative efffect on turnover time in the queue.

=== Interactive ===

For an interactive session use

<pre>
salloc --gres=gpu:4
</pre>

=== Automatic Re-submission and Job Dependencies ===

Commonly you may have a job that you know will take longer to run than what is permissible in the queue. As long as your program contains checkpoint or restart capability, you can have one job automatically submit the next. In the following example it is assumed that the program finishes before the time limit requested and then resubmits itself by logging into the development nodes. Job dependencies and a maximum number of job re-submissions are used to ensure sequential operation.

<pre>
#!/bin/bash

#SBATCH --nodes=1
#SBATCH --ntasks=20 # MPI tasks (needed for srun)
#SBATCH --time=00:10:00 # H:M:S
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node

cd $SLURM_SUBMIT_DIR

: ${job_number:="1"} # set job_nubmer to 1 if it is undefined
job_number_max=3

echo "hi from ${SLURM_JOB_ID}"

#RUN JOB HERE

# SUBMIT NEXT JOB
if [[ ${job_number} -lt ${job_number_max} ]]
then
(( job_number++ ))
next_jobid=$(ssh sgc01-ib0 "cd $SLURM_SUBMIT_DIR; /opt/slurm/bin/sbatch --export=job_number=${job_number} -d afterok:${SLURM_JOB_ID} thisscript.sh | awk '{print $4}'")
echo "submitted ${next_jobid}"
fi

sleep 15

echo "${SLURM_JOB_ID} done"

</pre>

== Software Installed ==

=== IBM PowerAI ===

The PowerAI platform contains popular open machine learning frameworks such as '''Caffe, TensorFlow, and Torch'''. Run the <tt>module avail</tt> command for a complete listing. More information is available at this link: https://developer.ibm.com/linuxonpower/deep-learning-powerai/releases/. Release 4.0 is currently installed.

==== GNU Compilers ====

More recent versions of the GNU Compiler Collection (C/C++/Fortran) are provided in the IBM Advanced Toolchain with enhancements for the POWER8 CPU. To load the newer advance toolchain version use:

Advanced Toolchain V10.0
<pre>
module load gcc/6.3.1
</pre>

Advanced Toolchain V11.0
<pre>
module load gcc/7.2.1
</pre>

More information about the IBM Advanced Toolchain can be found here: [https://developer.ibm.com/linuxonpower/advance-toolchain/ https://developer.ibm.com/linuxonpower/advance-toolchain/]

==== IBM XL Compilers ====

To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run

<pre>
module load xlc/13.1.5
module load xlf/15.1.5
</pre>

IBM XL Compilers are enabled for use with NVIDIA GPUs, including support for OpenMP 4.5 GPU offloading and integration with NVIDIA's nvcc command to compile host-side code for the POWER8 CPU.

Information about the IBM XL Compilers can be found at the following links:

[https://www.ibm.com/support/knowledgecenter/SSXVZZ_13.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++]

[https://www.ibm.com/support/knowledgecenter/SSAT4T_15.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]

==== NVIDIA GPU Driver Version ====

The current NVIDIA driver version is 384.66

==== CUDA ====

The current installed CUDA Tookits is are version 8.0 and version 9.0

<pre>
module load cuda/8.0
</pre>

or

<pre>
module load cuda/9.0
</pre>

The CUDA driver is installed locally, however the CUDA Toolkit is installed in:

<pre>
/usr/local/cuda-8.0
/usr/local/cuda-9.0
</pre>

Note that the <tt>/usr/local/cuda</tt> directory is linked to the <tt>/usr/local/cuda-9.0</tt> directory.

Documentation and API reference information for the CUDA Toolkit can be found here: [http://docs.nvidia.com/cuda/index.html http://docs.nvidia.com/cuda/index.html]

==== OpenMPI ====

Currently OpenMPI has been setup on the 14 nodes connected over EDR Infiniband.

<pre>
$ module load openmpi/2.1.1-gcc-5.4.0
$ module load openmpi/2.1.1-XL-13_15.1.5
</pre>

== Other Software ==

Other software packages can be installed onto the SOSCIP GPU Platform. It is best to try installing new software in your own home directory, which will give you control of the software (e.g. exact version, configuration, installing sub-packages, etc.).

In the following subsections are instructions for installing several common software packages.

=== Anaconda (Python) ===

Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation.

Anaconda can be downloaded from here: [https://www.anaconda.com/download/#linux https://www.anaconda.com/download/#linux]

NOTE: Be sure to download the '''Power8''' installer.

TIP: If you plan to use Tensorflow within Anaconda, download the Python 2.7 version of Anaconda

=== Keras ===

Keras ([https://keras.io/ https://keras.io/]) is a popular high-level deep learning software development framework. It runs on top of other deep-learning frameworks such as TensorFlow.

The easiest way to install Keras is to install Anaconda first, then install Keras by using using the pip command.

Keras uses TensorFlow underneath to run neural network models. Before running code using Keras, be sure to load the PowerAI TensorFlow module and the cuda module.

=== PyTorch ===

PyTorch is the Python implementation of the Torch framework for deep learning.

It is suggested that you use PyTorch within Anaconda.

There is currently no build of PyTorch for POWER8-based systems. You will need to compile it from source.

Obtain the source code from here: [http://pytorch.org/ http://pytorch.org/]

Before building PyTorch, make sure to load cuda by running

<pre>
module load cuda/8.0
</pre>

NOTE: Do not have the gcc modules loaded when building PyTorch. Use the default version of gcc (currently v5.4.0) included with the operating system. Build will fail with later versions of gcc.

== LINKS ==

[https://www.olcf.ornl.gov/kb_articles/summitdev-quickstart/#System_Overview Summit Dev System at ORNL]

SOSCIP GPU

2017-12-20T14:32:31Z

Wagnerse: /* CUDA */

{{Infobox Computer
|image=[[Image:S882lc.png|center|300px|thumb]]
|name=SOSCIP GPU
|installed=September 2017
|operatingsystem= Ubuntu 16.04 le
|loginnode= sgc01
|nnodes= 14x Power 8 with 4x NVIDIA P100
|rampernode=512 GB
|corespernode= 2 x 10core (20 physical, 160 SMT)
|interconnect=Infiniband EDR
|vendorcompilers=xlc/xlf, nvcc
}}

== SOSCIP ==

The SOSCIP GPU Cluster is a Southern Ontario Smart Computing Innovation Platform ([http://soscip.org/ SOSCIP]) resource located at theUniversity of Toronto's SciNet HPC facility. The SOSCIP multi-university/industry consortium is funded by the Ontario Government and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].

== Support Email ==

Please use [mailto:soscip-support@scinet.utoronto.ca <soscip-support@scinet.utoronto.ca>] for SOSCIP GPU specific inquiries.

== Specifications==

The SOSCIP GPU Cluster consists of of 14 IBM Power 822LC "Minsky" Servers each with 2x10core 3.25GHz Power8 CPUs and 512GB Ram. Similar to Power 7, the Power 8 utilizes Simultaneous MultiThreading (SMT), but extends the design to 8 threads per core allowing the 20 physical cores to support up to 160 threads. Each node has 4x NVIDIA Tesla P100 GPUs each with 16GB of RAM with CUDA Capability 6.0 (Pascal) connected using NVlink.

== Compile/Devel/Test ==

Access is provided through the BGQ login node, '''<tt> bgqdev.scinet.utoronto.ca </tt>''' using ssh, and from there you can proceed to the GPU development node '''<tt>sgc01-ib0</tt>'''.

== Filesystem ==

The filesystem is shared with the BGQ system. See [https://wiki.scinet.utoronto.ca/wiki/index.php/BGQ#Filesystem here ] for details.

== Job Submission ==

The SOSCIP GPU cluster uses [https://slurm.schedmd.com/ SLURM ] as a job scheduler and jobs are scheduled by node, ie 20 cores and 4 GPUs each. Jobs are submitted from the development node '''<tt>sgc01</tt>'''. The maximum walltime per job is 12 hours (except in the 'long' queue, see below) with up to 8 nodes.

<pre>
$ sbatch myjob.script
</pre>

Where myjob.script is

<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=20 # MPI tasks (needed for srun)
#SBATCH --time=00:10:00 # H:M:S
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node

cd $SLURM_SUBMIT_DIR

hostname
nvidia-smi
</pre>

You can queury job information using

<pre>
squeue
</pre>

To cancel a job use

<pre>
scancel $JOBID
</pre>

=== Longer jobs ===

If your job takes more than 12 hours, the sbatch command will not let you submit your job. There is, however, a way to have jobs up to 24 hours long, by specifying "-p long" as an option (i.e., add <tt>#SBATCH -p long</tt> to your job script). The priority of such jobs may be throttled in the future if we see that the 'long' queue is having a negative efffect on turnover time in the queue.

=== Interactive ===

For an interactive session use

<pre>
salloc --gres=gpu:4
</pre>

=== Automatic Re-submission and Job Dependencies ===

Commonly you may have a job that you know will take longer to run than what is permissible in the queue. As long as your program contains checkpoint or restart capability, you can have one job automatically submit the next. In the following example it is assumed that the program finishes before the time limit requested and then resubmits itself by logging into the development nodes. Job dependencies and a maximum number of job re-submissions are used to ensure sequential operation.

<pre>
#!/bin/bash

#SBATCH --nodes=1
#SBATCH --ntasks=20 # MPI tasks (needed for srun)
#SBATCH --time=00:10:00 # H:M:S
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node

cd $SLURM_SUBMIT_DIR

: ${job_number:="1"} # set job_nubmer to 1 if it is undefined
job_number_max=3

echo "hi from ${SLURM_JOB_ID}"

#RUN JOB HERE

# SUBMIT NEXT JOB
if [[ ${job_number} -lt ${job_number_max} ]]
then
(( job_number++ ))
next_jobid=$(ssh sgc01-ib0 "cd $SLURM_SUBMIT_DIR; /opt/slurm/bin/sbatch --export=job_number=${job_number} -d afterok:${SLURM_JOB_ID} thisscript.sh | awk '{print $4}'")
echo "submitted ${next_jobid}"
fi

sleep 15

echo "${SLURM_JOB_ID} done"

</pre>

== Software Installed ==

=== IBM PowerAI ===

The PowerAI platform contains popular open machine learning frameworks such as '''Caffe, TensorFlow, and Torch'''. Run the <tt>module avail</tt> command for a complete listing. More information is available at this link: https://developer.ibm.com/linuxonpower/deep-learning-powerai/releases/. Release 4.0 is currently installed.

==== GNU Compilers ====

More recent versions of the GNU Compiler Collection (C/C++/Fortran) are provided in the IBM Advanced Toolchain with enhancements for the POWER8 CPU. To load the newer advance toolchain version use:

Advanced Toolchain V10.0
<pre>
module load gcc/6.3.1
</pre>

Advanced Toolchain V11.0
<pre>
module load gcc/7.2.1
</pre>

More information about the IBM Advanced Toolchain can be found here: [https://developer.ibm.com/linuxonpower/advance-toolchain/ https://developer.ibm.com/linuxonpower/advance-toolchain/]

==== IBM XL Compilers ====

To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run

<pre>
module load xlc/13.1.5
module load xlf/15.1.5
</pre>

IBM XL Compilers are enabled for use with NVIDIA GPUs, including support for OpenMP 4.5 GPU offloading and integration with NVIDIA's nvcc command to compile host-side code for the POWER8 CPU.

Information about the IBM XL Compilers can be found at the following links:

[https://www.ibm.com/support/knowledgecenter/SSXVZZ_13.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++]

[https://www.ibm.com/support/knowledgecenter/SSAT4T_15.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]

==== NVIDIA GPU Driver Version ====

The current NVIDIA driver version is 384.66

==== CUDA ====

The current installed CUDA Tookits is are version 8.0 and version 9.0

<pre>
module load cuda/8.0
</pre>

or

<pre>
module load cuda/9.0
</pre>

The CUDA driver is installed locally, however the CUDA Toolkit is installed in:

<pre>
/usr/local/cuda-8.0
/usr/local/cuda-9.0
</pre>

Note that the <tt>/usr/local/cuda</tt> directory is linked to the <tt>/usr/local/cuda-9.0</tt> directory.

Documentation and API reference information for the CUDA Toolkit can be found here: [http://docs.nvidia.com/cuda/index.html http://docs.nvidia.com/cuda/index.html]

==== OpenMPI ====

Currently OpenMPI has been setup on the 14 nodes connected over EDR Infiniband.

<pre>
$ module load openmpi/2.1.1-gcc-5.4.0
$ module load openmpi/2.1.1-XL-13_15.1.5
</pre>

== Other Software ==

Other software packages can be installed onto the SOSCIP GPU Platform. It is best to try installing new software in your own home directory, which will give you control of the software (e.g. exact version, configuration, installing sub-packages, etc.).

In the following subsections are instructions for installing several common software packages.

=== Anaconda (Python) ===

Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation.

Anaconda can be downloaded from here: [https://www.anaconda.com/download/#linux https://www.anaconda.com/download/#linux]

NOTE: Be sure to download the '''Power8''' installer.

TIP: If you plan to use Tensorflow within Anaconda, download the Python 2.7 version of Anaconda

=== Keras ===

Keras ([https://keras.io/ https://keras.io/]) is a popular high-level deep learning software development framework. It runs on top of other deep-learning frameworks such as TensorFlow.

The easiest way to install Keras is to install Anaconda first, then install Keras by using using the pip command.

Keras uses TensorFlow underneath to run neural network models. Before running code using Keras, be sure to load the PowerAI TensorFlow module and the cuda module.

=== PyTorch ===

PyTorch is the Python implementation of the Torch framework for deep learning.

It is suggested that you use PyTorch within Anaconda.

There is currently no build of PyTorch for POWER8-based systems. You will need to compile it from source.

Obtain the source code from here: [http://pytorch.org/ http://pytorch.org/]

Before building PyTorch, make sure to load cuda by running

<pre>
module load cuda/8.0
</pre>

NOTE: Do not have the gcc modules loaded when building PyTorch. Use the default version of gcc (currently v5.4.0) included with the operating system. Build will fail with later versions of gcc.

SOSCIP GPU

2017-12-06T15:45:03Z

Wagnerse: /* IBM XL Compilers */

{{Infobox Computer
|image=[[Image:S882lc.png|center|300px|thumb]]
|name=SOSCIP GPU
|installed=September 2017
|operatingsystem= Ubuntu 16.04 le
|loginnode= sgc01
|nnodes= 14x Power 8 with 4x NVIDIA P100
|rampernode=512 GB
|corespernode= 2 x 10core (20 physical, 160 SMT)
|interconnect=Infiniband EDR
|vendorcompilers=xlc/xlf, nvcc
}}

== SOSCIP ==

The SOSCIP GPU Cluster is a Southern Ontario Smart Computing Innovation Platform ([http://soscip.org/ SOSCIP]) resource located at theUniversity of Toronto's SciNet HPC facility. The SOSCIP multi-university/industry consortium is funded by the Ontario Government and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].

== Support Email ==

Please use [mailto:soscip-support@scinet.utoronto.ca <soscip-support@scinet.utoronto.ca>] for SOSCIP GPU specific inquiries.

== Specifications==

The SOSCIP GPU Cluster consists of of 14 IBM Power 822LC "Minsky" Servers each with 2x10core 3.25GHz Power8 CPUs and 512GB Ram. Similar to Power 7, the Power 8 utilizes Simultaneous MultiThreading (SMT), but extends the design to 8 threads per core allowing the 20 physical cores to support up to 160 threads. Each node has 4x NVIDIA Tesla P100 GPUs each with 16GB of RAM with CUDA Capability 6.0 (Pascal) connected using NVlink.

== Compile/Devel/Test ==

Access is provided through the BGQ login node, '''<tt> bgqdev.scinet.utoronto.ca </tt>''' using ssh, and from there you can proceed to the GPU development node '''<tt>sgc01-ib0</tt>'''.

== Filesystem ==

The filesystem is shared with the BGQ system. See [https://wiki.scinet.utoronto.ca/wiki/index.php/BGQ#Filesystem here ] for details.

== Job Submission ==

The SOSCIP GPU cluster uses [https://slurm.schedmd.com/ SLURM ] as a job scheduler and jobs are scheduled by node, ie 20 cores and 4 GPUs each. Jobs are submitted from the development node '''<tt>sgc01</tt>'''. The maximum walltime per job is 12 hours (except in the 'long' queue, see below) with up to 8 nodes.

<pre>
$ sbatch myjob.script
</pre>

Where myjob.script is

<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=20 # MPI tasks (needed for srun)
#SBATCH --time=00:10:00 # H:M:S
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node

cd $SLURM_SUBMIT_DIR

hostname
nvidia-smi
</pre>

You can queury job information using

<pre>
squeue
</pre>

To cancel a job use

<pre>
scancel $JOBID
</pre>

=== Longer jobs ===

If your job takes more than 12 hours, the sbatch command will not let you submit your job. There is, however, a way to have jobs up to 24 hours long, by specifying "-p long" as an option (i.e., add <tt>#SBATCH -p long</tt> to your job script). The priority of such jobs may be throttled in the future if we see that the 'long' queue is having a negative efffect on turnover time in the queue.

=== Interactive ===

For an interactive session use

<pre>
salloc --gres=gpu:4
</pre>

=== Automatic Re-submission and Job Dependencies ===

Commonly you may have a job that you know will take longer to run than what is permissible in the queue. As long as your program contains checkpoint or restart capability, you can have one job automatically submit the next. In the following example it is assumed that the program finishes before the time limit requested and then resubmits itself by logging into the development nodes. Job dependencies and a maximum number of job re-submissions are used to ensure sequential operation.

<pre>
#!/bin/bash

#SBATCH --nodes=1
#SBATCH --ntasks=20 # MPI tasks (needed for srun)
#SBATCH --time=00:10:00 # H:M:S
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node

cd $SLURM_SUBMIT_DIR

: ${job_number:="1"} # set job_nubmer to 1 if it is undefined
job_number_max=3

echo "hi from ${SLURM_JOB_ID}"

#RUN JOB HERE

# SUBMIT NEXT JOB
if [[ ${job_number} -lt ${job_number_max} ]]
then
(( job_number++ ))
next_jobid=$(ssh sgc01-ib0 "cd $SLURM_SUBMIT_DIR; /opt/slurm/bin/sbatch --export=job_number=${job_number} -d afterok:${SLURM_JOB_ID} thisscript.sh | awk '{print $4}'")
echo "submitted ${next_jobid}"
fi

sleep 15

echo "${SLURM_JOB_ID} done"

</pre>

== Software Installed ==

=== IBM PowerAI ===

The PowerAI platform contains popular open machine learning frameworks such as '''Caffe, TensorFlow, and Torch'''. Run the <tt>module avail</tt> command for a complete listing. More information is available at this link: https://developer.ibm.com/linuxonpower/deep-learning-powerai/releases/. Release 4.0 is currently installed.

==== GNU Compilers ====

More recent versions of the GNU Compiler Collection (C/C++/Fortran) are provided in the IBM Advanced Toolchain with enhancements for the POWER8 CPU. To load the newer advance toolchain version use:

Advanced Toolchain V10.0
<pre>
module load gcc/6.3.1
</pre>

Advanced Toolchain V11.0
<pre>
module load gcc/7.2.1
</pre>

More information about the IBM Advanced Toolchain can be found here: [https://developer.ibm.com/linuxonpower/advance-toolchain/ https://developer.ibm.com/linuxonpower/advance-toolchain/]

==== IBM XL Compilers ====

To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run

<pre>
module load xlc/13.1.5
module load xlf/15.1.5
</pre>

IBM XL Compilers are enabled for use with NVIDIA GPUs, including support for OpenMP 4.5 GPU offloading and integration with NVIDIA's nvcc command to compile host-side code for the POWER8 CPU.

Information about the IBM XL Compilers can be found at the following links:

[https://www.ibm.com/support/knowledgecenter/SSXVZZ_13.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++]

[https://www.ibm.com/support/knowledgecenter/SSAT4T_15.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]

==== NVIDIA GPU Driver Version ====

The current NVIDIA driver version is 384.66

==== CUDA ====

The current installed CUDA Tookits is are version 8.0 and version 9.0

<pre>
module load cuda/8.0
</pre>

or

<pre>
module load cuda/9.0
</pre>

The CUDA driver is installed locally, however the CUDA Toolkit is installed in:

<pre>
/usr/local/cuda-8.0
/usr/local/cuda-9.0
</pre>

Note that the <tt>/usr/local/cuda</tt> directory is linked to the <tt>/usr/local/cuda-9.0</tt> directory.

==== OpenMPI ====

Currently OpenMPI has been setup on the 14 nodes connected over EDR Infiniband.

<pre>
$ module load openmpi/2.1.1-gcc-5.4.0
$ module load openmpi/2.1.1-XL-13_15.1.5
</pre>

== Other Software ==

Other software packages can be installed onto the SOSCIP GPU Platform. It is best to try installing new software in your own home directory, which will give you control of the software (e.g. exact version, configuration, installing sub-packages, etc.).

In the following subsections are instructions for installing several common software packages.

=== Anaconda (Python) ===

Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation.

Anaconda can be downloaded from here: [https://www.anaconda.com/download/#linux https://www.anaconda.com/download/#linux]

NOTE: Be sure to download the '''Power8''' installer.

TIP: If you plan to use Tensorflow within Anaconda, download the Python 2.7 version of Anaconda

=== Keras ===

Keras ([https://keras.io/ https://keras.io/]) is a popular high-level deep learning software development framework. It runs on top of other deep-learning frameworks such as TensorFlow.

The easiest way to install Keras is to install Anaconda first, then install Keras by using using the pip command.

Keras uses TensorFlow underneath to run neural network models. Before running code using Keras, be sure to load the PowerAI TensorFlow module and the cuda module.

=== PyTorch ===

PyTorch is the Python implementation of the Torch framework for deep learning.

It is suggested that you use PyTorch within Anaconda.

There is currently no build of PyTorch for POWER8-based systems. You will need to compile it from source.

Obtain the source code from here: [http://pytorch.org/ http://pytorch.org/]

Before building PyTorch, make sure to load cuda by running

<pre>
module load cuda/8.0
</pre>

NOTE: Do not have the gcc modules loaded when building PyTorch. Use the default version of gcc (currently v5.4.0) included with the operating system. Build will fail with later versions of gcc.

SOSCIP GPU

2017-12-06T15:39:35Z

Wagnerse: /* IBM PowerAI */

{{Infobox Computer
|image=[[Image:S882lc.png|center|300px|thumb]]
|name=SOSCIP GPU
|installed=September 2017
|operatingsystem= Ubuntu 16.04 le
|loginnode= sgc01
|nnodes= 14x Power 8 with 4x NVIDIA P100
|rampernode=512 GB
|corespernode= 2 x 10core (20 physical, 160 SMT)
|interconnect=Infiniband EDR
|vendorcompilers=xlc/xlf, nvcc
}}

== SOSCIP ==

The SOSCIP GPU Cluster is a Southern Ontario Smart Computing Innovation Platform ([http://soscip.org/ SOSCIP]) resource located at theUniversity of Toronto's SciNet HPC facility. The SOSCIP multi-university/industry consortium is funded by the Ontario Government and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].

== Support Email ==

Please use [mailto:soscip-support@scinet.utoronto.ca <soscip-support@scinet.utoronto.ca>] for SOSCIP GPU specific inquiries.

== Specifications==

The SOSCIP GPU Cluster consists of of 14 IBM Power 822LC "Minsky" Servers each with 2x10core 3.25GHz Power8 CPUs and 512GB Ram. Similar to Power 7, the Power 8 utilizes Simultaneous MultiThreading (SMT), but extends the design to 8 threads per core allowing the 20 physical cores to support up to 160 threads. Each node has 4x NVIDIA Tesla P100 GPUs each with 16GB of RAM with CUDA Capability 6.0 (Pascal) connected using NVlink.

== Compile/Devel/Test ==

Access is provided through the BGQ login node, '''<tt> bgqdev.scinet.utoronto.ca </tt>''' using ssh, and from there you can proceed to the GPU development node '''<tt>sgc01-ib0</tt>'''.

== Filesystem ==

The filesystem is shared with the BGQ system. See [https://wiki.scinet.utoronto.ca/wiki/index.php/BGQ#Filesystem here ] for details.

== Job Submission ==

The SOSCIP GPU cluster uses [https://slurm.schedmd.com/ SLURM ] as a job scheduler and jobs are scheduled by node, ie 20 cores and 4 GPUs each. Jobs are submitted from the development node '''<tt>sgc01</tt>'''. The maximum walltime per job is 12 hours (except in the 'long' queue, see below) with up to 8 nodes.

<pre>
$ sbatch myjob.script
</pre>

Where myjob.script is

<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=20 # MPI tasks (needed for srun)
#SBATCH --time=00:10:00 # H:M:S
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node

cd $SLURM_SUBMIT_DIR

hostname
nvidia-smi
</pre>

You can queury job information using

<pre>
squeue
</pre>

To cancel a job use

<pre>
scancel $JOBID
</pre>

=== Longer jobs ===

If your job takes more than 12 hours, the sbatch command will not let you submit your job. There is, however, a way to have jobs up to 24 hours long, by specifying "-p long" as an option (i.e., add <tt>#SBATCH -p long</tt> to your job script). The priority of such jobs may be throttled in the future if we see that the 'long' queue is having a negative efffect on turnover time in the queue.

=== Interactive ===

For an interactive session use

<pre>
salloc --gres=gpu:4
</pre>

=== Automatic Re-submission and Job Dependencies ===

Commonly you may have a job that you know will take longer to run than what is permissible in the queue. As long as your program contains checkpoint or restart capability, you can have one job automatically submit the next. In the following example it is assumed that the program finishes before the time limit requested and then resubmits itself by logging into the development nodes. Job dependencies and a maximum number of job re-submissions are used to ensure sequential operation.

<pre>
#!/bin/bash

#SBATCH --nodes=1
#SBATCH --ntasks=20 # MPI tasks (needed for srun)
#SBATCH --time=00:10:00 # H:M:S
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node

cd $SLURM_SUBMIT_DIR

: ${job_number:="1"} # set job_nubmer to 1 if it is undefined
job_number_max=3

echo "hi from ${SLURM_JOB_ID}"

#RUN JOB HERE

# SUBMIT NEXT JOB
if [[ ${job_number} -lt ${job_number_max} ]]
then
(( job_number++ ))
next_jobid=$(ssh sgc01-ib0 "cd $SLURM_SUBMIT_DIR; /opt/slurm/bin/sbatch --export=job_number=${job_number} -d afterok:${SLURM_JOB_ID} thisscript.sh | awk '{print $4}'")
echo "submitted ${next_jobid}"
fi

sleep 15

echo "${SLURM_JOB_ID} done"

</pre>

== Software Installed ==

=== IBM PowerAI ===

The PowerAI platform contains popular open machine learning frameworks such as '''Caffe, TensorFlow, and Torch'''. Run the <tt>module avail</tt> command for a complete listing. More information is available at this link: https://developer.ibm.com/linuxonpower/deep-learning-powerai/releases/. Release 4.0 is currently installed.

==== GNU Compilers ====

More recent versions of the GNU Compiler Collection (C/C++/Fortran) are provided in the IBM Advanced Toolchain with enhancements for the POWER8 CPU. To load the newer advance toolchain version use:

Advanced Toolchain V10.0
<pre>
module load gcc/6.3.1
</pre>

Advanced Toolchain V11.0
<pre>
module load gcc/7.2.1
</pre>

More information about the IBM Advanced Toolchain can be found here: [https://developer.ibm.com/linuxonpower/advance-toolchain/ https://developer.ibm.com/linuxonpower/advance-toolchain/]

==== IBM XL Compilers ====

To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run

<pre>
module load xlc/13.1.5
module load xlf/15.1.5
</pre>

Information about the IBM XL Compilers can be found at the following links:

[https://www.ibm.com/support/knowledgecenter/SSXVZZ_13.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++]

[https://www.ibm.com/support/knowledgecenter/SSAT4T_15.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]

==== NVIDIA GPU Driver Version ====

The current NVIDIA driver version is 384.66

==== CUDA ====

The current installed CUDA Tookits is are version 8.0 and version 9.0

<pre>
module load cuda/8.0
</pre>

or

<pre>
module load cuda/9.0
</pre>

The CUDA driver is installed locally, however the CUDA Toolkit is installed in:

<pre>
/usr/local/cuda-8.0
/usr/local/cuda-9.0
</pre>

Note that the <tt>/usr/local/cuda</tt> directory is linked to the <tt>/usr/local/cuda-9.0</tt> directory.

==== OpenMPI ====

Currently OpenMPI has been setup on the 14 nodes connected over EDR Infiniband.

<pre>
$ module load openmpi/2.1.1-gcc-5.4.0
$ module load openmpi/2.1.1-XL-13_15.1.5
</pre>

== Other Software ==

Other software packages can be installed onto the SOSCIP GPU Platform. It is best to try installing new software in your own home directory, which will give you control of the software (e.g. exact version, configuration, installing sub-packages, etc.).

In the following subsections are instructions for installing several common software packages.

=== Anaconda (Python) ===

Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation.

Anaconda can be downloaded from here: [https://www.anaconda.com/download/#linux https://www.anaconda.com/download/#linux]

NOTE: Be sure to download the '''Power8''' installer.

TIP: If you plan to use Tensorflow within Anaconda, download the Python 2.7 version of Anaconda

=== Keras ===

Keras ([https://keras.io/ https://keras.io/]) is a popular high-level deep learning software development framework. It runs on top of other deep-learning frameworks such as TensorFlow.

The easiest way to install Keras is to install Anaconda first, then install Keras by using using the pip command.

Keras uses TensorFlow underneath to run neural network models. Before running code using Keras, be sure to load the PowerAI TensorFlow module and the cuda module.

=== PyTorch ===

PyTorch is the Python implementation of the Torch framework for deep learning.

It is suggested that you use PyTorch within Anaconda.

There is currently no build of PyTorch for POWER8-based systems. You will need to compile it from source.

Obtain the source code from here: [http://pytorch.org/ http://pytorch.org/]

Before building PyTorch, make sure to load cuda by running

<pre>
module load cuda/8.0
</pre>

NOTE: Do not have the gcc modules loaded when building PyTorch. Use the default version of gcc (currently v5.4.0) included with the operating system. Build will fail with later versions of gcc.

SOSCIP GPU

2017-12-06T15:38:59Z

Wagnerse: /* Keras */

{{Infobox Computer
|image=[[Image:S882lc.png|center|300px|thumb]]
|name=SOSCIP GPU
|installed=September 2017
|operatingsystem= Ubuntu 16.04 le
|loginnode= sgc01
|nnodes= 14x Power 8 with 4x NVIDIA P100
|rampernode=512 GB
|corespernode= 2 x 10core (20 physical, 160 SMT)
|interconnect=Infiniband EDR
|vendorcompilers=xlc/xlf, nvcc
}}

== SOSCIP ==

The SOSCIP GPU Cluster is a Southern Ontario Smart Computing Innovation Platform ([http://soscip.org/ SOSCIP]) resource located at theUniversity of Toronto's SciNet HPC facility. The SOSCIP multi-university/industry consortium is funded by the Ontario Government and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].

== Support Email ==

Please use [mailto:soscip-support@scinet.utoronto.ca <soscip-support@scinet.utoronto.ca>] for SOSCIP GPU specific inquiries.

== Specifications==

The SOSCIP GPU Cluster consists of of 14 IBM Power 822LC "Minsky" Servers each with 2x10core 3.25GHz Power8 CPUs and 512GB Ram. Similar to Power 7, the Power 8 utilizes Simultaneous MultiThreading (SMT), but extends the design to 8 threads per core allowing the 20 physical cores to support up to 160 threads. Each node has 4x NVIDIA Tesla P100 GPUs each with 16GB of RAM with CUDA Capability 6.0 (Pascal) connected using NVlink.

== Compile/Devel/Test ==

Access is provided through the BGQ login node, '''<tt> bgqdev.scinet.utoronto.ca </tt>''' using ssh, and from there you can proceed to the GPU development node '''<tt>sgc01-ib0</tt>'''.

== Filesystem ==

The filesystem is shared with the BGQ system. See [https://wiki.scinet.utoronto.ca/wiki/index.php/BGQ#Filesystem here ] for details.

== Job Submission ==

The SOSCIP GPU cluster uses [https://slurm.schedmd.com/ SLURM ] as a job scheduler and jobs are scheduled by node, ie 20 cores and 4 GPUs each. Jobs are submitted from the development node '''<tt>sgc01</tt>'''. The maximum walltime per job is 12 hours (except in the 'long' queue, see below) with up to 8 nodes.

<pre>
$ sbatch myjob.script
</pre>

Where myjob.script is

<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=20 # MPI tasks (needed for srun)
#SBATCH --time=00:10:00 # H:M:S
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node

cd $SLURM_SUBMIT_DIR

hostname
nvidia-smi
</pre>

You can queury job information using

<pre>
squeue
</pre>

To cancel a job use

<pre>
scancel $JOBID
</pre>

=== Longer jobs ===

If your job takes more than 12 hours, the sbatch command will not let you submit your job. There is, however, a way to have jobs up to 24 hours long, by specifying "-p long" as an option (i.e., add <tt>#SBATCH -p long</tt> to your job script). The priority of such jobs may be throttled in the future if we see that the 'long' queue is having a negative efffect on turnover time in the queue.

=== Interactive ===

For an interactive session use

<pre>
salloc --gres=gpu:4
</pre>

=== Automatic Re-submission and Job Dependencies ===

Commonly you may have a job that you know will take longer to run than what is permissible in the queue. As long as your program contains checkpoint or restart capability, you can have one job automatically submit the next. In the following example it is assumed that the program finishes before the time limit requested and then resubmits itself by logging into the development nodes. Job dependencies and a maximum number of job re-submissions are used to ensure sequential operation.

<pre>
#!/bin/bash

#SBATCH --nodes=1
#SBATCH --ntasks=20 # MPI tasks (needed for srun)
#SBATCH --time=00:10:00 # H:M:S
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node

cd $SLURM_SUBMIT_DIR

: ${job_number:="1"} # set job_nubmer to 1 if it is undefined
job_number_max=3

echo "hi from ${SLURM_JOB_ID}"

#RUN JOB HERE

# SUBMIT NEXT JOB
if [[ ${job_number} -lt ${job_number_max} ]]
then
(( job_number++ ))
next_jobid=$(ssh sgc01-ib0 "cd $SLURM_SUBMIT_DIR; /opt/slurm/bin/sbatch --export=job_number=${job_number} -d afterok:${SLURM_JOB_ID} thisscript.sh | awk '{print $4}'")
echo "submitted ${next_jobid}"
fi

sleep 15

echo "${SLURM_JOB_ID} done"

</pre>

== Software Installed ==

=== IBM PowerAI ===

The PowerAI platform contains popular open machine learning frameworks such as '''Caffe, Tensorflow, and Torch'''. Run the <tt>module avail</tt> command for a complete listing. More information is available at this link: https://developer.ibm.com/linuxonpower/deep-learning-powerai/releases/. Release 4.0 is currently installed.

==== GNU Compilers ====

More recent versions of the GNU Compiler Collection (C/C++/Fortran) are provided in the IBM Advanced Toolchain with enhancements for the POWER8 CPU. To load the newer advance toolchain version use:

Advanced Toolchain V10.0
<pre>
module load gcc/6.3.1
</pre>

Advanced Toolchain V11.0
<pre>
module load gcc/7.2.1
</pre>

More information about the IBM Advanced Toolchain can be found here: [https://developer.ibm.com/linuxonpower/advance-toolchain/ https://developer.ibm.com/linuxonpower/advance-toolchain/]

==== IBM XL Compilers ====

To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run

<pre>
module load xlc/13.1.5
module load xlf/15.1.5
</pre>

Information about the IBM XL Compilers can be found at the following links:

[https://www.ibm.com/support/knowledgecenter/SSXVZZ_13.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++]

[https://www.ibm.com/support/knowledgecenter/SSAT4T_15.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]

==== NVIDIA GPU Driver Version ====

The current NVIDIA driver version is 384.66

==== CUDA ====

The current installed CUDA Tookits is are version 8.0 and version 9.0

<pre>
module load cuda/8.0
</pre>

or

<pre>
module load cuda/9.0
</pre>

The CUDA driver is installed locally, however the CUDA Toolkit is installed in:

<pre>
/usr/local/cuda-8.0
/usr/local/cuda-9.0
</pre>

Note that the <tt>/usr/local/cuda</tt> directory is linked to the <tt>/usr/local/cuda-9.0</tt> directory.

==== OpenMPI ====

Currently OpenMPI has been setup on the 14 nodes connected over EDR Infiniband.

<pre>
$ module load openmpi/2.1.1-gcc-5.4.0
$ module load openmpi/2.1.1-XL-13_15.1.5
</pre>

== Other Software ==

Other software packages can be installed onto the SOSCIP GPU Platform. It is best to try installing new software in your own home directory, which will give you control of the software (e.g. exact version, configuration, installing sub-packages, etc.).

In the following subsections are instructions for installing several common software packages.

=== Anaconda (Python) ===

Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation.

Anaconda can be downloaded from here: [https://www.anaconda.com/download/#linux https://www.anaconda.com/download/#linux]

NOTE: Be sure to download the '''Power8''' installer.

TIP: If you plan to use Tensorflow within Anaconda, download the Python 2.7 version of Anaconda

=== Keras ===

Keras ([https://keras.io/ https://keras.io/]) is a popular high-level deep learning software development framework. It runs on top of other deep-learning frameworks such as TensorFlow.

The easiest way to install Keras is to install Anaconda first, then install Keras by using using the pip command.

Keras uses TensorFlow underneath to run neural network models. Before running code using Keras, be sure to load the PowerAI TensorFlow module and the cuda module.

=== PyTorch ===

PyTorch is the Python implementation of the Torch framework for deep learning.

It is suggested that you use PyTorch within Anaconda.

There is currently no build of PyTorch for POWER8-based systems. You will need to compile it from source.

Obtain the source code from here: [http://pytorch.org/ http://pytorch.org/]

Before building PyTorch, make sure to load cuda by running

<pre>
module load cuda/8.0
</pre>

NOTE: Do not have the gcc modules loaded when building PyTorch. Use the default version of gcc (currently v5.4.0) included with the operating system. Build will fail with later versions of gcc.

SOSCIP GPU

2017-12-06T15:36:08Z

Wagnerse: /* Keras */

{{Infobox Computer
|image=[[Image:S882lc.png|center|300px|thumb]]
|name=SOSCIP GPU
|installed=September 2017
|operatingsystem= Ubuntu 16.04 le
|loginnode= sgc01
|nnodes= 14x Power 8 with 4x NVIDIA P100
|rampernode=512 GB
|corespernode= 2 x 10core (20 physical, 160 SMT)
|interconnect=Infiniband EDR
|vendorcompilers=xlc/xlf, nvcc
}}

== SOSCIP ==

The SOSCIP GPU Cluster is a Southern Ontario Smart Computing Innovation Platform ([http://soscip.org/ SOSCIP]) resource located at theUniversity of Toronto's SciNet HPC facility. The SOSCIP multi-university/industry consortium is funded by the Ontario Government and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].

== Support Email ==

Please use [mailto:soscip-support@scinet.utoronto.ca <soscip-support@scinet.utoronto.ca>] for SOSCIP GPU specific inquiries.

== Specifications==

The SOSCIP GPU Cluster consists of of 14 IBM Power 822LC "Minsky" Servers each with 2x10core 3.25GHz Power8 CPUs and 512GB Ram. Similar to Power 7, the Power 8 utilizes Simultaneous MultiThreading (SMT), but extends the design to 8 threads per core allowing the 20 physical cores to support up to 160 threads. Each node has 4x NVIDIA Tesla P100 GPUs each with 16GB of RAM with CUDA Capability 6.0 (Pascal) connected using NVlink.

== Compile/Devel/Test ==

Access is provided through the BGQ login node, '''<tt> bgqdev.scinet.utoronto.ca </tt>''' using ssh, and from there you can proceed to the GPU development node '''<tt>sgc01-ib0</tt>'''.

== Filesystem ==

The filesystem is shared with the BGQ system. See [https://wiki.scinet.utoronto.ca/wiki/index.php/BGQ#Filesystem here ] for details.

== Job Submission ==

The SOSCIP GPU cluster uses [https://slurm.schedmd.com/ SLURM ] as a job scheduler and jobs are scheduled by node, ie 20 cores and 4 GPUs each. Jobs are submitted from the development node '''<tt>sgc01</tt>'''. The maximum walltime per job is 12 hours (except in the 'long' queue, see below) with up to 8 nodes.

<pre>
$ sbatch myjob.script
</pre>

Where myjob.script is

<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=20 # MPI tasks (needed for srun)
#SBATCH --time=00:10:00 # H:M:S
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node

cd $SLURM_SUBMIT_DIR

hostname
nvidia-smi
</pre>

You can queury job information using

<pre>
squeue
</pre>

To cancel a job use

<pre>
scancel $JOBID
</pre>

=== Longer jobs ===

If your job takes more than 12 hours, the sbatch command will not let you submit your job. There is, however, a way to have jobs up to 24 hours long, by specifying "-p long" as an option (i.e., add <tt>#SBATCH -p long</tt> to your job script). The priority of such jobs may be throttled in the future if we see that the 'long' queue is having a negative efffect on turnover time in the queue.

=== Interactive ===

For an interactive session use

<pre>
salloc --gres=gpu:4
</pre>

=== Automatic Re-submission and Job Dependencies ===

Commonly you may have a job that you know will take longer to run than what is permissible in the queue. As long as your program contains checkpoint or restart capability, you can have one job automatically submit the next. In the following example it is assumed that the program finishes before the time limit requested and then resubmits itself by logging into the development nodes. Job dependencies and a maximum number of job re-submissions are used to ensure sequential operation.

<pre>
#!/bin/bash

#SBATCH --nodes=1
#SBATCH --ntasks=20 # MPI tasks (needed for srun)
#SBATCH --time=00:10:00 # H:M:S
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node

cd $SLURM_SUBMIT_DIR

: ${job_number:="1"} # set job_nubmer to 1 if it is undefined
job_number_max=3

echo "hi from ${SLURM_JOB_ID}"

#RUN JOB HERE

# SUBMIT NEXT JOB
if [[ ${job_number} -lt ${job_number_max} ]]
then
(( job_number++ ))
next_jobid=$(ssh sgc01-ib0 "cd $SLURM_SUBMIT_DIR; /opt/slurm/bin/sbatch --export=job_number=${job_number} -d afterok:${SLURM_JOB_ID} thisscript.sh | awk '{print $4}'")
echo "submitted ${next_jobid}"
fi

sleep 15

echo "${SLURM_JOB_ID} done"

</pre>

== Software Installed ==

=== IBM PowerAI ===

The PowerAI platform contains popular open machine learning frameworks such as '''Caffe, Tensorflow, and Torch'''. Run the <tt>module avail</tt> command for a complete listing. More information is available at this link: https://developer.ibm.com/linuxonpower/deep-learning-powerai/releases/. Release 4.0 is currently installed.

==== GNU Compilers ====

More recent versions of the GNU Compiler Collection (C/C++/Fortran) are provided in the IBM Advanced Toolchain with enhancements for the POWER8 CPU. To load the newer advance toolchain version use:

Advanced Toolchain V10.0
<pre>
module load gcc/6.3.1
</pre>

Advanced Toolchain V11.0
<pre>
module load gcc/7.2.1
</pre>

More information about the IBM Advanced Toolchain can be found here: [https://developer.ibm.com/linuxonpower/advance-toolchain/ https://developer.ibm.com/linuxonpower/advance-toolchain/]

==== IBM XL Compilers ====

To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run

<pre>
module load xlc/13.1.5
module load xlf/15.1.5
</pre>

Information about the IBM XL Compilers can be found at the following links:

[https://www.ibm.com/support/knowledgecenter/SSXVZZ_13.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++]

[https://www.ibm.com/support/knowledgecenter/SSAT4T_15.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]

==== NVIDIA GPU Driver Version ====

The current NVIDIA driver version is 384.66

==== CUDA ====

The current installed CUDA Tookits is are version 8.0 and version 9.0

<pre>
module load cuda/8.0
</pre>

or

<pre>
module load cuda/9.0
</pre>

The CUDA driver is installed locally, however the CUDA Toolkit is installed in:

<pre>
/usr/local/cuda-8.0
/usr/local/cuda-9.0
</pre>

Note that the <tt>/usr/local/cuda</tt> directory is linked to the <tt>/usr/local/cuda-9.0</tt> directory.

==== OpenMPI ====

Currently OpenMPI has been setup on the 14 nodes connected over EDR Infiniband.

<pre>
$ module load openmpi/2.1.1-gcc-5.4.0
$ module load openmpi/2.1.1-XL-13_15.1.5
</pre>

== Other Software ==

Other software packages can be installed onto the SOSCIP GPU Platform. It is best to try installing new software in your own home directory, which will give you control of the software (e.g. exact version, configuration, installing sub-packages, etc.).

In the following subsections are instructions for installing several common software packages.

=== Anaconda (Python) ===

Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation.

Anaconda can be downloaded from here: [https://www.anaconda.com/download/#linux https://www.anaconda.com/download/#linux]

NOTE: Be sure to download the '''Power8''' installer.

TIP: If you plan to use Tensorflow within Anaconda, download the Python 2.7 version of Anaconda

=== Keras ===

Keras is a popular high-level deep learning software development framework. It runs on top of other deep-learning frameworks such as TensorFlow.

The easiest way to install Keras is to install Anaconda first, then install Keras by using using the pip command.

Keras uses TensorFlow underneath to run neural network models. Before running code using Keras, be sure to load the PowerAI TensorFlow module and the cuda module.

=== PyTorch ===

PyTorch is the Python implementation of the Torch framework for deep learning.

It is suggested that you use PyTorch within Anaconda.

There is currently no build of PyTorch for POWER8-based systems. You will need to compile it from source.

Obtain the source code from here: [http://pytorch.org/ http://pytorch.org/]

Before building PyTorch, make sure to load cuda by running

<pre>
module load cuda/8.0
</pre>

NOTE: Do not have the gcc modules loaded when building PyTorch. Use the default version of gcc (currently v5.4.0) included with the operating system. Build will fail with later versions of gcc.

SOSCIP GPU

2017-12-06T15:32:50Z

Wagnerse:

{{Infobox Computer
|image=[[Image:S882lc.png|center|300px|thumb]]
|name=SOSCIP GPU
|installed=September 2017
|operatingsystem= Ubuntu 16.04 le
|loginnode= sgc01
|nnodes= 14x Power 8 with 4x NVIDIA P100
|rampernode=512 GB
|corespernode= 2 x 10core (20 physical, 160 SMT)
|interconnect=Infiniband EDR
|vendorcompilers=xlc/xlf, nvcc
}}

== SOSCIP ==

The SOSCIP GPU Cluster is a Southern Ontario Smart Computing Innovation Platform ([http://soscip.org/ SOSCIP]) resource located at theUniversity of Toronto's SciNet HPC facility. The SOSCIP multi-university/industry consortium is funded by the Ontario Government and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].

== Support Email ==

Please use [mailto:soscip-support@scinet.utoronto.ca <soscip-support@scinet.utoronto.ca>] for SOSCIP GPU specific inquiries.

== Specifications==

The SOSCIP GPU Cluster consists of of 14 IBM Power 822LC "Minsky" Servers each with 2x10core 3.25GHz Power8 CPUs and 512GB Ram. Similar to Power 7, the Power 8 utilizes Simultaneous MultiThreading (SMT), but extends the design to 8 threads per core allowing the 20 physical cores to support up to 160 threads. Each node has 4x NVIDIA Tesla P100 GPUs each with 16GB of RAM with CUDA Capability 6.0 (Pascal) connected using NVlink.

== Compile/Devel/Test ==

Access is provided through the BGQ login node, '''<tt> bgqdev.scinet.utoronto.ca </tt>''' using ssh, and from there you can proceed to the GPU development node '''<tt>sgc01-ib0</tt>'''.

== Filesystem ==

The filesystem is shared with the BGQ system. See [https://wiki.scinet.utoronto.ca/wiki/index.php/BGQ#Filesystem here ] for details.

== Job Submission ==

The SOSCIP GPU cluster uses [https://slurm.schedmd.com/ SLURM ] as a job scheduler and jobs are scheduled by node, ie 20 cores and 4 GPUs each. Jobs are submitted from the development node '''<tt>sgc01</tt>'''. The maximum walltime per job is 12 hours (except in the 'long' queue, see below) with up to 8 nodes.

<pre>
$ sbatch myjob.script
</pre>

Where myjob.script is

<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=20 # MPI tasks (needed for srun)
#SBATCH --time=00:10:00 # H:M:S
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node

cd $SLURM_SUBMIT_DIR

hostname
nvidia-smi
</pre>

You can queury job information using

<pre>
squeue
</pre>

To cancel a job use

<pre>
scancel $JOBID
</pre>

=== Longer jobs ===

If your job takes more than 12 hours, the sbatch command will not let you submit your job. There is, however, a way to have jobs up to 24 hours long, by specifying "-p long" as an option (i.e., add <tt>#SBATCH -p long</tt> to your job script). The priority of such jobs may be throttled in the future if we see that the 'long' queue is having a negative efffect on turnover time in the queue.

=== Interactive ===

For an interactive session use

<pre>
salloc --gres=gpu:4
</pre>

=== Automatic Re-submission and Job Dependencies ===

Commonly you may have a job that you know will take longer to run than what is permissible in the queue. As long as your program contains checkpoint or restart capability, you can have one job automatically submit the next. In the following example it is assumed that the program finishes before the time limit requested and then resubmits itself by logging into the development nodes. Job dependencies and a maximum number of job re-submissions are used to ensure sequential operation.

<pre>
#!/bin/bash

#SBATCH --nodes=1
#SBATCH --ntasks=20 # MPI tasks (needed for srun)
#SBATCH --time=00:10:00 # H:M:S
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node

cd $SLURM_SUBMIT_DIR

: ${job_number:="1"} # set job_nubmer to 1 if it is undefined
job_number_max=3

echo "hi from ${SLURM_JOB_ID}"

#RUN JOB HERE

# SUBMIT NEXT JOB
if [[ ${job_number} -lt ${job_number_max} ]]
then
(( job_number++ ))
next_jobid=$(ssh sgc01-ib0 "cd $SLURM_SUBMIT_DIR; /opt/slurm/bin/sbatch --export=job_number=${job_number} -d afterok:${SLURM_JOB_ID} thisscript.sh | awk '{print $4}'")
echo "submitted ${next_jobid}"
fi

sleep 15

echo "${SLURM_JOB_ID} done"

</pre>

== Software Installed ==

=== IBM PowerAI ===

The PowerAI platform contains popular open machine learning frameworks such as '''Caffe, Tensorflow, and Torch'''. Run the <tt>module avail</tt> command for a complete listing. More information is available at this link: https://developer.ibm.com/linuxonpower/deep-learning-powerai/releases/. Release 4.0 is currently installed.

==== GNU Compilers ====

More recent versions of the GNU Compiler Collection (C/C++/Fortran) are provided in the IBM Advanced Toolchain with enhancements for the POWER8 CPU. To load the newer advance toolchain version use:

Advanced Toolchain V10.0
<pre>
module load gcc/6.3.1
</pre>

Advanced Toolchain V11.0
<pre>
module load gcc/7.2.1
</pre>

More information about the IBM Advanced Toolchain can be found here: [https://developer.ibm.com/linuxonpower/advance-toolchain/ https://developer.ibm.com/linuxonpower/advance-toolchain/]

==== IBM XL Compilers ====

To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run

<pre>
module load xlc/13.1.5
module load xlf/15.1.5
</pre>

Information about the IBM XL Compilers can be found at the following links:

[https://www.ibm.com/support/knowledgecenter/SSXVZZ_13.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++]

[https://www.ibm.com/support/knowledgecenter/SSAT4T_15.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]

==== NVIDIA GPU Driver Version ====

The current NVIDIA driver version is 384.66

==== CUDA ====

The current installed CUDA Tookits is are version 8.0 and version 9.0

<pre>
module load cuda/8.0
</pre>

or

<pre>
module load cuda/9.0
</pre>

The CUDA driver is installed locally, however the CUDA Toolkit is installed in:

<pre>
/usr/local/cuda-8.0
/usr/local/cuda-9.0
</pre>

Note that the <tt>/usr/local/cuda</tt> directory is linked to the <tt>/usr/local/cuda-9.0</tt> directory.

==== OpenMPI ====

Currently OpenMPI has been setup on the 14 nodes connected over EDR Infiniband.

<pre>
$ module load openmpi/2.1.1-gcc-5.4.0
$ module load openmpi/2.1.1-XL-13_15.1.5
</pre>

== Other Software ==

Other software packages can be installed onto the SOSCIP GPU Platform. It is best to try installing new software in your own home directory, which will give you control of the software (e.g. exact version, configuration, installing sub-packages, etc.).

In the following subsections are instructions for installing several common software packages.

=== Anaconda (Python) ===

Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation.

Anaconda can be downloaded from here: [https://www.anaconda.com/download/#linux https://www.anaconda.com/download/#linux]

NOTE: Be sure to download the '''Power8''' installer.

TIP: If you plan to use Tensorflow within Anaconda, download the Python 2.7 version of Anaconda

=== Keras ===

Keras is a popular high-level deep learning software development framework. It runs on top of other deep-learning frameworks such as TensorFlow.

The easiest way to install Keras is to install Anaconda first, then install Keras by using using the pip command.

Keras uses TensorFlow underneath to run neural network models. Before running code using Keras, be sure to load the PowerAI TensorFlow module.

=== PyTorch ===

PyTorch is the Python implementation of the Torch framework for deep learning.

It is suggested that you use PyTorch within Anaconda.

There is currently no build of PyTorch for POWER8-based systems. You will need to compile it from source.

Obtain the source code from here: [http://pytorch.org/ http://pytorch.org/]

Before building PyTorch, make sure to load cuda by running

<pre>
module load cuda/8.0
</pre>

NOTE: Do not have the gcc modules loaded when building PyTorch. Use the default version of gcc (currently v5.4.0) included with the operating system. Build will fail with later versions of gcc.

SOSCIP GPU

2017-12-06T15:04:51Z

Wagnerse: /* IBM PowerAI */

SOSCIP GPU

2017-12-06T15:04:29Z

Wagnerse: /* Software Installed */

SOSCIP GPU

2017-12-06T15:01:42Z

Wagnerse: /* Software */

SOSCIP GPU

2017-11-23T16:15:40Z

Wagnerse: /* CUDA */

SOSCIP GPU

2017-11-21T21:38:37Z

Wagnerse: /* Driver Version */

SOSCIP GPU

2017-11-21T21:38:09Z

Wagnerse: /* IBM Compilers */

SOSCIP GPU

2017-11-21T21:35:12Z

Wagnerse: /* GNU Compilers */

SOSCIP GPU

2017-10-10T18:40:56Z

Wagnerse: /* IBM PowerAI */