SciNet User Support Library

Welcome to the SciNet User Support library. Here you will find up-to-date manuals put together by SciNet staff and users, as well as links to external resources, to help you make use of SciNet resources for computational scientific discovery.

System Status System status can now be found at docs.scinet.utoronto.ca Mon 23 Apr 2018 GPC-compute is decommissioned, GPC-storage available until 30 May 2018 Thu 18 Apr 2018 Niagara system will undergo an upgrade to its Infiniband network between 9am and 12pm, should be transparent to users, however there is a chance of network interruption. Fri 13 Apr 2018 HPSS system will be down for a few hours on Mon, Apr/16, 9AM, for hardware upgrades, in preparation for the eventual move to the Niagara side. Tue 10 Apr 2018 Niagara is open to users. Wed 4 Apr 2018 We are very close to the production launch of Niagara, the new system installed at SciNet. While the RAC allocation year officially starts today, April 4/18, the Niagara system is still undergoing some final tuning and software updates, so the plan is to officially open it to users on next week. All active GPC users will have their accounts, $HOME, and $PROJECT, transferred to the new Niagara system. Those of you who are new to SciNet, but got RAC allocations on Niagara, will have your accounts created and ready for you to login. We are planning an extended Intro to SciNet/Niagara session, available in person at our office, and webcast on Vidyo and possibly other means, on Wednesday April 11 at noon EST.
QuickStart Guides Login essentials GPC: General Purpose Cluster TCS: Tightly Coupled System Data management Software and libraries Job scheduling system (Moab) Performance primer, GPC tools, TCS tools FAQ (frequently asked questions) SciNet Accounts: Application process Usage policy Usage reports Acknowledging SciNet User support	Manuals GPC (Intel) C & C++ compiler Fortran compiler Intel Compiler Floating Point Consistency Math Kernel Library Math Kernel Library link line advisor Intel MPI library TCS (IBM) C compiler, language, optimization C++ compiler, language, optimization Fortran compiler language, optimization UPC compiler, language, optimization, library, user's guide, programmer's guide ESSL high performance math library V4 (V3) LoadLeveler: using & administering Performance tuning Parallel environment Cluster information center has more (also explains error codes!) Adaptive Computing/Cluster Resources MOAB workload manager TORQUE resource manager
Tutorials WARNING: SciNet is in the process of replacing this wiki with a new documentation site. The links below may still contain valuable information, but for current information, please go to https://docs.scinet.utoronto.ca Training material For upcoming classes, see our Training and Education website! SciNet Basics SciNet User Tutorial SciNet Command Line Utilities Intro to SciNet May 2014: Video/Slides SciNet Resources: Video/ Slides Essentials Frequenty asked questions Ssh GPC quickstart TCS quickstart ARC/GPU quickstart ARC/Cell quickstart Important .bashrc guidelines Workflow Optimization (w/focus on Large Scale BioInformatics) Software and libraries Installing your own modules User-space modules and packages (April 2011 SNUG TechTalk) HPSS - SciNet's storage capacity expansion BGQ Hardware Overview Slides / Video Recording Intro to Using the BGQ Slides /Video Recording Linux Linux Tutorial (warning: the last part of this tutorial is specific to the csh shell, whereas SciNet uses the bash shell). Linux Command Line: A Primer (June 2012) Slides, Files Introduction to the Linux Shell, SciNet, Mar 2012: Slides and Data files Batch job management Job and Queue Management Tools (Techtalk Mar.11, 2015) Workflow Optimization (w/focus on Large Scale BioInformatics) GNU Parallel (Techtalk Nov 14, 2012) Job Monitoring on SciNet and Job Efficiency Wallclock time Checkpoints Signals Moab Serial Jobs (including GNU Parallel) Ramdisk Moab workload manager Moab commands Torque resource manager Torque PBS commands Parallel environment Cluster information center (with error codes) LoadLeveler: using & administering Programming General Performance Tuning with the IBM XL Compilers: Slides fromt the SciNet Develop Seminar by Kit Barton, Sep 17, 2012. Remote Development, slides from TechTalk Jun 13, 2012 Scientific Software Development Course, part I of the SciNet's Scientific Computing Course Software Carpentry Resources Version Control: Video/ Slides IBM AIX Workshop, SciNet, Nov 2008 IBM Compiler Workshop, SciNet, Feb 2009 SNUG Techtalk Dec 2011 Intel Compiler Optimizations Fortran Modern Fortran Course (1 day), SciNet, 19 Apr 2011 Slides Source Code [http://support.scinet.utoronto.ca/Manuals/xlf-compiler.pdf IBM Fortran compiler language, optimization C++ Slides and recording of the SciNet Developer Seminar on C++11, March 20, 2013 Scientific C++ Course (1 day), SciNet, 15 March 2011 Slides (updated on Apr 26, 2012) Example source code Videos of the Scientific C++ class Intel C & C++ compiler ro/en-us/fortran/lin/compiler_f/index.htm Intel Fortran compiler] IBM C++ compiler, language, optimization C C refresher: Video/ Slides IBM C compiler, language, optimization Cilk Plus CilkPlus - Getting Started (slides of the May 2014 TechTalk by Chris Szalwinski) Hadoop Introduction to Hadoop for HPCers, Part I - MapReduce: Slides, Source Code, Virtual Machine Perl Perl Python Python IPython Notebook on GPC (January 2014 TechTalk) Research Computing with Python (Modular Course, Fall 2013) Python GUIs with Python and TkInter slides code f2py: Fortran and Python (June 2011 TechTalk by Pierre de Buyl) R R Statistical Package Lua Scripting HALMD with Lua and Luabind (May 2011 TechTalk by Peter Colberg) Parallel Programming 2014 Ontario Summerschool on High Performance Computing Central (incl. OpenMP and MPI) High Performance Scientific Computing Course 2014 (OpenMP and MPI, with video recordings of lectures) MPI 3 Developer Seminar OpenMP 4 Developer Seminar Ontario Summerschool on High Performance Computing Central (2013) High Performance Scientific Computing, part 3 of SciNet's Scientific Computing Course (Winter 2012) Parallel Programming Course (5 days), SciNet, May 2011 Videos, slides and code Parallel Computing for Computational Fluid Dynamics (CFD), SciNet, 23 March 2011 Slides Source Code Intro to Practical Parallel Programming (1 day), SciNet, 22 Sept 2010: Morning Slides, Intro and OpenMP Afternoon Slides, MPI Example source code Parallel Scientific Computing Workshop (5 days), SciNet, Aug 2009: Slides Video Virtual School for CSE Web courses (Jul/Aug 2010): Petascale programming environments and tools Big data for science Proven algorithmic techniques for many-core processors LLNL MPI Tutorial: This was the basis for the MPI workshop at SciNet. Intel MPI library GPC MPI Versions Co-array Fortran on the GPC IBM MPI Workshop, SciNet, Feb 2009 IBM UPC compiler, language, optimization, library, user's guide, programmer's guide GPU Computing 1-day "Introduction to GPGPU Programming with CUDA" Course (May 2015): [1]. 2-day "Programming GPUs with CUDA" Course (2015 Summer School): [2] New Features in CUDA 5 & 6, Nov. 2014 Intro to GPU Computing Using CUDA (WestGrid Spring 2014 Seminar Series) 1.5 hour intro to CUDA, March 2013: Slides and Source Code CITA/SciNet CUDA Minicourse, Fall 2012 SciNet GPU Workshop July 2010 Intro to GPGPU Programming: Video/ Slides (from 5 day parallel programming course at SciNet, May 2011) 1-day intro to GPGPU using CUDA Course (Aug 2011): Source Code, Slides. NVidia archived courses for GPGPU Programming PGI Compiler User's Guide PGI Compiler Reference Manual PGI Fortran reference PGI CUDA Fortran Programming Guide and Reference PGI OpenACC Getting Started Guide Performance Tuning Performance and Profiling Course, April 2013 Introduction To Performance Performance tools for GPC and TCS Dec 2010 SNUG TechTalk: Profiling Tools on GPC Performance tuning MPI Tuning Parameters - SNUG TechTalk, Feb 2012 High Performance Computing Book Online version of an older O'Reilly book which covers the basics of (mostly serial) programming for performance. Covers the most important issues today (such as cache) very clearly. How to Write Fast Numerical Code Good introduction to thinking about performance and cache. Debugging Parallel Debugging with DDT Debugging with GDB and DDT, half-day session at the Ontario HPC Summerschool 2012 Central Slides, Code. TechTalk: Debuggers & Parallel Debugging on SciNet - gdb, ddd, padb], SciNet User Group Meeting, Nov 2010 Video Allinea DDT (Distributed Debugging Tool) User Guide Math libraries (BLAS, LAPACK, FFT) Intel Math Kernel Library (MKL): An overview (TechTalk, March, 2012) Numerical Tools for Physical Scientists, part 2 of SciNet's Scientific computing course, covers, random nubers, blas, lapack, fft, ... Intel Compiler Floating Point Consistency Math Kernel Library (MKL) Math Kernel Library's Vector Statistical Library Math Kernel Library link line advisor ($MKLPATH → ${MKLPATH} in makefiles) ESSL high performance math library V4 (V3) Parallel ESSL high performance math library V3.3 Linear Algebra Libraries by Claire Mouton. 2009 INRIA Technical Report on existing linear algebra libraries for C++ (also here: [3]) I/O Introduction to NetCDF4 binary files with Python, C++ and R (TechTalk March 2014) Using MySQL on the GPC Relational Database Basics Sep 2012 SNUG TechTalk: Science=Data Data management Intro to Parallel I/O, SciNet, Oct 6th, 2010: Morning & MPI-IO Slides NetCDF/HDF5 Slides Source Code. Intro to Parallel I/O, SciNet, Feb, 2013 Video of the Introduction Video of the parallel hdf5 and netcdf part NetCDF/HDF5 Slides Video of the MPI-IO part Half-day HPCS2012 Parallel I/O tutorial, covering MPI-IO, HDF5, NetCDF, based on the above: slides (pdf) and source code. Sept 2010 SNUG TechTalk: Parallel File System and IO Video File System and I/O dos and don'ts So you have 40TB of Data -- an overview of things to consider with large data sets. May 2012 SNUG TechTalk: ADIOS for Parallel IO slides and source code Writting / Reading a table in hdf5 Writting / Reading a table in NetCDF Intro to Parallel I/O, SciNet, Sept 215 MPI-IO Slides and Sample codes NETCDF & HDF5 Infiniband Networking TechTalk on SciNet's Infiniband Network & MPI options Visualization Using Paraview VNC on the GPC (see also slides of the TechTalk on VNC) Visualization Software on the GPC Remote visualization (X-forwarding and VNC) Using Paraview gnuplot, xmgrace, remote visualization tools (X-forwarding and VNC), python's matplotlib Brief overview of ParaView & VisIt VisIt Basics Scientific Visualization using VisIt (Basics and Advanced), M.Ponce and A.Razoumov Intro to Complex Networks Visualization, with Python Science Illustrated: Two-day symposium on Visualizing Science, Feb 2011 Videos of the talks given at Science Illustrated (recorded by KMDI at UoT): Opening remarks by Paul Young Information Visualization and the Myth of Information Overload by Christopher Collins Beyond Basic Visualization by Ramses Van Zon Network Visualization & Analysis by Igor Jurisica Simulation and Visualization of Blood Flow by David Steinman Scientific Visualizations: Does the Science Matter? by Thomas Lucas How can visualization impact public perception of science? Panelists: Jay Ingram, Peter Calamai, Reni Barlow, Hooley McLaughlin How Info Graphics are Created for the Mainstream Media by Peter Calamai Design Boot Camp by Graham Huber Visualization Large Datasets by Jonathan Dursi Visualizing Colliding Black Holes by Herald Pfeiffer Closing remarks by Mubdi Rahman Applications Currently, information on the following applications is available: Athena Amber CCSM3/CCSM4 FLASH3 Gamess. See also here Gromacs LAMMPS NAMD Paraview VASP See also User Codes Development Systems KNL SNUG TechTalk October 2016 - Intel Xeon Phi Knights Landing Manuals Intel compilers and libraries (GPC) C & C++ compiler Fortran compiler Intel Compiler Floating Point Consistency Intel Compiler Optimization Guide Math Kernel Library (MKL) Math Kernel Library's Vector Statistical Library Intel MPI library Math Kernel Library link line advisor ($MKLPATH → ${MKLPATH} in makefiles) IBM compilers and libraries (TCS/P7) C compiler, language, optimization C++ compiler, language, optimization Fortran compiler language, optimization UPC compiler, language, optimization, library, user's guide, programmer's guide ESSL high performance math library V4 (V3) ESSL high performance math library V5.1 for Linux on Power Performance tuning Parallel environment Cluster information center (with error codes) LoadLeveler: using & administering PGI compilers (ARC) Compiler User's Guide Compiler Reference Manual Fortran reference CUDA Fortran Programming Guide and Reference OpenACC Getting Started Guide (Note: $PGI/linux86-64/12.5/doc contains a newer version.) Scheduler (Adaptive Computing/Cluster Resources) Moab workload manager Moab commands Torque resource manager Torque PBS commands DDT Debugger (Allinea) Distributed Debugging Tool User Guide
What's New On The Wiki July 2010: Added an Intel Whitepaper on Floating Point Consistency to the Manual Section (Northrup 13 July) Tips for running the climate modelling code CCSM4 on the TCS from the Peltier group (Guido 11 July) How to apply file access control to specific members of your group (Pinto 7 July) More tips on using GAMESS(US) on the GPC from the Fekl group (Mzd 7 July). Explanation of what PolicyViolation means and why your job won't start with one (Ljdursi 1 July) June 2010: Tips for using HyperThreading on the GPC (Rzon) How to trap signals in C, C++ and Fortran (Rzon) Printing to standard output in .bashrc can break lots of things. This and other tips in important .bashrc guidelines (Rzon) Instructions on using the ParaView Visualization Package on GPC (Ljdursi) Tips on using the Vasp code from the Polanyi Group (Imcnab) Updated tips on using Ram Disk on GPC jobs (Rzon) More sophisticated methods to run serial jobs on GPC (Rzon) `What's new' section on the wiki main page (Ljdursi)	User-Supplied Content Share your expertise with the SciNet Community! Running specific codes on SciNet machines Tips for performance or convenience on SciNet machines Running serial batch jobs (including GNU parallel) Using the ramdisk on SciNet's GPC User-space perl modules Packing up large data sets and putting them on HPSS Remote batch