Difference between revisions of "Knowledge Base: Tutorials and Manuals"

From oldwiki.scinet.utoronto.ca
Jump to navigation Jump to search
m
 
(162 intermediate revisions by 6 users not shown)
Line 1: Line 1:
−
{|-
+
__NOINDEX__
−
|valign="top" width="50%" style="padding:1em; border:1px solid #8fa1b2; background-color:#F0F0F0"|
+
 
−
'''Tutorials and workshops'''
+
{| style="border-spacing: 8px; width:100%"
−
* '''''SciNet Class Schedule'''''<br>Let us know which one(s) you will attend by [mailto:support@scinet.utoronto.ca email]<br>All classes will be given at the [http://www.scinet.utoronto.ca/about/Contact.htm SciNet headquarters].
+
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|
−
** "Intro to SciNet" classes. These will take about 90 minutes of your time and you will learn a lot on how to use SciNet's systems. Even experienced SciNet users may still pick up some pointers.<br>- '''Fri 3 Sept, 12-1:30'''. First of the year, with Pizza!<br>- '''Fri 10 Sept, 12-1:30.'''<br>Please bring a print-out of the latest User Tutorial to the class.<br>More will be given throughout the year depending on user interest.
+
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. The links below may still contain valuable information, but for current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''
−
** TechTalk: These half-hour sessions will be part of the SciNet User Group meetings, every second Wednesday of the month at '''12 noon'''. SNUG meeting will also include user discussion and pizza. TechTalk Topics:<br>- '''Wed 8 Sept:''' The GPFS File Systems<br>- '''Wed 13 Oct:''' Version control on SciNet - svn, git, mercurial<br>- '''Wed 10 Nov:''' Debuggers & parallel debugging on SciNet - gdb, ddd, padb<br>- '''Wed 8 Dec:'''  Performance and profiling on SciNet - gprof, scalasca, peekperf
+
|}
−
** Classes<br>- '''Sept 22:'''  Intro to Parallel Programming (whole day)<br>- '''Oct 6:''' Parallel I/O  (whole day)<br>- '''May 2011:''' 5-day Parallel Programming course.<br>OpenMP, MPI and more will be covered. The aim is for you to leave the course able to start parallelizing your code.
+
 
−
* [https://support.scinet.utoronto.ca/wiki/images/5/54/SciNet_Tutorial.pdf SciNet User Tutorial]
+
__TOC__
−
* [[SciNet GPU Workshop July 2010]]
+
 
−
* Parallel Scientific Computing Workshop, SciNet, Aug 2009<br />[[ Parallel_Scientific_Computing_-_Aug_09  | Slides ]]/[http://www.cita.utoronto.ca/~ljdursi/PSP/ Video]
+
=Training material=
 +
 
 +
For upcoming classes, see our [https://support.scinet.utoronto.ca/education/ Training and Education website]!
 +
==SciNet Basics==
 +
* [[Media:SciNet_Tutorial.pdf|SciNet User Tutorial]]
 +
* [[SciNet Command Line Utilities]]
 +
* Intro to SciNet May 2014: [http://support.scinet.utoronto.ca/CourseVideo/SciNetIntroMay2014/IntroMay2014.html Video]/[[Media:IntroScinetMay2014.pdf|Slides]]
 +
* SciNet Resources: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Monday_Morning_SciNet_Resources/Monday_Morning_SciNet_Resources.mp4 Video]/ [[Media:Monday_Morning_SciNet_Resources.pdf|Slides]]
 +
* [[Essentials]]
 +
* [[FAQ|Frequenty asked questions]]
 +
* [[Ssh]]
 +
* [[GPC_Quickstart|GPC quickstart]]
 +
* [[TCS_Quickstart|TCS quickstart]]
 +
* [[GPU_Devel_Nodes|ARC/GPU quickstart]]
 +
* [[Cell_Devel_Nodes|ARC/Cell quickstart]]
 +
* [[Important .bashrc guidelines]]
 +
* [[Media:LargeScaleBio.pdf‎|Workflow Optimization (w/focus on Large Scale BioInformatics)]]
 +
* [[Software_and_Libraries | Software and libraries]]
 +
* [[Installing your own modules]]
 +
* [[Media:SNUGlocalsetup.pdf|User-space modules and packages (April 2011 SNUG TechTalk)]]
 +
* [[Media:HPSS_rationale.pdf|HPSS - SciNet's storage capacity expansion]]
 +
* BGQ Hardware Overview [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides ]/ [https://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 Video Recording ]
 +
* Intro to Using the BGQ [[Media:Bgqintro.pdf|Slides ]]/[https://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 Video Recording ]
 +
 
 +
==Linux==
 +
* [http://www.ee.surrey.ac.uk/Teaching/Unix/index.html Linux Tutorial] (warning: the last part of this tutorial is specific to the csh shell, whereas SciNet uses the bash shell).
 +
* Linux Command Line: A Primer (June 2012) [[Media:SS_IntroToShell.pdf|Slides,]] [[Media:SS_IntroToShell.tgz|Files]]
 +
* Introduction to the Linux Shell, SciNet, Mar 2012: [[Media:IntroToShell.pdf|Slides]] and [[Media:Shell-data.tgz|Data files]]
 +
 
 +
==Batch job management==
 +
* [https://support.scinet.utoronto.ca/education/go.php/183/file_storage/index.php/download/1/files%5B%5D/3702/ Job and Queue Management Tools] (Techtalk Mar.11, 2015)
 +
* [[Media:LargeScaleBio.pdf‎|Workflow Optimization (w/focus on Large Scale BioInformatics)]]
 +
* [[Media:Tech-talk-gnu-parallel.pdf|GNU Parallel (Techtalk Nov 14, 2012)]]
 +
* [[Media:TechTalkJobMonitoring.pdf|Job Monitoring on SciNet and Job Efficiency]]
 +
<!-- * [[Media:Snugtrackjob.pdf|Job Monitoring on SciNet and Job Efficiency]] -->
 +
* [[Wallclock time]]
 +
* [[Checkpoints]]
 +
* [[Using_Signals|Signals]]
 +
* [[Moab]]
 +
* [[User_Serial|Serial Jobs (including GNU Parallel)]]
 +
* [[User_Ramdisk|Ramdisk]]
 +
* [http://www.clusterresources.com/products/mwm/docs/index.shtml Moab workload manager]
 +
* [http://www.clusterresources.com/products/mwm/docs/a.gcommandoverview.shtml Moab commands]
 +
* [http://www.clusterresources.com/products/torque/docs/ Torque resource manager]
 +
* [http://www.clusterresources.com/products/torque/docs/a.acommands.shtml Torque PBS commands]
 +
* [http://support.scinet.utoronto.ca/Manuals/PE5.1-operationanduse.pdf Parallel environment]
 +
* [http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp Cluster information center] (with error codes)
 +
* [http://support.scinet.utoronto.ca/Manuals/LL-usingandadministering.pdf LoadLeveler: using & administering]
 +
 
 +
==Programming==
 +
===General===
 +
* [[Media:SciDev-XLCompilers.pdf|Performance Tuning with the IBM XL Compilers]]: Slides fromt the SciNet Develop Seminar by Kit Barton, Sep 17, 2012.
 +
* [[Media:Remotescinet.pdf‎|Remote Development]], slides from TechTalk Jun 13, 2012
 +
* [[Scientific Software Development Course]], part I of the SciNet's Scientific Computing Course
 +
* [http://software-carpentry.org Software Carpentry Resources]
 +
* Version Control: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Thursday_Morning_BP_Revision_Control/Thursday_Morning_BP_Revision_Control.mp4 Video]/ [[Media:Snug_techtalk_revcontrol.pdf | Slides]]
 
* [[IBM_Nov_Workshop | IBM AIX Workshop, SciNet, Nov 2008 ]]  
 
* [[IBM_Nov_Workshop | IBM AIX Workshop, SciNet, Nov 2008 ]]  
−
* [[IBM_Feb_Workshop | IBM MPI Workshop, SciNet, Feb 2009]]
 
 
* [[IBM_Compiler_Workshop | IBM Compiler Workshop, SciNet, Feb 2009]]
 
* [[IBM_Compiler_Workshop | IBM Compiler Workshop, SciNet, Feb 2009]]
 +
* SNUG Techtalk Dec 2011 [[Media:Snug_techtalk_compiler.pdf | Intel Compiler Optimizations]]
 +
 +
===Fortran===
 +
* Modern Fortran Course (1 day), SciNet, 19 Apr 2011
 +
** [[Media:ModernFortran.pdf | Slides]]
 +
** [[Media:ModernFortran.tgz | Source Code]]
 +
* [http://software.intel.com/sites/products/documentation/hpc/compilerp* [http://support.scinet.utoronto.ca/Manuals/xlf-compiler.pdf IBM Fortran compiler] [http://support.scinet.utoronto.ca/Manuals/xlf-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlf-proguide.pdf optimization]
 +
 +
===C++===
 +
* [[Media:Cpp11.pdf|Slides]] and [http://support.scinet.utoronto.ca/CourseVideo/Cpp11/cpp11.html recording] of the SciNet Developer Seminar on C++11, March 20, 2013
 +
* Scientific C++ Course (1 day), SciNet, 15 March 2011
 +
** [[Media:Scientific-c%2B%2B.pdf|Slides]] (updated on Apr 26, 2012)
 +
** [[Media:Scinetcppexamples.tgz|Example source code]]
 +
** [[Videos_of_the_One-Day_Scientific_C%2B%2B_Class | Videos of the Scientific C++ class]]
 +
* [http://software.intel.com/sites/products/documentation/hpc/compilerpro/en-us/cpp/lin/compiler_c/index.htm Intel C & C++ compiler]
 +
ro/en-us/fortran/lin/compiler_f/index.htm Intel Fortran compiler]
 +
* [http://support.scinet.utoronto.ca/Manuals/xlC++-compiler.pdf IBM C++ compiler], [http://support.scinet.utoronto.ca/Manuals/xlC++-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlC++-proguide.pdf optimization]
 +
 +
===C===
 +
* C refresher: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Monday_Morning_C_Review/Monday_Morning_C_Review.mp4 Video]/ [[Media:Monday_Morning_C_Review.pdf| Slides]]
 +
* [http://support.scinet.utoronto.ca/Manuals/xlc-compiler.pdf IBM C compiler], [http://support.scinet.utoronto.ca/Manuals/xlc-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlc-proguide.pdf optimization]
 +
 +
===Cilk Plus===
 +
* [[Media:TechTalkSzalwinskiCilkPlus.pdf|CilkPlus - Getting Started]] (slides of the May 2014 TechTalk by Chris Szalwinski)
 +
 +
===Hadoop===
 +
* Introduction to Hadoop for HPCers, Part I - MapReduce: [[Media:Hadoop-PartI.pdf|Slides]], [[Media:HadoopPart1examples.tgz|Source Code]], [http://support.scinet.utoronto.ca/~ljdursi/SciNetHadoopVM.zip Virtual Machine]
  
−
''External:''
+
===Perl===
 +
* [[Perl]]
 +
===Python===
 +
* [[Python]]
 +
* [[IPython Notebook on GPC]] (January 2014 TechTalk)
 +
* [[Research Computing with Python]] (Modular Course, Fall 2013)
 +
* [[Intro to Tkinter|Python GUIs with Python and TkInter]] [[Media:Tkinter.pdf|slides]] [[Media:Tkinter_code.tgz|code]]
 +
* [http://support.scinet.utoronto.ca/Snug/scinet-f2py/scinet-f2py.html f2py: Fortran and Python] (June 2011 TechTalk by Pierre de Buyl)
  
−
* [http://www.vscse.org/news/apply_summer2010.html Virtual School for CSE] Web courses (Jul/Aug 2010):
+
===R===
 +
* [[R Statistical Package]]
 +
===Lua===
 +
* [[Media:PeterColberg_Lua_scinet.pdf | Scripting HALMD with Lua and Luabind]] (May 2011 TechTalk by Peter Colberg)
 +
 
 +
==Parallel Programming==
 +
* [[2014 Ontario Summerschool on High Performance Computing Central]] (incl. OpenMP and MPI)
 +
* [[High Performance Scientific Computing Course 2014]] (OpenMP and MPI, with video recordings of lectures)
 +
* [[Media:SciNet_MPI3.0_seminar.pdf | MPI 3 Developer Seminar]]
 +
* [[Media:Ds-openmp.pdf| OpenMP 4 Developer Seminar]]
 +
* [[Ontario Summerschool on High Performance Computing Central]] (2013)
 +
* [[High Performance Scientific Computing]], part 3 of SciNet's Scientific Computing Course (Winter 2012)
 +
* Parallel Programming Course (5 days), SciNet, May 2011
 +
** [[Parallel_Scientific_Computing_-_May_2011 | Videos, slides and code]]
 +
* Parallel Computing for Computational Fluid Dynamics (CFD), SciNet, 23 March 2011
 +
** [[Media:parCFD-mpi.pdf | Slides]]
 +
** [[Media:parCFD.tgz | Source Code]]
 +
* Intro to Practical Parallel Programming (1 day), SciNet, 22 Sept 2010:
 +
**[[Media:PPP-Intro-Morning.pdf|Morning Slides, Intro and OpenMP ]]
 +
**[[Media:PPP-Intro-Afternoon.pdf|Afternoon Slides, MPI]]
 +
**[[Media:Intro-ppp.tgz|Example source code]]
 +
* Parallel Scientific Computing Workshop (5 days), SciNet, Aug 2009:
 +
**[[ Parallel_Scientific_Computing_-_Aug_09 | Slides ]]
 +
**[http://www.cita.utoronto.ca/~ljdursi/PSP/ Video]
 +
* [http://www.vscse.org/  Virtual School for CSE] Web courses (Jul/Aug 2010):
 
** Petascale programming environments and tools
 
** Petascale programming environments and tools
 
** Big data for science
 
** Big data for science
 
** Proven algorithmic techniques for many-core processors
 
** Proven algorithmic techniques for many-core processors
 +
* [https://computing.llnl.gov/tutorials/mpi/ LLNL MPI Tutorial]: This was the basis for the MPI workshop at SciNet.
 +
* [http://software.intel.com/sites/products/documentation/hpc/mpi/linux/reference_manual.pdf Intel MPI library]
 +
* [[GPC MPI Versions]]
 +
* [[Co-array Fortran on the GPC]]
 +
* [[IBM_Feb_Workshop | IBM MPI Workshop, SciNet, Feb 2009]]
 +
* [http://support.scinet.utoronto.ca/Manuals/UPC/compiler.pdf IBM UPC compiler], [http://support.scinet.utoronto.ca/Manuals/UPC/langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/UPC/upcopt.pdf optimization], [http://support.scinet.utoronto.ca/Manuals/UPC/standlib.pdf library], [http://support.scinet.utoronto.ca/Manuals/UPC/upcusersguide.pdf user's guide], [http://support.scinet.utoronto.ca/Manuals/UPC/proguide.pdf programmer's guide]
 +
 +
==GPU Computing==
 +
* 1-day "Introduction to GPGPU Programming with CUDA" Course (May 2015): [https://support.scinet.utoronto.ca/education/go.php/251/index.php/ib/1//p_course/251].
 +
* 2-day "Programming GPUs with CUDA" Course (2015 Summer School): [https://wiki.scinet.utoronto.ca/wiki/index.php/2015_Ontario_Summer_School_for_High_Performance_Computing_Central#Programming_GPUs_with_CUDA]
 +
* [[Media:SNUG_NOV_CUDA.pdf | New Features in CUDA 5 & 6, Nov. 2014]]
 +
* [[Media:Westgrid_CUDA.pdf | Intro to GPU Computing Using CUDA]] (WestGrid Spring 2014 Seminar Series)
 +
* 1.5 hour intro to CUDA, March 2013: [[Media:CUDA-Graphics-Intro-2013.pdf | Slides]] and [[Media:CUDA-Graphics-Intro-2013.tgz | Source Code]]
 +
* [[CUDA_Minicourse_Fall_2012 | CITA/SciNet CUDA Minicourse, Fall 2012]]
 +
* [[SciNet GPU Workshop July 2010]]
 +
* Intro to GPGPU Programming: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Friday_Morning_GPGPU/Friday_Morning_GPGPU.mp4 Video]/ [[Media:Gpgpu.pdf | Slides]]<br />(from 5 day parallel programming course at SciNet, May 2011)
 +
* 1-day intro to GPGPU using CUDA Course (Aug 2011): [[Media:Intro-gpu.tgz | Source Code]], [[Media:IntroGPGPU-Aug2011.pdf | Slides]].
 
* [http://developer.nvidia.com/object/cuda_training.html  NVidia archived courses for GPGPU Programming]
 
* [http://developer.nvidia.com/object/cuda_training.html  NVidia archived courses for GPGPU Programming]
−
* [https://computing.llnl.gov/tutorials/mpi/ LLNL MPI Tutorial]<br />This was the basis for the MPI workshop at SciNet.
+
* [http://www.pgroup.com/doc/pgiug.pdf PGI Compiler User's Guide]
−
* [http://cnx.org/content/col11136/latest/  High Performance Computing Book]<br />Online version of an older O'Reilly book which covers the basics of (mostly serial) programming for performance.  Covers the most important issues today very clearly.
+
* [http://www.pgroup.com/doc/pgiref.pdf PGI Compiler Reference Manual]
−
* [http://www.ece.cmu.edu/~franzf/papers/gttse07.pdf  How to Write Fast Numerical Code ]<br />Good introduction to thinking about performance.
+
* [http://www.pgroup.com/doc/pgifortref.pdf PGI Fortran reference]
−
|valign="top" style="padding:1em; border:1px solid #8fa1b2; background-color:#F0F0F0"|
+
* [http://www.pgroup.com/doc/pgicudaforug.pdf PGI CUDA Fortran Programming Guide and Reference]
−
'''Manuals'''
+
* [http://www.pgroup.com/doc/openACC_gs.pdf PGI OpenACC Getting Started Guide]
 +
 
 +
==Performance Tuning==
 +
* [[Performance and Profiling Course, April 2013]]
 +
* [[Introduction To Performance]]
 +
* Performance tools for [[Performance_And_Debugging_Tools:_GPC | GPC ]] and [[Performance_And_Debugging_Tools:_TCS | TCS ]]
 +
* Dec 2010 SNUG TechTalk: [[Media:ProfillingTechTalk-Dec2010.pdf | Profiling Tools on GPC]]
 +
* [http://support.scinet.utoronto.ca/Manuals/JUMP-AIX-POWER6-AppsPerformanceTuning-wp032008.pdf Performance tuning]
 +
* [[Media:Mpi-tuning-parameters.pdf‎ | MPI Tuning Parameters]] - SNUG TechTalk, Feb 2012
 +
* [http://cnx.org/content/col11136/latest/  High Performance Computing Book]<br />Online version of an older O'Reilly book which covers the basics of (mostly serial) programming for performance.  Covers the most important issues today (such as cache) very clearly.
 +
* [http://www.ece.cmu.edu/~franzf/papers/gttse07.pdf  How to Write Fast Numerical Code ]<br />Good introduction to thinking about performance and cache.
 +
 
 +
==Debugging==
 +
* [[Parallel Debugging with DDT]]
 +
* [[Media:SS_Debug.pdf|Debugging with GDB and DDT, half-day session at the Ontario HPC Summerschool 2012 Central<br>Slides]], [[Media:SS_Debug.tgz|Code]].
 +
* [[Media:Snugdebug.pdf|TechTalk: Debuggers & Parallel Debugging on SciNet - gdb, ddd, padb], SciNet User Group Meeting, Nov 2010]]<br/ > [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Thursday_Morning_Debugging/Thursday_Morning_Debugging.mp4 Video]
 +
* [http://www.allinea.com/downloads/userguide.pdf Allinea DDT (Distributed Debugging Tool) User Guide]
 +
 
 +
==Math libraries (BLAS, LAPACK, FFT)==
 +
* [[Media:MKLTechTalkMarch2012.pdf|Intel Math Kernel Library (MKL): An overview]] (TechTalk, March, 2012)
 +
* [[Numerical Tools for Physical Scientists]], part 2 of SciNet's Scientific computing course, covers, random nubers, blas, lapack, fft, ...
 +
* [[Media:FP_Consistency.pdf|Intel Compiler Floating Point Consistency]]
 +
* [http://software.intel.com/sites/products/documentation/hpc/mkl/lin/index.htm Math Kernel Library (MKL)]
 +
* [http://software.intel.com/sites/products/documentation/hpc/mkl/vsl/vslnotes.pdf Math Kernel Library's Vector Statistical Library]
 +
* [http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor Math Kernel Library link line advisor]<br/>($MKLPATH &rarr; ${MKLPATH} in makefiles)
 +
* [http://publib.boulder.ibm.com/epubs/pdf/am501405.pdf ESSL high performance math library V4] ([http://publib.boulder.ibm.com/epubs/pdf/am601305.pdf V3])
 +
* [http://publib.boulder.ibm.com/epubs/pdf/am601305.pdf Parallel ESSL high performance math library V3.3]
 +
* [http://hal.inria.fr/inria-00576469 Linear Algebra Libraries] by Claire Mouton. 2009 INRIA Technical Report on existing linear algebra libraries for C++ (also here: [http://arxiv.org/abs/1103.3020])
 +
 
 +
==I/O==
 +
 
 +
* [[Media:NetCDF.pdf|Introduction to NetCDF4 binary files with Python, C++ and R (TechTalk March 2014)]]
 +
* [[Using MySQL on the GPC]]
 +
* [https://support.scinet.utoronto.ca/education/go.php/19/content.php/cid/59/ Relational Database Basics]
 +
* [[Media:SCIENCEDATA.pdf‎|Sep 2012 SNUG TechTalk: Science=Data]]
 +
* [[Data_Management|Data management]]
 +
* Intro to Parallel I/O, SciNet, Oct 6th, 2010:
 +
**[[Media:Parallel_io_course.pdf‎|Morning & MPI-IO Slides]]
 +
**[[Media:Netcdfhdf5.pdf|NetCDF/HDF5 Slides]]
 +
**[[Media:ParIO.tgz|Source Code]]. 
 +
* Intro to Parallel I/O, SciNet, Feb, 2013
 +
** [http://support.scinet.utoronto.ca/CourseVideo/ParallelIOcourse/pario-intro/pario-intro.mp4 Video of the Introduction]
 +
** [http://support.scinet.utoronto.ca/CourseVideo/ParallelIOcourse/pario-netcdfhdf5/pario-netcdfhdf5.mp4 Video of the parallel hdf5 and netcdf part] [[Media:Netcdfhdf5.pdf|NetCDF/HDF5 Slides]]
 +
** [http://support.scinet.utoronto.ca/CourseVideo/ParallelIOcourse/pario-mpiio/pario-mpiio.mp4 Video of the MPI-IO part]
 +
* Half-day HPCS2012 Parallel I/O tutorial, covering MPI-IO, HDF5, NetCDF, based on the above:  [[Media:ParIO-HPCS2012.pdf|slides (pdf)]] and [[Media:ParIO-HPCS2012.tgz|source code]].
 +
* [[Media:Snugio.pdf|Sept 2010 SNUG TechTalk: Parallel File System and IO]] <br/ >[http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Friday_Morning_IO/Friday_Morning_IO.mp4 Video]
 +
* [[File System and I/O dos and don'ts]]
 +
* [[Media:40TB.pdf|So you have 40TB of Data]] -- an overview of things to consider with large data sets.
 +
* [[Media:Adios-techtalk-may2012.pdf|May 2012 SNUG TechTalk: ADIOS for Parallel IO slides]] and [[Media:Adios-techtalk-may2012-src.tgz|source code]]
 +
* [[hdf5_table|Writting / Reading a table in hdf5]]
 +
* [[NetCDF_table|Writting / Reading a table in NetCDF]]
 +
* Intro to Parallel I/O, SciNet, Sept 215
 +
** MPI-IO [[Media:ParIO.MPIIO.2015.pdf |Slides]] and [[Media:ParIO.MPIIO.tar.gz | Sample codes]]
 +
** [[Media:netcdf2015.pdf | NETCDF]] & [[Media:hfd5.pdf | HDF5]]
 +
 
 +
==Infiniband Networking==
 +
* [[Media:Snug_techtalk_Infiniband.pdf | TechTalk on SciNet's Infiniband Network & MPI options ]]
 +
 
 +
==Visualization==
 +
* [[Using Paraview]]
 +
* [[VNC|VNC on the GPC]] (see also [[Media:Ttvnc.pdf|slides of the TechTalk on VNC]])
 +
* [[Software_and_Libraries#anchor_viz|Visualization Software on the GPC]]
 +
* [https://wiki.scinet.utoronto.ca/wiki/images/5/51/Remoteviz.pdf Remote visualization (X-forwarding and VNC)]
 +
* [[Using Paraview]]
 +
* [https://support.scinet.utoronto.ca/~mponce/ss2016/ss2016_visualization-I.pdf gnuplot, xmgrace, remote visualization tools (X-forwarding and VNC), python's matplotlib]
 +
* [https://support.scinet.utoronto.ca/~mponce/ss2016/ss2016_visualization-II.pdf  Brief overview of ParaView & VisIt]
 +
* [https://support.scinet.utoronto.ca/education/go.php/242/file_storage/index.php/download/1/files%5B%5D/6399/ VisIt Basics]
 +
* [https://support.scinet.utoronto.ca/education/go.php/273/file_storage/index.php/download/1/files%5B%5D/7363/ Scientific Visualization using VisIt (Basics and Advanced)], M.Ponce and A.Razoumov
 +
* [https://support.scinet.utoronto.ca/education/get.php/8_ComplexNetworks.pdf Intro to Complex Networks Visualization, with Python]
 +
 
 +
 
 +
* [http://scienceillustrated.ca Science Illustrated:] Two-day symposium on Visualizing Science, Feb 2011
 +
* [http://www.kmdi.utoronto.ca/story/2011/03/si-science-illustrated-symposium-success Videos of the talks given at Science Illustrated] (recorded by [http://www.kmdi.utoronto.ca KMDI] at [http://www.utoronto.ca UoT]):
 +
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=94ff5cd5-be6e-4fc6-9be6-dd2222342bcd Opening remarks] by Paul Young
 +
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=4255c34e-15e7-4b24-ba99-78f5c8fa4381 Information Visualization and the Myth of Information Overload] by Christopher Collins
 +
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=adcf02bf-16cb-46cc-8cdc-1a65e9071d6b Beyond Basic Visualization] by Ramses Van Zon
 +
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=47baf346-3599-4fa3-9b10-6a58faa6b33c Network Visualization &amp; Analysis] by Igor Jurisica
 +
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=7e754a2e-7be5-476e-bb54-37def37bc07e Simulation and Visualization of Blood Flow] by David Steinman
 +
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=d7622587-2c31-49c1-99d7-f0f16c078801 Scientific Visualizations: Does the Science Matter?] by Thomas Lucas
 +
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=9963c637-6840-454f-a57f-9a1be6456616 How can visualization impact public perception of science?] Panelists: Jay Ingram, Peter Calamai, Reni Barlow, Hooley McLaughlin
 +
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=88e50cff-db9b-4c71-b10d-781fec60a2c0 How Info Graphics are Created for the Mainstream Media] by Peter Calamai
 +
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=3897e2a3-1fda-42be-ab78-edbab090fd9e Design Boot Camp] by Graham Huber
 +
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=4c27ed76-7292-407e-83a6-814e1461eccd Visualization Large Datasets] by Jonathan Dursi
 +
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=7d49c845-3937-44e2-a300-9b8ffe57a857 Visualizing Colliding Black Holes] by Herald Pfeiffer
 +
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=7d7e4803-39cd-4c8e-a443-bc2a7b1b3c28 Closing remarks] by Mubdi Rahman
 +
 
 +
==Applications==
 +
{{:Knowledge Base: Applications}}
 +
* See also [[User Codes]]
 +
 
 +
== Development Systems ==
 +
 
 +
=== KNL ===
 +
 
 +
* SNUG TechTalk October 2016 -  [[Media:Snug_techtalk_KNL.pdf | Intel Xeon Phi Knights Landing  ]]
  
−
''Intel (GPC):''
+
=Manuals=
−
* [http://software.intel.com/sites/products/documentation/hpc/compilerpro/en-us/cpp/lin/compiler_c/index.htm C & C++ compiler]
+
 
−
* [http://software.intel.com/sites/products/documentation/hpc/compilerpro/en-us/fortran/lin/compiler_f/index.htm Fortran compiler]
+
==Intel compilers and libraries (GPC)==
−
* [https://support.scinet.utoronto.ca/wiki/images/f/f2/FP_Consistency.pdf Intel Compiler Floating Point Consistency]
+
* [http://software.intel.com/sites/products/documentation/doclib/stdxe/2013/composerxe/compiler/cpp-lin/index.htm C & C++ compiler]
−
* [http://software.intel.com/sites/products/documentation/hpc/mkl/lin/index.htm Math Kernel Library]  
+
* [http://software.intel.com/sites/products/documentation/doclib/stdxe/2013/composerxe/compiler/fortran-lin/index.htm Fortran compiler]
−
* [http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor Math Kernel Library link line advisor]
+
* [[Media:FP_Consistency.pdf|Intel Compiler Floating Point Consistency]]
 +
* [[Media:Compiler_qrg12.pdf‎|Intel Compiler Optimization Guide]]
 +
* [http://software.intel.com/sites/products/documentation/hpc/mkl/lin/index.htm Math Kernel Library (MKL)]  
 +
* [http://software.intel.com/sites/products/documentation/hpc/mkl/vsl/vslnotes.pdf Math Kernel Library's Vector Statistical Library]
 
* [http://software.intel.com/sites/products/documentation/hpc/mpi/linux/reference_manual.pdf Intel MPI library]
 
* [http://software.intel.com/sites/products/documentation/hpc/mpi/linux/reference_manual.pdf Intel MPI library]
 +
* [http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor Math Kernel Library link line advisor]<br/>($MKLPATH &rarr; ${MKLPATH} in makefiles)
  
−
''IBM (TCS):''
+
==IBM compilers and libraries (TCS/P7)==
 
* [http://support.scinet.utoronto.ca/Manuals/xlc-compiler.pdf C compiler], [http://support.scinet.utoronto.ca/Manuals/xlc-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlc-proguide.pdf optimization]
 
* [http://support.scinet.utoronto.ca/Manuals/xlc-compiler.pdf C compiler], [http://support.scinet.utoronto.ca/Manuals/xlc-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlc-proguide.pdf optimization]
 
* [http://support.scinet.utoronto.ca/Manuals/xlC++-compiler.pdf C++ compiler], [http://support.scinet.utoronto.ca/Manuals/xlC++-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlC++-proguide.pdf optimization]
 
* [http://support.scinet.utoronto.ca/Manuals/xlC++-compiler.pdf C++ compiler], [http://support.scinet.utoronto.ca/Manuals/xlC++-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlC++-proguide.pdf optimization]
Line 40: Line 267:
 
* [http://support.scinet.utoronto.ca/Manuals/UPC/compiler.pdf UPC compiler], [http://support.scinet.utoronto.ca/Manuals/UPC/langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/UPC/upcopt.pdf optimization], [http://support.scinet.utoronto.ca/Manuals/UPC/standlib.pdf library], [http://support.scinet.utoronto.ca/Manuals/UPC/upcusersguide.pdf user's guide], [http://support.scinet.utoronto.ca/Manuals/UPC/proguide.pdf programmer's guide]
 
* [http://support.scinet.utoronto.ca/Manuals/UPC/compiler.pdf UPC compiler], [http://support.scinet.utoronto.ca/Manuals/UPC/langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/UPC/upcopt.pdf optimization], [http://support.scinet.utoronto.ca/Manuals/UPC/standlib.pdf library], [http://support.scinet.utoronto.ca/Manuals/UPC/upcusersguide.pdf user's guide], [http://support.scinet.utoronto.ca/Manuals/UPC/proguide.pdf programmer's guide]
 
* [http://publib.boulder.ibm.com/epubs/pdf/am501405.pdf ESSL high performance math library V4] ([http://publib.boulder.ibm.com/epubs/pdf/am601305.pdf V3])
 
* [http://publib.boulder.ibm.com/epubs/pdf/am501405.pdf ESSL high performance math library V4] ([http://publib.boulder.ibm.com/epubs/pdf/am601305.pdf V3])
 +
* [[Media:essl51.pdf|ESSL high performance math library V5.1 for Linux on Power]]
 
* [http://support.scinet.utoronto.ca/Manuals/JUMP-AIX-POWER6-AppsPerformanceTuning-wp032008.pdf Performance tuning]
 
* [http://support.scinet.utoronto.ca/Manuals/JUMP-AIX-POWER6-AppsPerformanceTuning-wp032008.pdf Performance tuning]
 
* [http://support.scinet.utoronto.ca/Manuals/PE5.1-operationanduse.pdf Parallel environment]
 
* [http://support.scinet.utoronto.ca/Manuals/PE5.1-operationanduse.pdf Parallel environment]
−
* [http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp Cluster information center] (also explains error codes!)
+
* [http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp Cluster information center] (with error codes)
 +
* [http://support.scinet.utoronto.ca/Manuals/LL-usingandadministering.pdf LoadLeveler: using & administering]
  
−
''Scheduler''
+
==PGI compilers (ARC)==
−
* [http://www.clusterresources.com/products/mwm/docs/index.shtml Moab workload manager](Adaptive Computing/Cluster Resources)
+
* [http://www.pgroup.com/doc/pgiug.pdf Compiler User's Guide]
−
* [http://www.clusterresources.com/products/mwm/docs/a.gcommandoverview.shtml Moab commands] (Adaptive Computing/Cluster Resources)
+
* [http://www.pgroup.com/doc/pgiref.pdf Compiler Reference Manual]
−
* [http://www.clusterresources.com/products/torque/docs/ Torque resource manager] (Adaptive Computing/Cluster Resources)
+
* [http://www.pgroup.com/doc/pgifortref.pdf Fortran reference]
−
* [http://www.clusterresources.com/products/torque/docs/a.acommands.shtml Torque PBS commands] (Adaptive Computing/Cluster Resources)
+
* [http://www.pgroup.com/doc/pgicudaforug.pdf CUDA Fortran Programming Guide and Reference]
−
* [http://support.scinet.utoronto.ca/Manuals/LL-usingandadministering.pdf LoadLeveler: using & administering] (IBM)
+
* [http://www.pgroup.com/doc/openACC_gs.pdf OpenACC Getting Started Guide]<br>(Note: $PGI/linux86-64/12.5/doc contains a newer version.)
−
|}
+
 
 +
==Scheduler (Adaptive Computing/Cluster Resources)==
 +
* [http://docs.adaptivecomputing.com/mwm/archive/7-0/help.htm Moab workload manager]
 +
* [http://docs.adaptivecomputing.com/mwm/archive/7-0/Content/a.gcommandoverview.html Moab commands]
 +
* [http://docs.adaptivecomputing.com/torque/archive/4-0-1/help.htm Torque resource manager]  
 +
* [http://docs.adaptivecomputing.com/torque/archive/4-0-1/help.htm#topics/12-appendices/commandsOverview.htm Torque PBS commands]
 +
 
 +
==DDT Debugger (Allinea)==
 +
* [http://www.allinea.com/downloads/userguide.pdf Distributed Debugging Tool User Guide]

Latest revision as of 13:30, 9 August 2018


WARNING: SciNet is in the process of replacing this wiki with a new documentation site. The links below may still contain valuable information, but for current information, please go to https://docs.scinet.utoronto.ca

Training material

For upcoming classes, see our Training and Education website!

SciNet Basics

Linux

  • Linux Tutorial (warning: the last part of this tutorial is specific to the csh shell, whereas SciNet uses the bash shell).
  • Linux Command Line: A Primer (June 2012) Slides, Files
  • Introduction to the Linux Shell, SciNet, Mar 2012: Slides and Data files

Batch job management

Programming

General

Fortran

C++

ro/en-us/fortran/lin/compiler_f/index.htm Intel Fortran compiler]

C

Cilk Plus

Hadoop

Perl

Python

R

Lua

Parallel Programming

GPU Computing

Performance Tuning

Debugging

Math libraries (BLAS, LAPACK, FFT)

I/O

Infiniband Networking

Visualization


Applications

Currently, information on the following applications is available:

Development Systems

KNL

Manuals

Intel compilers and libraries (GPC)

IBM compilers and libraries (TCS/P7)

PGI compilers (ARC)

Scheduler (Adaptive Computing/Cluster Resources)

DDT Debugger (Allinea)

Retrieved from ‘https://oldwiki.scinet.utoronto.ca/index.php?title=Knowledge_Base:_Tutorials_and_Manuals&oldid=9382’