Half Day Parallel I/O Course - February 2013

2013-03-22T21:29:27Z

Esalesde: Fix direct links to videos.

== Goal ==

This half-day course given on February 27, 2013 at SciNet, was on parallel I/O techniques. Doing large-scale computation can be much easier than recording the results of the computation. In this half day course, a variety of strategies for doing large-scale I/O were discussed either with parallel or large numbers of serial jobs; we then focused on technologies for truly Parallel I/O, including MPI-IO, HDF5, and NetCDF.

== Slides (PDF) ==
# [[Media:Parallel_io_intro.pdf | Introduction to Parallel I/O]]
# [[Media:Parallel_io_course_mpi_io.pdf | Parallel I/O with MPI (MPI-IO)]]
# [[Media:Netcdfhdf5.pdf | Parallel I/O with NetCDF and HDF5]]

== Videos ==
# [http://support.scinet.utoronto.ca/CourseVideo/ParallelIOcourse/pario-intro/pario-intro.html Introduction to Parallel I/O] [[File:Morning_intro.png|180px|Click for Video|link=http://support.scinet.utoronto.ca/CourseVideo/ParallelIOcourse/pario-intro/pario-intro.html]] [http://support.scinet.utoronto.ca/CourseVideo/ParallelIOcourse/pario-intro/pario-intro.mp4 direct link]
# [http://support.scinet.utoronto.ca/CourseVideo/ParallelIOcourse/pario-mpiio/pario-mpiio.html Parallel I/O with MPI (MPI-IO)] [[File:Morning_intro.png|180px|Click for Video|link=http://support.scinet.utoronto.ca/CourseVideo/ParallelIOcourse/pario-mpiio/pario-mpiio.html]] [http://support.scinet.utoronto.ca/CourseVideo/ParallelIOcourse/pario-mpiio/pario-mpiio.mp4 direct link]
# [http://support.scinet.utoronto.ca/CourseVideo/ParallelIOcourse/pario-netcdfhdf5/pario-netcdfhdf5.html Parallel I/O with NetCDF and HDF5] (apologies for the poor sound quality) [[File:Morning_intro.png|180px|Click for Video|link=http://support.scinet.utoronto.ca/CourseVideo/ParallelIOcourse/pario-netcdfhdf5/pario-netcdfhdf5.html]] [http://support.scinet.utoronto.ca/CourseVideo/ParallelIOcourse/pario-netcdfhdf5/pario-netcdfhdf5.mp4 direct link]

== Code ==
* [[Media:ScinetCourseParIO.tgz | Hands-on example code from /scinet/course/parIO/]]

Python

2011-06-22T01:38:32Z

Esalesde: Remove duplicate line

[http://www.python.org/ Python] is programing language that continues to grow in popularity for scientific computing. It is very fast to write code in, but the software that results is much much slower than C or Fortran; one should be wary of doing too much compute-intensive work in Python. There is a dizzying amount of documentation available for programming in Python on the [http://python.org/ Python.org webpage]; and an excellent set of material for teaching scientists to program in Python is available at the [http://software-carpentry.org/4_0/python/ Software Carpentry homepage].

__FORCETOC__

== Python on the GPC ==

At the time of writing, we have two versions of python installed on GPC - the default, version 2.6, and the newest in the 2.7 release series, 2.7.1. To use either, you will have to have lines like
<source lang=bash>
module load gcc python
</source>
or
<source lang=bash>
module load gcc python/2.7.1 intel/intel-v12.0.0.084
</source>

in your <tt>.bashrc</tt>.

== Modules installed system-wide ==

Many optional packages are available for Python which greatly extend the language adding important new functionality. Those packages which are likely to be important to all of our users — eg, [http://numpy.scipy.org/ NumPy], [http://www.scipy.org/ SciPy], and [http://matplotlib.sourceforge.net/ Matplotlib] are installed system-wide.

Below is a list of the packages currently installed system-wide.

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Module
!{{Hl2}}| Version
!{{Hl2}}| Comments
|-
|[http://www.scipy.org/ SciPy]
| 0.7.1 (Python 2.6.2); 0.9.0 (Python 2.7.1)
| An Open-source software for mathematics, science, and engineering. Version in Python 2.7.1 is linked against very fast MKL numerical libraries.
|-
|[http://numpy.scipy.org/ NumPy]
| 1.3.0 (Python 2.6.2); 1.5.1 (Python 2.7.1)
| NumPy is the fundamental package needed for scientific computing with Python. Contains fast arrays, tools for integrating C/C++ and Fortran code, linear algebra solvers, etc. SciPy is built on top of NumPy.
|-
| [http://mpi4py.scipy.org/ mpi4py]
| 1.2.1 (Python 2.6.2); 1.2.2 (Python 2.7.1)
| A pythonic interface to mpi. Available with openmpi; must load an openmpi module for this to work.
|-
| [http://www.scipy.org/SciPyPackages/NumExpr Numexpr]
| 1.3.1
| Fast, memory-efficient elementwise operations on Numpy arrays.
|-
| [http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ ScientificPython]
| 2.8
| A collection of scientific python utilities.
|-
| [http://yt.enzotools.org/ yt]
| 1.7 (Python 2.6.2 only)
| A collection of python tools for analyzing astrophysical simulation output.
|-
| [http://ipython.scipy.org/moin/ iPython]
| 0.10
| An enhanced interactive python.
|-
| [http://matplotlib.sourceforge.net/ Matplotlib], pylab
| 0.99.0 (Python 2.6.2); 1.0.1 (Python 2.7.1)
| Matlab-like plotting for python.
|-
| [http://www.pytables.org/moin PyTables]
| 2.2.1
| Fast and efficient access to HDF5 files (and HDF5-format NetCDF4 files.) Requires the <tt>hdf5/184-p1-v18-serial-gcc</tt> module to be loaded.
|-
| [http://code.google.com/p/netcdf4-python/ NetCDF4-python]
| 0.9.3 (Python 2.6.2); 0.9.4 (Python 2.7.1)
| Python interface to NetCDF4 files. Requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module to be loaded.
|-
| [http://www.pyngl.ucar.edu/Nio.shtml pyNIO]
| 1.3.0
| Another Python interface to NetCDF4 files; again, requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module.
|-
| [http://pysvn.tigris.org/ PySVN]
| 1.7.1
| Python interface to the svn version control system. Requires the <tt>svn</tt> module to be loaded.
|-
| [http://mercurial.selenic.com/ Mercurial]
| 1.3.1 (Python 2.6.2); 1.8.2 (Python 2.7.1)
| A distributed version-control system written in Python.
|-
| [http://cython.org/ Cython]
| 0.12.1 (Python 2.6.2); 0.14.1 (Python 2.7.1)
| Cython is a compiler which compiles Python-like code files to C code and allows them to be easily called from Python.
|-
| [http://code.google.com/p/python-nose/ nose]
| 0.11.2
| A unit-testing framework for python.
|-
| [http://pypi.python.org/pypi/setuptools setuptools]
| 0.6c11
| Enables easy installation of new python modules
|-
|}

== Installing your own Python Modules ==

Python provides an easy way for users to install the libraries they need in their home directories rather than having them installed system-wide. There are so many optional packages for Python people could potentially want (see e.g. http://pypi.python.org/pypi), that we recommend users install these additional packages locally in their home directories. This is almost certainly the easiest way to deal with the wide range of packages, ensure they're up to date, and ensure that users' package choices don't conflict.

To install your own Python modules, follow the instructions below. Where the instructions say <tt>python2.X</tt>, type <tt>python2.6</tt> or <tt>python2.7</tt> depending on the version of python you are using.

* First, create a directory in your home directory, <tt>${HOME}/lib/python2.X/site-packages</tt>, where the packages will go.
* Next, in your <tt>.bashrc</tt>, *after* you <tt>module load python</tt> and in the "GPC" section, add the following line:
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages/
</source>

* Re-load the modified .bashrc by typing <tt>source ~/.bashrc</tt>.

* Now, if it's a standard python package and instructions say that you can use easy_intall to install it,
** install with the following command. where <tt>packagename</tt> is the name of the package you are installing:
<source lang=bash>
easy_install --prefix=${HOME} -O1 [packagename]
</source>

** Continue doing this until all of the packages you need to install are successfully installed.
* If easy_install isn't an option for your package, and the installation instructions instead talk about downloading a file and using <tt>python setup.py install</tt> then instead:
** Download the relevant files
** You will probably have to uncompress and untar them: <tt>tar -xzvf packagename.tgz</tt> or <tt>tar -xjvf packagename.bz2</tt>.
** cd into the newly created directory, and run
<source lang=bash>
python setup.py install --prefix=${HOME}
</source>

* Now, the install process may have added some .egg files or directories to your path. For each .egg directory, add that to your python path as well in your .bashrc, in the same place as you had updated PYTHONPATH before: eg,
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages:${HOME}/lib/python2.X/site-packages/packagename1-x.y.z-yy2.X.egg:${HOME}/lib/python2.X/site-packages/packagename2-a.b.c-py2.X.egg
</source>

* You should now be done! Now, re-source your .bashrc and test your new python modules.

* In order to keep your .bashrc relatively uncluttered, and to avoid potential conflicts among software modules, we recommend that users create their own modules (for the "module" system, not specifically python modules).

[[Brian|Here]] is an example module for the [[Brian]] package, including instructions for the installation of the python [[Brian]] package itself.

oldwiki.scinet.utoronto.ca - User contributions [en-gb]

Half Day Parallel I/O Course - February 2013

Python