oldwiki.scinet.utoronto.ca - User contributions [en-gb]

HPSS

2012-05-28T19:42:48Z

Knecht: Changes to exit status in examples.

= '''High Performance Storage System''' =

The High Performance Storage System ([http://www.hpss-collaboration.org/index.shtml HPSS]) is a tape-backed hierarchical storage system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data can be returned to the active GPFS filesystem when it is needed.

Because this system is intended for large data storage, it is accessible only to groups who have been awarded storage space at SciNet beyond 5TB in the yearly RAC resource allocation round.

Access and transfer of data into and out of HPSS is done under the control of the user, whose interaction is expected to be scripted and submitted as a batch job, using one or more of the following utilities:
* [http://www.mgleicher.us/GEL/hsi HSI] is a client with an ftp-like functionality which can be used to archive and retrieve large files. It is also useful for browsing the contents of HPSS.
* [http://www.mgleicher.us/GEL/htar HTAR] is a utility that creates tar formatted archives directly into HPSS. It also creates a separate index file (.idx) that can be accessed and browsed quickly.
* [https://support.scinet.utoronto.ca/wiki/index.php/ISH ISH] is a TUI utility that can perform an inventory of the files and directories in your tarballs.

== '''Why should I use and trust HPSS?''' ==
* 10+ years history, used by about 40 facilities in the [http://www.top500.org “Top 500”] HPC list
* very reliable, data redundancy and data insurance built-in.
* highly scalable, reasonable performance at SciNet - Ingest: ~24 TB/day, Recall: ~12 TB/day (aggregated)
* HSI/HTAR clients also very reliable and used on several HPSS sites. ISH was written at SciNet.
* [[Media:HPSS_rationale_SNUG.pdf|HPSS fits well with the Storage Capacity Expansion Plan at SciNet]] (pdf presentation)

== '''Guidelines''' ==
* Expanded storage capacity is provided on tape -- a media that is not suited for storing small files. Files smaller than ~200MB should be grouped into tarballs with tar or htar.
* Optimal performance for aggregated transfers and allocation on tapes is obtained with tarballs of size around 100GB.
* We strongly urge that you use the sample scripts we are providing as the basis for your job submissions.
* Make sure to check the application's exit code and returned logs for errors after any data transfer or tarball creation process

== '''Access Through the Queue System''' ==
All access to the archive system is done through the [[Moab|GPC queue system]].

* Job submissions should be done to the 'archive' queue
* Short jobs are limited to 1H walltime by default. Long jobs (> 1H) are limited to 72H walltime.
* Users are limited to only 1 long job and 1 short job at the same time.
* There can only be 5 long jobs running at any given time overall. Remaining submissions will be placed on hold for the time being. So far we have not seen a need for overall limit on short jobs.

The status of pending jobs can be monitored with showq specifying the archive queue:
<pre>
showq -w class=archiveshort
OR
showq -w class=archivelong
</pre>

=== Scripted File Transfers ===
File transfers in and out of the HPSS should be scripted into jobs and submitted to the ''archive'' queue. See generic example below.
<source lang="bash">
#!/bin/bash
#PBS -l walltime=72:00:00
#PBS -q archive
#PBS -N htar_create_tarball_in_hpss
#PBS -j oe
#PBS -m e

echo "Creating a htar of finished-job1/ directory tree into HPSS"

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

cd $SCRATCH/workarea/
htar -cpf $ARCHIVE/finished-job1.tar finished-job1/
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HTAR returned non-zero code.'
/scinet/gpc/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>
'''Note:''' Always trap the execution of your jobs for abnormal terminations, and be sure to return the exit code

=== Job Dependencies ===

Typically data will be recalled to /scratch when it is needed for analysis. Job dependencies can be constructed so that analysis jobs wait in the queue for data recalls before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the archive recalling job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency (lookup [https://support.scinet.utoronto.ca/wiki/index.php/HPSS#Sample_data_recall data-recall.sh samples]):
<pre>
gpc04 $ qsub $(qsub data-recall.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-recalled-data.sh
</pre>

== '''HTAR''' ==
''' Please aggregate small files (<~200MB) into tarballs or htar files. '''

HTAR is a utility that is used for aggregating a set of files and directories, by using a sophisticated multithreaded buffering scheme to write files directly from GPFS into HPSS, creating an archive file that conforms to the POSIX TAR specification, thereby achieving a high rate of performance. HTAR does not do gzip compression, however it already has a built-in checksum algorithm.

'''Caution'''
* Files larger than 68 GB cannot be stored in an HTAR archive. If you attempt to start a transfer with any files larger than 68GB the whole HTAR session will fail, and you'll get a notification listing all those files, so that you can transfer them with HSI. unintentionally overwriting the htar destination file in HPSS
* Files with pathnames too long will be skipped (greater than 100 characters), so as to conform with TAR protocol [[(POSIX 1003.1 USTAR)]] -- Note that the HTAR will erroneously indicate success, however will produce exit code 70. For now, you can check for this type of error by "grep Warning my.output" after the job has completed.
* Unlike with cput/cget in HSI, "prompt before overwrite", this is not the default with (h)tar. Be careful not to unintentionally overwrite a previous htar destination file in HPSS. There could be a similar situation when extracting material back into GPFS and overwriting the originals. Be sure to double-check the logic in your scripts.
* Check the HTAR exit code and log file before removing any files from the GPFS active filesystems.

=== HTAR Usage ===
* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, and preserve mask attributes (-p), enter:
<pre>
htar -cpf files.tar file1 file2
OR
htar -cpf $ARCHIVE/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cpf subdirA.tar subdirA
</pre>

* To extract all files from the archive file called ''proj1.tar'' in HPSS into the ''project1/src'' directory in GPFS, and use the time of extraction as the modification time, enter:
<pre>
htar -xpmf proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

==== Sample tarball create ====
<source lang="bash">
#!/bin/bash
#PBS -l walltime=72:00:00
#PBS -q archive
#PBS -N htar_create_tarball_in_hpss
#PBS -j oe
#PBS -m e

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be /archive/$(id -gn)/$(whoami)/

DEST=$ARCHIVE/finished-job1.tar

# htar WILL overwrite an existing file with the same name so check beforehand.
if [ -f $DEST ];
then
echo 'File $DEST already exists. Nothing has been done'
exit 1
fi

cd $SCRATCH/workarea/
htar -cpf $DEST finished-job1/
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HTAR returned non-zero code.'
/scinet/gpc/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

'''Note:''' If you attempt to start a transfer with any files larger than 68GB the whole HTAR session will fail, and you'll get a notification listing all those files, so that you can transfer them with HSI.
<pre>
----------------------------------------
INFO: File too large for htar to handle: finished-job1/file1 (86567185745 bytes)
INFO: File too large for htar to handle: finished-job1/file2 (71857244579 bytes)
ERROR: 2 oversize member files found - please correct and retry
ERROR: [FATAL] error(s) generating filename list
HTAR: HTAR FAILED
###WARNING htar returned non-zero exit status
</pre>

==== Sample tarball list ====
<source lang="bash">
#!/bin/bash
#PBS -l walltime=72:00:00
#PBS -q archive
#PBS -N htar_list_tarball_in_hpss
#PBS -j oe
#PBS -m e

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

DEST=$ARCHIVE/finished-job1.tar

htar -tvf $DEST
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HTAR returned non-zero code.'
/scinet/gpc/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

==== Sample tarball extract ====
<source lang="bash">
#!/bin/bash
#PBS -l walltime=72:00:00
#PBS -q archive
#PBS -N htar_extract_tarball_from_hpss
#PBS -j oe
#PBS -m e

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

cd $SCRATCH/recalled-from-hpss
htar -xpmf $ARCHIVE/finished-job1.tar
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HTAR returned non-zero code.'
/scinet/gpc/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

== '''HSI''' ==

HSI may be the primary client with which some users will interact with HPSS. It provides an ftp-like interface for archiving and retrieving tarballs or [https://support.scinet.utoronto.ca/wiki/index.php/HPSS#Sample_transferring_directories directory trees]. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents in HPSS. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally saves or replaces a HPSSpath file to GPFSpath if the GPFS version is new or has been updated
cput [options] GPFSpath [: HPSSpath]
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to GPFS only if a GPFS version does not already exist.
cget [options] [GPFSpath :] HPSSpath
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of HPSS.
|-
| lcd,lls
| ''Local'' commands to GPFS
|}

*There are 3 distinctions about HSI that you should keep in mind, and that can generate a bit of confusion when you're first learning how to use it:
** HSI doesn't currently support renaming directories paths during transfers on-the-fly, therefore the syntax for cput/cget may not work as one would expect in some scenarios, requiring some workarounds.
** HSI has an operator ":" which separates the GPFSpath and HPSSpath, and must be surrounded by whitespace (one or more space characters)
** The order for referring to files in HSI syntax is different from FTP. In HSI the general format is always the same, GPFS first, HPSS second, cput or cget:
<pre>
GPFSfile : HPSSfile
</pre>
For example, when using HSI to store the tarball file from GPFS into HPSS, then recall it to GPFS, the following commands could be used:
<pre>
cput tarball-in-GPFS : tarball-in-HPSS
cget tarball-recalled : tarball-in-HPSS
</pre>

unlike with FTP, where the following syntax would be used:
<pre>
put tarball-in-GPFS tarball-in-HPSS
get tarball-in-HPSS tarball-recalled
</pre>
* Simple commands can be executed on a single line.
<pre>
hsi "mkdir LargeFilesDir; cd LargeFilesDir; cput tarball-in-GPFS : tarball-in-HPSS"
</pre>

* More complex sequences can be performed using an except such as this:
<pre>
hsi <<EOF
mkdir LargeFilesDir
cd LargeFilesDir
cput tarball-in-GPFS : tarball-in-HPSS
lcd $SCRATCH/LargeFilesDir2/
cput -Ruph *
end
EOF
</pre>

* The commands below are equivalent, but we recommend that you always use full path, and organize the contents of HPSS, where the default HSI directory placement is $ARCHIVE:
<pre>
hsi cput tarball
hsi cput tarball : tarball
hsi cput $SCRATCH/tarball : $ARCHIVE/tarball
</pre>

* There are no known issues renaming files on-the-fly:
<pre>
hsi cput $SCRATCH/tarball1 : $ARCHIVE/tarball2
hsi cget $SCRATCH/tarball3 : $ARCHIVE/tarball2
</pre>

* However the syntax forms such as the ones below will fail, since they rename the directory paths.
<pre>
hsi cput -Ruph $SCRATCH/LargeFilesDir : $ARCHIVE/LargeFilesDir (FAILS)
OR
hsi cget -Ruph $SCRATCH/LargeFilesDir : $ARCHIVE/LargeFilesDir2 (FAILS)
OR
hsi cput -Ruph $SCRATCH/LargeFilesDir/* : $ARCHIVE/LargeFilesDir2 (FAILS)
OR
hsi cget -Ruph $SCRATCH/LargeFilesDir : $ARCHIVE/LargeFilesDir (FAILS)
</pre>

One workaround is the following 2-steps process, where you do a "lcd " in GPFS first, and recursively transfer the whole directory (-R), keeping the same name. You may use '-u' option to resume a previously disrupted session, and the '-p' to preserve timestamp, and '-h' to keep the links.
<pre>
hsi <<EOF
lcd $SCRATCH
cget -Ruph LargeFilesDir
end
EOF
</pre>

Another workaround is do a "lcd" into the GPFSpath first and a "cd" in the HPSSpath, but transfer the files individually with the '*' wild character. This option lets you change the directory name:
<pre>
hsi <<EOF
lcd $SCRATCH/LargeFilesDir
mkdir $ARCHIVE/LargeFilesDir2
cd $ARCHIVE/LargeFilesDir2
cput -Ruph *
end
EOF
</pre>

=== Documentation ===
Complete documentation on HSI is available from the Gleicher Enterprises links below. You may peruse those links and come with alternative syntax forms. You may even be already familiar with HPSS/HSI from other HPC facilities, that may or not have procedures similar to ours. HSI doesn't always work as expected when you go outside of our recommended syntax, so '''we strongly urge that you use the sample scripts we are providing as the basis''' for your job submissions
* [http://www.mgleicher.us/GEL/hsi/ HSI Introduction]
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help hsi help]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]
'''Note:''' HSI returns the highest-numbered exit code, in case of multiple operations in the same hsi session. You may use '/scinet/gpc/bin/exit2msg $status' to translate those codes into intelligible messages

=== Typical Usage Scripts===
The most common interactions will be ''putting'' data into HPSS, examining the contents (ls,ish), and ''getting'' data back onto GPFS for inspection or analysis.

==== Sample '''data offload''' ====
<source lang="bash">
#!/bin/bash

# This script is named: data-offload.sh
#PBS -l walltime=72:00:00
#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -me

trap "echo 'Job script not completed';exit 129" TERM INT
# individual tarballs already exist

/usr/local/bin/hsi -v <<EOF1
mkdir put-away
cd put-away
cput $SCRATCH/workarea/finished-job1.tar.gz : finished-job1.tar.gz
end
EOF1
status=$?
if [ ! $status == 0 ];then
echo 'HSI returned non-zero code.'
/scinet/gpc/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi

/usr/local/bin/hsi -v <<EOF2
mkdir put-away
cd put-away
cput $SCRATCH/workarea/finished-job2.tar.gz : finished-job2.tar.gz
end
EOF2
status=$?
if [ ! $status == 0 ];then
echo 'HSI returned non-zero code.'
/scinet/gpc/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi

trap - TERM INT
</source>

'''Note:''' as in the above example, we recommend that you capture the (highest-numbered) exit code for each hsi session independently. And remember, you may improve your exit code verbosity by adding the excerpt below to your scripts:
<source lang="bash">
if [ ! $status == 0 ];then
echo 'HSI returned non-zero code.'
/scinet/gpc/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

==== Sample '''data list''' ====
A very trivial way to list the contents of HPSS would be to just submit the HSI 'ls' command.
<source lang="bash">
#!/bin/bash

# This script is named: data-list.sh
#PBS -l walltime=1:00:00
#PBS -q archive
#PBS -N hpss_ls
#PBS -j oe
#PBS -me

/usr/local/bin/hsi -v <<EOF
cd put-away
ls -R
end
EOF
</source>
''Warning: if you have a lot of files, the ls command will take a long time to complete. For instance, about 400,000 files can be listed in about an hour. Adjust the walltime accordingly, and be on the safe side.''

However, we provide a much more useful and convenient way to explore the contents of HPSS with the inventory shell [[ISH]]. This example creates an index of all the files in a user's portion of the namespace. The list is placed in the directory /home/$(whoami)/.ish_register that can be inspected from the gpc-devel nodes.
<source lang="bash">
#!/bin/bash

# This script is named: data-list.sh
#PBS -l walltime=1:00:00
#PBS -q archive
#PBS -N hpss_index
#PBS -j oe
#PBS -me

INDEX_DIR=$HOME/.ish_register
if ! [ -e "$INDEX_DIR" ]; then
mkdir -p $INDEX_DIR
fi

export ISHREGISTER="$INDEX_DIR"
/scinet/gpc/bin/ish hindex
</source>
''Note: the above warning on collecting the listing for many files applies here too.''

This index can be browsed or searched with ISH on the development nodes.
<source lang="bash">
gpc-f104n084-$ /scinet/gpc/bin/ish ~/.ish_register/hpss.igz
[ish]hpss.igz> help
</source>

ISH is a powerful tool that is also useful for creating and browsing indices of tar and htar archives, so please look at the [[ISH|documentation]] or built in help.

==== Sample '''data recall''' ====
<source lang="bash">
#!/bin/bash

# This script is named: data-recall.sh
#PBS -l walltime=72:00:00
#PBS -q archive
#PBS -N recall_files
#PBS -j oe
#PBS -me

trap "echo 'Job script not completed';exit 129" TERM INT

mkdir -p $SCRATCH/recalled-from-hpss

# individual tarballs previously organized in HPSS inside the put-away-on-2010/ folder
hsi -v << EOF
cget $SCRATCH/recalled-from-hpss/Jan-2010-jobs.tar.gz : $ARCHIVE/put-away-on-2010/Jan-2010-jobs.tar.gz
cget $SCRATCH/recalled-from-hpss/Feb-2010-jobs.tar.gz : $ARCHIVE/put-away-on-2010/Feb-2010-jobs.tar.gz
end
EOF
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/gpc/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

We should emphasize that a single ''cget'' of multiple files (rather than several separate gets) allows HSI to do optimization, as in the following example:
<source lang="bash">
#!/bin/bash

# This script is named: data-recall.sh
#PBS -l walltime=72:00:00
#PBS -q archive
#PBS -N recall_files_optimized
#PBS -j oe
#PBS -me

trap "echo 'Job script not completed';exit 129" TERM INT
mkdir -p $SCRATCH/recalled-from-hpss

# individual tarballs previously organized in HPSS inside the put-away-on-2010/ folder
hsi -v << EOF
lcd $SCRATCH/recalled-from-hpss/
cd $ARCHIVE/put-away-on-2010/
cget Jan-2010-jobs.tar.gz Feb-2010-jobs.tar.gz
end
EOF
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/gpc/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

==== Sample '''transferring directories''' ====
Remember, it's not possible to rename directories on-the-fly:
<pre>
hsi cget -Ruph $SCRATCH/LargeFiles-recalled : $ARCHIVE/LargeFiles (FAILS)
</pre>

One workaround is transfer the whole directory (and sub-directories) recursively:
<source lang="bash">
#!/bin/bash

# This script is named: data-recall.sh
#PBS -l walltime=72:00:00
#PBS -q archive
#PBS -N recall_directories
#PBS -j oe
#PBS -me

trap "echo 'Job script not completed';exit 129" TERM INT

mkdir -p $SCRATCH/recalled

hsi -v << EOF
lcd $SCRATCH/recalled
cd $ARCHIVE/
cget -Ruph LargeFiles
end
EOF
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/gpc/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

Another workaround is to transfer files and subdirectories individually with the "*" wild character:
<source lang="bash">
#!/bin/bash

# This script is named: data-recall.sh
#PBS -l walltime=72:00:00
#PBS -q archive
#PBS -N recall_directories
#PBS -j oe
#PBS -me

trap "echo 'Job script not completed';exit 129" TERM INT

mkdir -p $SCRATCH/LargeFiles-recalled

hsi -v << EOF
lcd $SCRATCH/LargeFiles-recalled
cd $ARCHIVE/LargeFiles
cget -Ruph *
end
EOF
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/gpc/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help''']

== '''[[ISH|ISH]]''' ==
=== [[ISH|Documentation and Usage]] ===

== '''File and directory management''' ==
=== Moving/renaming ===
* you may use 'mv' or 'cp' in the same way as the linux version.
<source lang="bash">
#!/bin/bash
#PBS -l walltime=72:00:00
#PBS -q archive
#PBS -N deletion_script
#PBS -j oe
#PBS -m e

echo "HPSS file and directory management"

trap "echo 'Job script not completed';exit 129" TERM INT

/usr/local/bin/hsi -v <<EOF1
mkdir $ARCHIVE/2011
mv $ARCHIVE/oldjobs $ARCHIVE/2011
cp -r $ARCHIVE/almostfinished/*done $ARCHIVE/2011
end
EOF1
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/gpc/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

=== Deletions ===
==== Recommendations ====
* Be careful with the use of 'cd' commands to non-existing directories before the 'rm' command. Results may be unpredictable
* Avoid the use of the stand alone wild character '''*'''. If necessary, whenever possible have it bound to common patterns, such as '*.tmp', so to limit unintentional mis-happens
* Avoid using relative paths, even the env variable $ARCHIVE. Better to explicitly expand the full paths in your scripts
* Avoid using recursive/looped deletion instructions on $SCRATCH contents from the archive job scripts. Even on $ARCHIVE contents, it may be better to do it as an independent job submission, after you have verified that the original ingestion into HPSS finished without any issues.

==== Typical example ====
<source lang="bash">
#!/bin/bash
#PBS -l walltime=72:00:00
#PBS -q archive
#PBS -N deletion_script
#PBS -j oe
#PBS -m e

echo "Deletion of an outdated directory tree into HPSS"

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that the initial directory in HPSS ($ARCHIVE) has the path explicitly expanded

/usr/local/bin/hsi -v <<EOF1
rm /archive/s/scinet/pinto/*.tmp
rm -R /archive/s/scinet/pinto/obsolete
end
EOF1
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/gpc/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

==== Deleting with an interactive HSI session ====
* You may feel more comfortable acquiring an interactive shell, starting an HSI session and proceeding with your deletions that way. Keep in mind, you're restricted to 1H.

<pre>
gpc-f103n084-$ qsub -q archive -I
qsub: waiting for job 11611291.gpc-sched to start
qsub: job 11611291.gpc-sched ready

----------------------------------------
Begin PBS Prologue Mon May 28 13:15:28 EDT 2012 1338225328
Job ID: 11611291.gpc-sched
Username: pinto
Group: scinet
Nodes: gpc-archive01
End PBS Prologue Mon May 28 13:15:28 EDT 2012 1338225328
----------------------------------------
hpss-archive01-$ hsi
******************************************************************
* Welcome to HPSS@SciNet - High Perfomance Storage System *
* *
* Contact Information: support@scinet.utoronto.ca *
* NOTE: do not transfer SMALL FILES with HSI. Use HTAR instead *
* CHECK THE INTEGRITY OF YOUR TARBALLS *
******************************************************************
Username: pinto UID: 10010 Acct: 10010(10010) Copies: 2 Firewall: off [hsi.4.0.1 Thu Mar 22 11:44:03 EDT 2012]
[HSI]/archive/s/scinet/pinto-> rm -R junk

</pre>

== '''HPSS for the 'Watchmaker' ''' ==
=== Efficient alternative to htar ===
<source lang="bash">
#!/bin/bash
#PBS -l walltime=72:00:00
#PBS -q archive
#PBS -N tar_create_tarball_in_hpss_with_hsi_by_piping
#PBS -j oe
#PBS -m e

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

# When using a pipeline like this
set -o pipefail

# to put
tar -c $SCRATCH/mydir | hsi cput - : $ARCHIVE/mydir.tar
status=$?
if [ ! $status == 0 ]; then
echo 'TAR+HSI+piping returned non-zero code.'
/scinet/gpc/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi

# to immediately generate an index
ish hindex $ARCHIVE/mydir.tar
status=$?
if [ ! $status == 0 ]; then
echo 'ISH returned non-zero code.'
/scinet/gpc/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi

# to get
#cd $SCRATCH
#hsi cget - : $ARCHIVE/mydir.tar | tar -xv
#status=$?
# if [ ! $status == 0 ]; then
# echo 'TAR+HSI+piping returned non-zero code.'
# /scinet/gpc/bin/exit2msg $status
# exit $status
#else
# echo 'TRANSFER SUCCESSFUL'
#fi

trap - TERM INT

</source>
'''Notes:'''
* Combining commands in this fashion, besides being HPSS-friendly, should not be that noticeably slower than the recursive put with HSI that stores each file one by one. However, reading the files back from tape in this format will be many times faster. It would also overcome the current 68GB limit on the size of stored files that we have with htar.
* To top things off, we recommend indexing with ish (in the same script) immediately after the tarball creation , while it resides in the HPSS cache. It would be as if htar was used.
* To ensure that an error at any stage of the pipeline shows up in the returned status use: ''set -o pipefail'' (The default is to return the status of the last command in the pipeline and this is not what you want.)

=== Content Verification ===
Specifies that HTAR should generate CRC checksums when creating the archive.

<source lang="bash">
#!/bin/bash
#PBS -l walltime=72:00:00
#PBS -q archive
#PBS -N htar_create_tarball_in_hpss_with_checksum_verification
#PBS -j oe
#PBS -m e

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

cd $SCRATCH/workarea

# to put
htar -cpf $ARCHIVE/finished-job1.tar -Hcrc -Hverify=1 finished-job1/

# to get
#mkdir $SCRATCH/verification
#cd $SCRATCH/verification
#htar -Hcrc -xvpmf $ARCHIVE/finished-job1.tar

status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HTAR returned non-zero code.'
/scinet/gpc/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

=== Sample '''verify checksum''' ===
This will checksum the contents of the HPSSpath against the original GPFSpath after the transfer has finished.
<source lang="bash">
#!/bin/bash
#PBS -l walltime=72:00:00
#PBS -q archive
#PBS -N checksum_verified_transfer
#PBS -j oe
#PBS -me

thefile=<GPFSpath>
storedfile=<HPSSpath>

# Generate checksum on fly using a named pipe so that file is only read from GPFS once
mkfifo /tmp/NPIPE
cat $thefile | tee /tmp/NPIPE | hsi -q put - : $storedfile &
pid=$!
md5sum /tmp/NPIPE |tee /tmp/$fname.md5
rm -f /tmp/NPIPE

# Check the exit code of the HSI process
wait $pid
status=$?

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/gpc/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi

# change filename to stdin in checksum file
sed -i.1 "s+/tmp/NPIPE+-+" /tmp/$fname.md5

# verify checksum
hsi -q get - : $storedfile | md5sum -c /tmp/$fname.md5
status=$?

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/gpc/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

[[Data Management|BACK TO Data Management]]

HPSS

2011-06-27T20:42:56Z

Knecht: /* Other HSI Examples */

= '''High Performance Storage System''' =

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The High Performance Storage System ([http://www.hpss-collaboration.org/index.shtml HPSS]) is a tape-backed hierarchical storage system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Access and transfer of data into and out of HPSS is done under the control of the user, whose interaction is expected to be scripted and submitted as a batch job, using one or more of the following utilities:
* [http://www.mgleicher.us/GEL/hsi HSI] is a client with an ftp-like interface which can be used to archive and retrieve large files. It is also useful for browsing the contents of HPSS.
* [http://www.mgleicher.us/GEL/htar HTAR] is a utility that creates tar formatted archives directly into HPSS. It also creates a separate index file (.idx) that can be accessed quickly.
* [https://support.scinet.utoronto.ca/wiki/index.php/ISH ISH] is a TUI utility that can perform an inventory of the files and directories in your tarballs.

== '''Why should I use and trust HPSS?''' ==
* 10+ years history, used by 50+ facilities in the “Top 500” HPC list
* very reliable, data redundancy and data insurance built-in.
* highly scalable, reasonable performance at SciNet - Ingest: ~12 TB/day, Recall: ~24 TB/day (aggregated)
* HSI/HTAR clients also very reliable and used on several HPSS sites. ISH was written at SciNet.
* [[Media:HPSS_rational.pdf|HPSS fits well with the Storage Capacity Expansion Plan at SciNet]] (pdf presentation)

== '''Guidelines''' ==
* Expanded storage capacity is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into tarballs with tar or htar.
* The maximum size of a file that can be transferred into the repository is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application's exit code and the returned log file for errors after all data transfers and any tarball creation process.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== '''Access Through the Queue System''' ==
All access to the archive system is done through the [[Moab|GPC queue system]].
=== Scripted File Transfers ===
File transfers in and out of the HPSS should be scripted into jobs and submitted to the ''archive'' queue. See HSI example below.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_put_file_in_hpss
#PBS -j oe
#PBS -me

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
status=$?
if [ ! $status == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
exit $status
</pre>

The status of pending jobs can be monitored with showq specifying the archive queue:
<pre>
showq -w class=archive
</pre>

=== Recalling Data for Analysis ===

Typically, data will be recalled to the /scratch file system when it is needed for analysis. Job dependencies can be used to have analysis jobs wait in the queue for data recalls before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency:
<pre>
gpc04 $ qsub $(qsub data-recall.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-recalled-data.sh
</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with HPSS. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents in HPSS. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the file does not already exist in HPSS
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local storage only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of HPSS.
|-
| lcd,lls
| ''Local'' commands.
|}

* Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz"
</pre>
* More complex sequences can be performed using a script such as this:
<pre>
hsi <<EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
status=$?
if [ ! $status == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
exit status
</pre>

=== HSI Documentation ===
Complete documentation of HSI is available on the Gleicher Enterprises web site.
* [http://www.mgleicher.us/GEL/hsi/ HSI Introduction]
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help hsi help]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Typical Usage ===
The most common interactions will be ''putting'' data into HPSS, examining the contents (ls), and ''getting'' data back onto one of the active filesystems for inspection or analysis.

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -me

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away
cd put-away
cput /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
cput /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF
status=$?
if [ ! $status == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
exit $status
</pre>

* sample '''data list'''
A convenient way to explore the contents of HPSS is with the inventory shell [[ISH]]. This example creates an index of all the files in a user's portion of the namespace. The list is placed in the file /home/$USER/HPSSdm/hsi.igz that can be inspected from the gpc-devel nodes.
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_index
#PBS -j oe
#PBS -me

TODAY=$(date +%Y%m%d)
INDEX_DIR=/home/$USER/HPSSdm
if [[ -! -e $INDEX_DIR ]];then
mkdir $INDEX_DIR
fi

export ISHREGISTER=$HOME/HPSSdm
ish hindex
</pre>

This index can be browsed or searched with ISH on the development nodes.
<pre>
gpc-f104n084-$ ish ~/HPSSdm/hsi.igz
[ish]hsi.igz> help
</pre>

ISH is a powerful tool that is also useful for creating and browsing indices of tar and htar archives, so please look at the [[ISH|documentation]] or built in help.

* sample '''data recall'''
- This example should be modified to emphasize that a single ''cget'' of multiple files (rather than several separate gets) allows HSI to do optimization.
<pre>
#!/bin/bash

# This script is named: data-recall.sh

#PBS -q archive
#PBS -N recall
#PBS -j oe
#PBS -me

mkdir -p /scratch/$USER/recalled-from-hpss
hsi -v << EOF
cget /scratch/$USER/recalled-from-hpss/Jan-2010-jobs.tar.gz : put-away-on-2010/Jan-2010-jobs.tar.gz
cget /scratch/$USER/recalled-from-hpss/Feb-2010-jobs.tar.gz : put-away-on-2010/Feb-2010-jobs.tar.gz
EOF
status=$?

exit $status

</pre>

=== HSI vs. FTP ===
HSI syntax and usage is very similar to that of FTP. Please note the following information adapted from the HSI man page:

HSI supports several of the commonly used FTP commands, including "dir","get","ls","mdelete","mget","put","mput" and "prompt", with the following differences:

* The "dir" command is an alias for "ls" in HSI. The "ls" command supports an extensive set of options for displaying files, including wildcard pattern-matching, and the ability to recursively list a directory tree
* The "put" and "get" family of commands support recursion
* There are "conditional put" and "conditional" get commands (cput, cget)
* The syntax for renaming files during transfers with HSI is different from FTP. With HSI, the general format is always
<pre>
"local_file : hpss_file"
</pre>
and multiple such pairs may be specified on a single command line.

For example, when using HSI to store the local file "file1" as "hpss_file1" into HPSS, then retrieve it back to the local filesystem as "file1.bak", the following commands could be used:
<pre>
put file1 : hpss_file1
get file1.bak : hpss_file1
</pre>

* unlike with FTP, where the following syntax would be used:
<pre>
put file1 hpss_file1
get hpss_file1 file1.bak
</pre>
* The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.

=== Other HSI Examples ===

* Creating tar archive of C source programs and header files on the fly by piping stdout:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HSI pathnames must be surrounded by whitespace (one or more space characters)

* Retrieve the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>
* Put a subdirectory ''LargeFiles'' and all its contents recursively. You may use '-u' option to resume a previously disrupted session.
<pre>
cput -R -u LargeFiles
</pre>
* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help''']

* verify checksum
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N checksum_verified_transfer
#PBS -j oe
#PBS -me

thefile=<localpath>
storedfile=<hpsspath>

# Generate checksum on fly using a named pipe so that file is only read from GPFS once
mkfifo /tmp/NPIPE
cat $thefile | tee /tmp/NPIPE | hsi -q put - : $storedfile &
pid=$!
md5sum /tmp/NPIPE |tee /tmp/$fname.md5
rm /tmp/NPIPE

# Check the exit code of the HSI process
wait $pid
sc=$?
if [ $sc != 0 ];then
echo "File transfer failed"
exit $sc
fi

# change filename to stdin in checksum file
sed -i.1 "s+/tmp/NPIPE+-+" /tmp/$fname.md5

# verify checksum
hsi -q get - : $storedfile | md5sum -c /tmp/$fname.md5
sc=$?
if [ $sc != 0 ]; then
echo '!!! Job Failed !!!'
echo 'error=' $sc
fi
</pre>

== '''HTAR''' ==
''' Please aggregate small files (<~100MB) into tarballs or htar files. '''

HTAR is a utility that is used for aggregating a set of files and directories, by using a sophisticated multithreaded buffering scheme to write files from the local filesystem directly into HPSS, creating an archive file that conforms to the POSIX TAR specification, thereby achieving a high rate of performance.

=== '''CAUTION''' ===
* Files larger than 68 GB cannot be stored in an htar archive (you'll get an error message for the whole operation)
* HTAR archives cannot contain more than 1 million files.
* Check the HTAR exit code and log file before removing any files from the active filesystems.

=== HTAR Usage ===
* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To extract all files from the ''project1/src'' directory in the archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

HPSS

2011-06-27T20:34:56Z

Knecht: /* Other HSI Examples */

= '''High Performance Storage System''' =

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The High Performance Storage System ([http://www.hpss-collaboration.org/index.shtml HPSS]) is a tape-backed hierarchical storage system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Access and transfer of data into and out of HPSS is done under the control of the user, whose interaction is expected to be scripted and submitted as a batch job, using one or more of the following utilities:
* [http://www.mgleicher.us/GEL/hsi HSI] is a client with an ftp-like interface which can be used to archive and retrieve large files. It is also useful for browsing the contents of HPSS.
* [http://www.mgleicher.us/GEL/htar HTAR] is a utility that creates tar formatted archives directly into HPSS. It also creates a separate index file (.idx) that can be accessed quickly.
* [https://support.scinet.utoronto.ca/wiki/index.php/ISH ISH] is a TUI utility that can perform an inventory of the files and directories in your tarballs.

== '''Why should I use and trust HPSS?''' ==
* 10+ years history, used by 50+ facilities in the “Top 500” HPC list
* very reliable, data redundancy and data insurance built-in.
* highly scalable, reasonable performance at SciNet - Ingest: ~12 TB/day, Recall: ~24 TB/day (aggregated)
* HSI/HTAR clients also very reliable and used on several HPSS sites. ISH was written at SciNet.
* [[Media:HPSS_rational.pdf|HPSS fits well with the Storage Capacity Expansion Plan at SciNet]] (pdf presentation)

== '''Guidelines''' ==
* Expanded storage capacity is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into tarballs with tar or htar.
* The maximum size of a file that can be transferred into the repository is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application's exit code and the returned log file for errors after all data transfers and any tarball creation process.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== '''Access Through the Queue System''' ==
All access to the archive system is done through the [[Moab|GPC queue system]].
=== Scripted File Transfers ===
File transfers in and out of the HPSS should be scripted into jobs and submitted to the ''archive'' queue. See HSI example below.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_put_file_in_hpss
#PBS -j oe
#PBS -me

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
status=$?
if [ ! $status == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
exit $status
</pre>

The status of pending jobs can be monitored with showq specifying the archive queue:
<pre>
showq -w class=archive
</pre>

=== Recalling Data for Analysis ===

Typically, data will be recalled to the /scratch file system when it is needed for analysis. Job dependencies can be used to have analysis jobs wait in the queue for data recalls before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency:
<pre>
gpc04 $ qsub $(qsub data-recall.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-recalled-data.sh
</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with HPSS. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents in HPSS. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the file does not already exist in HPSS
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local storage only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of HPSS.
|-
| lcd,lls
| ''Local'' commands.
|}

* Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz"
</pre>
* More complex sequences can be performed using a script such as this:
<pre>
hsi <<EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
status=$?
if [ ! $status == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
exit status
</pre>

=== HSI Documentation ===
Complete documentation of HSI is available on the Gleicher Enterprises web site.
* [http://www.mgleicher.us/GEL/hsi/ HSI Introduction]
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help hsi help]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Typical Usage ===
The most common interactions will be ''putting'' data into HPSS, examining the contents (ls), and ''getting'' data back onto one of the active filesystems for inspection or analysis.

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -me

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away
cd put-away
cput /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
cput /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF
status=$?
if [ ! $status == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
exit $status
</pre>

* sample '''data list'''
A convenient way to explore the contents of HPSS is with the inventory shell [[ISH]]. This example creates an index of all the files in a user's portion of the namespace. The list is placed in the file /home/$USER/HPSSdm/hsi.igz that can be inspected from the gpc-devel nodes.
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_index
#PBS -j oe
#PBS -me

TODAY=$(date +%Y%m%d)
INDEX_DIR=/home/$USER/HPSSdm
if [[ -! -e $INDEX_DIR ]];then
mkdir $INDEX_DIR
fi

export ISHREGISTER=$HOME/HPSSdm
ish hindex
</pre>

This index can be browsed or searched with ISH on the development nodes.
<pre>
gpc-f104n084-$ ish ~/HPSSdm/hsi.igz
[ish]hsi.igz> help
</pre>

ISH is a powerful tool that is also useful for creating and browsing indices of tar and htar archives, so please look at the [[ISH|documentation]] or built in help.

* sample '''data recall'''
- This example should be modified to emphasize that a single ''cget'' of multiple files (rather than several separate gets) allows HSI to do optimization.
<pre>
#!/bin/bash

# This script is named: data-recall.sh

#PBS -q archive
#PBS -N recall
#PBS -j oe
#PBS -me

mkdir -p /scratch/$USER/recalled-from-hpss
hsi -v << EOF
cget /scratch/$USER/recalled-from-hpss/Jan-2010-jobs.tar.gz : put-away-on-2010/Jan-2010-jobs.tar.gz
cget /scratch/$USER/recalled-from-hpss/Feb-2010-jobs.tar.gz : put-away-on-2010/Feb-2010-jobs.tar.gz
EOF
status=$?

exit $status

</pre>

=== HSI vs. FTP ===
HSI syntax and usage is very similar to that of FTP. Please note the following information adapted from the HSI man page:

HSI supports several of the commonly used FTP commands, including "dir","get","ls","mdelete","mget","put","mput" and "prompt", with the following differences:

* The "dir" command is an alias for "ls" in HSI. The "ls" command supports an extensive set of options for displaying files, including wildcard pattern-matching, and the ability to recursively list a directory tree
* The "put" and "get" family of commands support recursion
* There are "conditional put" and "conditional" get commands (cput, cget)
* The syntax for renaming files during transfers with HSI is different from FTP. With HSI, the general format is always
<pre>
"local_file : hpss_file"
</pre>
and multiple such pairs may be specified on a single command line.

For example, when using HSI to store the local file "file1" as "hpss_file1" into HPSS, then retrieve it back to the local filesystem as "file1.bak", the following commands could be used:
<pre>
put file1 : hpss_file1
get file1.bak : hpss_file1
</pre>

* unlike with FTP, where the following syntax would be used:
<pre>
put file1 hpss_file1
get hpss_file1 file1.bak
</pre>
* The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.

=== Other HSI Examples ===

* Creating tar archive of C source programs and header files on the fly by piping stdout:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HSI pathnames must be surrounded by whitespace (one or more space characters)

* Retrieve the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>
* Put a subdirectory ''LargeFiles'' and all its contents recursively. You may use '-u' option to resume a previously disrupted session.
<pre>
cput -R -u LargeFiles
</pre>
* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help''']

* verify checksum
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N checksum_verified_transfer
#PBS -j oe
#PBS -me

thefile=<localpath>
storedfile=<hpsspath>

# Generate checksum on fly using a named pipe so that file is only read from GPFS once
mkfifo /tmp/NPIPE
cat $thefile | tee /tmp/NPIPE | hsi -q put - : $storedfile &
pid=$!
md5sum /tmp/NPIPE |tee /tmp/$fname.md5
rm /tmp/NPIPE

# Check the exit code of the HSI process
wait $pid
sc=$?
if [ $sc != 0 ];then
echo "File transfer failed"
exit $sc
fi

# change filename to stdin in checksum file
sed -i.1 "s+/tmp/NPIPE+-+" /tmp/$fname.md5

# verify checksum
hsi -q get - : $storedfile | md5sum -c /tmp/$fname.md5
sc=$?
if [ $sc != 0 ]; then
echo '!!! Job Failed !!!'
echo 'error=' $sc
fi

== '''HTAR''' ==
''' Please aggregate small files (<~100MB) into tarballs or htar files. '''

HTAR is a utility that is used for aggregating a set of files and directories, by using a sophisticated multithreaded buffering scheme to write files from the local filesystem directly into HPSS, creating an archive file that conforms to the POSIX TAR specification, thereby achieving a high rate of performance.

=== '''CAUTION''' ===
* Files larger than 68 GB cannot be stored in an htar archive (you'll get an error message for the whole operation)
* HTAR archives cannot contain more than 1 million files.
* Check the HTAR exit code and log file before removing any files from the active filesystems.

=== HTAR Usage ===
* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To extract all files from the ''project1/src'' directory in the archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

HPSS

2011-06-24T22:20:41Z

Knecht: ISH corrections

= '''High Performance Storage System''' =

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The High Performance Storage System ([http://www.hpss-collaboration.org/index.shtml HPSS]) is a tape backed hierarchical storage system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Transfer of data into and out of the repository will be under the control of the user who will interact with the system using one or more of the following utilities:
* [http://www.mgleicher.us/GEL/hsi HSI] is a client with an ftp-like interface which can be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* [http://www.mgleicher.us/GEL/htar HTAR] is a utility that creates tar format archives resident in the repository. It also creates a separate index file (.idx) that can be accessed quickly.
* [https://support.scinet.utoronto.ca/wiki/index.php/ISH ISH] is a TUI utility that can perform an inventory of the files and directories in your tarballs.

User access will be controlled by the job scheduling system of the GPC. Transfer of data into or out of the repository is expected to be scripted and submitted as a batch job.

== '''Guidelines''' ==
* Expanded storage capacity is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into tarballs with tar or htar.
* The maximum size of a file that can be transferred into the repository is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application's exit code and the returned log file for errors after all data transfers and any tarball creation process.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== '''Access Through the Queue System''' ==
All access to the archive system is through the GPC queue system documented [[Moab|here]].
=== Scripted File Transfers ===
File transfers in and out of the repository should be scripted into jobs and submitted to the ''archive'' queue. Scripts should use the HSI and/or HTAR commands as in the example below.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_put_file_in_repo
#PBS -j oe
#PBS -me

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
status=$?
if [ ! $status == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
exit $status
</pre>

The status of pending jobs can be monitored with showq specifying the archive queue:
<pre>
showq -w class=archive
</pre>

=== Recalling Data for Analysis ===

Typically, data will be recalled to the /scratch file system when it is needed for analysis. Job dependencies can be used to have analysis jobs wait in the queue for data recalls before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency:
<pre>
gpc04 $ qsub $(qsub data-recall.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-recalled-data.sh
</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the repository. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the file does not already exist in the repository
|-
| cget
| Conditionally retrieves a copy of a file from the repository to your local storage only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of the repository.
|-
| lcd,lls
| ''Local'' commands.
|}

* Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz"
</pre>
* More complex sequences can be performed using a script such as this:
<pre>
hsi <<EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
status=$?
if [ ! $status == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
exit status
</pre>

=== HSI Documentation ===
Complete documentation of HSI is available on the Gleicher Enterprises web site.
* [http://www.mgleicher.us/GEL/hsi/ HSI Introduction]
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help hsi help]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Typical Usage ===
The most common interactions will be ''putting'' data into the repository, examining the contents of the repository (ls), and ''getting'' data back onto one of the active filesystems for inspection or analysis.

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -me

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away
cd put-away
cput /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
cput /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF
status=$?
if [ ! $status == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
exit $status
</pre>

* sample '''data list'''
A convenient way to explore the contents of the archive is with the inventory shell [[ISH]]. This example creates an index of all the files in a user's portion of the namespace. The list is placed in the file /home/$USER/HPSSdm/hsi.igz that can be inspected from the gpc-devel nodes.
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N repository_index
#PBS -j oe
#PBS -me

TODAY=$(date +%Y%m%d)
INDEX_DIR=/home/$USER/HPSSdm
if [[ -! -e $INDEX_DIR ]];then
mkdir $INDEX_DIR
fi

export ISHREGISTER=$HOME/HPSSdm
ish hindex
</pre>

This index can be browsed or seached with ISH on the development nodes.
<pre>
gpc-f104n084-$ ish ~/HPSSdm/hsi.igz
[ish]hsi.igz> help
</pre>

ISH is a powerful tool that is also useful for creating and browsing indices of tar and htar archives, so please look at the [[ISH|documentation]] or built in help.

* sample '''data recall'''
- This example should be modified to emphasize that a single ''cget'' of multiple files (rather than several separate gets) allows HSI to do optimization.
<pre>
#!/bin/bash

# This script is named: data-recall.sh

#PBS -q archive
#PBS -N recall
#PBS -j oe
#PBS -me

mkdir -p /scratch/$USER/recalled-from-repository
hsi -v << EOF
cget /scratch/$USER/recalled-from-repository/Jan-2010-jobs.tar.gz : put-away-on-2010/Jan-2010-jobs.tar.gz
cget /scratch/$USER/recalled-from-repository/Feb-2010-jobs.tar.gz : put-away-on-2010/Feb-2010-jobs.tar.gz
EOF
status=$?

exit $status

</pre>

=== HSI vs. FTP ===
HSI syntax and usage is very similar to that of FTP. Please note the following information adapted from the HSI man page:

HSI supports several of the commonly used FTP commands, including "dir","get","ls","mdelete","mget","put","mput" and "prompt", with the following differences:

* The "dir" command is an alias for "ls" in HSI. The "ls" command supports an extensive set of options for displaying files, including wildcard pattern-matching, and the ability to recursively list a directory tree
* The "put" and "get" family of commands support recursion
* There are "conditional put" and "conditional" get commands (cput, cget)

* The syntax for renaming files during transfers with HSI is different from FTP. With HSI, the general format is always
<pre>
"local_file : repository_file"
</pre>
and multiple such pairs may be specified on a single command line.
With FTP, the local filename is specified first on a "put" command, and second on a "get" command.

For example, when using HSI to store the local file "file1" as "repository_file1" on the archival system, then retrieve it back to the local filesystem as "file1.bak", the following commands could be used:
<pre>
put file1 : repository_file1
get file1.bak : repository_file1
</pre>

* With FTP, the following syntax would be used instead:
<pre>
put file1 repository_file1
get repository_file1 file1.bak
</pre>
* The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.

=== Other HSI Examples ===

* Creating tar archive of C source programs and header files on the fly by piping stdout:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HSI pathnames must be surrounded by whitespace (one or more space characters)

* Retrieve the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>
* Put a subdirectory ''LargeFiles'' and all its contents recursively. You may use '-u' option to resume a previously disrupted session.
<pre>
cput -R -u LargeFiles
</pre>
* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help''']

== '''HTAR''' ==
''' Please aggregate small files (<~100MB) into tarballs or htar files. '''

HTAR is a utility that is used for aggregating a set of files from the local filesystem directly into the repository, creating a file that conforms to the POSIX TAR specification.
It does this without having to first create an intermediate file on the local filesystem; instead, it uses a sophisticated multithreaded buffering scheme to write files directly into the repository, thereby achieving a high rate of performance.

=== '''CAUTION''' ===
* Files larger than 68 GB cannot be stored in an htar archive (you'll get an error message for the whole operation)
* HTAR archives cannot contain more than 1 million files.
* Check the HTAR exit code and log file before removing any files from the active filesystems.

=== HTAR Usage ===
* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

HPSS

2011-06-24T22:17:32Z

Knecht:

= '''High Performance Storage System''' =

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The High Performance Storage System ([http://www.hpss-collaboration.org/index.shtml HPSS]) is a tape backed hierarchical storage system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Transfer of data into and out of the repository will be under the control of the user who will interact with the system using one or more of the following utilities:
* [http://www.mgleicher.us/GEL/hsi HSI] is a client with an ftp-like interface which can be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* [http://www.mgleicher.us/GEL/htar HTAR] is a utility that creates tar format archives resident in the repository. It also creates a separate index file (.idx) that can be accessed quickly.
* [https://support.scinet.utoronto.ca/wiki/index.php/ISH ISH] is a TUI utility that can perform an inventory of the files and directories in your tarballs.

User access will be controlled by the job scheduling system of the GPC. Transfer of data into or out of the repository is expected to be scripted and submitted as a batch job.

== '''Guidelines''' ==
* Expanded storage capacity is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into tarballs with tar or htar.
* The maximum size of a file that can be transferred into the repository is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application's exit code and the returned log file for errors after all data transfers and any tarball creation process.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== '''Access Through the Queue System''' ==
All access to the archive system is through the GPC queue system documented [[Moab|here]].
=== Scripted File Transfers ===
File transfers in and out of the repository should be scripted into jobs and submitted to the ''archive'' queue. Scripts should use the HSI and/or HTAR commands as in the example below.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_put_file_in_repo
#PBS -j oe
#PBS -me

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
status=$?
if [ ! $status == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
exit $status
</pre>

The status of pending jobs can be monitored with showq specifying the archive queue:
<pre>
showq -w class=archive
</pre>

=== Recalling Data for Analysis ===

Typically, data will be recalled to the /scratch file system when it is needed for analysis. Job dependencies can be used to have analysis jobs wait in the queue for data recalls before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency:
<pre>
gpc04 $ qsub $(qsub data-recall.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-recalled-data.sh
</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the repository. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the file does not already exist in the repository
|-
| cget
| Conditionally retrieves a copy of a file from the repository to your local storage only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of the repository.
|-
| lcd,lls
| ''Local'' commands.
|}

* Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz"
</pre>
* More complex sequences can be performed using a script such as this:
<pre>
hsi <<EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
status=$?
if [ ! $status == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
exit status
</pre>

=== HSI Documentation ===
Complete documentation of HSI is available on the Gleicher Enterprises web site.
* [http://www.mgleicher.us/GEL/hsi/ HSI Introduction]
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help hsi help]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Typical Usage ===
The most common interactions will be ''putting'' data into the repository, examining the contents of the repository (ls), and ''getting'' data back onto one of the active filesystems for inspection or analysis.

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -me

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away
cd put-away
cput /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
cput /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF
status=$?
if [ ! $status == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
exit $status
</pre>

* sample '''data list'''
A convenient way to explore the contents of the archive is with the inventory shell [[ISH]]. This example creates an index of all the files in a user's portion of the namespace. The list is placed in the file /home/$USER/HPSSdm/hsi.igz that can be inspected from the gpc-devel nodes.
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N repository_index
#PBS -j oe
#PBS -me

TODAY=$(date +%Y%m%d)
INDEX_DIR=/home/$USER/HPSSdm
if [[ -! -e $INDEX_DIR ]];then
mkdir $INDEX_DIR
fi

export ISHREGISTER=$HOME/HPSSdm
ish hindex
</pre>

This index can be browsed with ISH on the development nodes.
<pre>
gpc-f104n084-$ ish $HPSSdm/hsi.idx
</pre>

ISH is a powerful tool that is also useful for creating and browsing indices of tar and htar archives, so please look at the [[ISH|documentation]] or built in help.

* sample '''data recall'''
- This example should be modified to emphasize that a single ''cget'' of multiple files (rather than several separate gets) allows HSI to do optimization.
<pre>
#!/bin/bash

# This script is named: data-recall.sh

#PBS -q archive
#PBS -N recall
#PBS -j oe
#PBS -me

mkdir -p /scratch/$USER/recalled-from-repository
hsi -v << EOF
cget /scratch/$USER/recalled-from-repository/Jan-2010-jobs.tar.gz : put-away-on-2010/Jan-2010-jobs.tar.gz
cget /scratch/$USER/recalled-from-repository/Feb-2010-jobs.tar.gz : put-away-on-2010/Feb-2010-jobs.tar.gz
EOF
status=$?

exit $status

</pre>

=== HSI vs. FTP ===
HSI syntax and usage is very similar to that of FTP. Please note the following information adapted from the HSI man page:

HSI supports several of the commonly used FTP commands, including "dir","get","ls","mdelete","mget","put","mput" and "prompt", with the following differences:

* The "dir" command is an alias for "ls" in HSI. The "ls" command supports an extensive set of options for displaying files, including wildcard pattern-matching, and the ability to recursively list a directory tree
* The "put" and "get" family of commands support recursion
* There are "conditional put" and "conditional" get commands (cput, cget)

* The syntax for renaming files during transfers with HSI is different from FTP. With HSI, the general format is always
<pre>
"local_file : repository_file"
</pre>
and multiple such pairs may be specified on a single command line.
With FTP, the local filename is specified first on a "put" command, and second on a "get" command.

For example, when using HSI to store the local file "file1" as "repository_file1" on the archival system, then retrieve it back to the local filesystem as "file1.bak", the following commands could be used:
<pre>
put file1 : repository_file1
get file1.bak : repository_file1
</pre>

* With FTP, the following syntax would be used instead:
<pre>
put file1 repository_file1
get repository_file1 file1.bak
</pre>
* The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.

=== Other HSI Examples ===

* Creating tar archive of C source programs and header files on the fly by piping stdout:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HSI pathnames must be surrounded by whitespace (one or more space characters)

* Retrieve the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>
* Put a subdirectory ''LargeFiles'' and all its contents recursively. You may use '-u' option to resume a previously disrupted session.
<pre>
cput -R -u LargeFiles
</pre>
* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help''']

== '''HTAR''' ==
''' Please aggregate small files (<~100MB) into tarballs or htar files. '''

HTAR is a utility that is used for aggregating a set of files from the local filesystem directly into the repository, creating a file that conforms to the POSIX TAR specification.
It does this without having to first create an intermediate file on the local filesystem; instead, it uses a sophisticated multithreaded buffering scheme to write files directly into the repository, thereby achieving a high rate of performance.

=== '''CAUTION''' ===
* Files larger than 68 GB cannot be stored in an htar archive (you'll get an error message for the whole operation)
* HTAR archives cannot contain more than 1 million files.
* Check the HTAR exit code and log file before removing any files from the active filesystems.

=== HTAR Usage ===
* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

HPSS

2011-06-24T15:09:52Z

Knecht: typo in example

= '''High Performance Storage System''' =

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The High Performance Storage System ([http://www.hpss-collaboration.org/index.shtml HPSS]) is a tape backed hierarchical storage system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with the system using one or more of the following utilities:
* [http://www.mgleicher.us/GEL/hsi HSI] is a client with an ftp-like interface which can be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* [http://www.mgleicher.us/GEL/htar HTAR] is a utility that creates tar format archives resident in the repository. It also creates a separate index file (.idx) that can be accessed quickly.
* [https://support.scinet.utoronto.ca/wiki/index.php/ISH ISH] is a TUI utility that can perform an inventory of the files and directories in your tarballs.

User access will be controlled by the job scheduling system of the GPC. Transfer of data into or out of the repository is expected to be scripted and submitted as a batch job.

== '''Guidelines''' ==
* Expanded storage capacity is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the repository is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application's exit code and the returned log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== '''Access Through the Queue System''' ==
All access to the archive system is through the GPC queue system documented [[Moab|here]].
=== Scripted File Transfers ===
File transfers in and out of the repository should be scripted into jobs and submitted to the ''archive'' queue. Scripts should use the HSI and/or HTAR commands as in the example below.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_put_file_in_repo
#PBS -j oe
#PBS -me

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
status=$?
if [ ! $status == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
exit $status
</pre>

The status of pending jobs can be monitored with showq specifying the archive queue:
<pre>
showq -w class=archive
</pre>

=== Recalling Data for Analysis ===

Typically, data will be recalled to the /scratch file system when it is needed for analysis. Job dependencies can be used to have analysis jobs wait in the queue for data recalls before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency:
<pre>
gpc04 $ qsub $(qsub data-recall.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-recalled-data.sh
</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the repository. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the file does not already exist in the repository |-
| cget
| Conditionally retrieves a copy of a file from the repository to your local storage only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of the repository.
|-
| lcd,lls
| ''Local'' commands.
|}

Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz"
</pre>

More complex operations can be performed using a task sequence such as this:
<pre>
hsi <<EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
status=$?
if [ ! $status == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
exit status
</pre>

=== HSI Documentation ===
Complete documentation of HSI is available on the Gleicher Enterprises web site.
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Typical Usage ===
The most common interactions will be ''putting'' data into the repository, examining the contents of the repository (ls), and ''getting'' data back onto one of the active filesystems for inspection or analysis.

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -me

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away
cd put-away
cput /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
cput /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF
status=$?
if [ ! $status == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
exit $status
</pre>

* sample '''data list'''
- This example performs a recursive list of all files in a user's portion of the namespace. The list is placed in a dated file in the directory /home/$USER/repository.ix that can be inspected from the gpc-devel nodes.
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N repository_dump
#PBS -j oe
#PBS -me

TODAY=$(date +%Y%m%d)
INDEX_DIR=/home/$USER/repository.ix
if [[ -! -e $INDEX_DIR ]];then
mkdir $INDEX_DIR
fi

/usr/local/bin/hsi <<EOF > $INDEX_DIR/contents-$TODAY
ls -lUR
EOF

</pre>

* sample '''data recall'''
- This example should be modified to emphasize that a single ''cget'' of multiple files (rather than several separate gets) allows HSI to do optimization.
<pre>
#!/bin/bash

# This script is named: data-recall.sh

#PBS -q archive
#PBS -N recall
#PBS -j oe
#PBS -me

mkdir -p /scratch/$USER/recalled-from-repository
hsi -v << EOF
cget /scratch/$USER/recalled-from-repository/Jan-2010-jobs.tar.gz : put-away-on-2010/Jan-2010-jobs.tar.gz
cget /scratch/$USER/recalled-from-repository/Feb-2010-jobs.tar.gz : put-away-on-2010/Feb-2010-jobs.tar.gz
EOF
status=$?

exit $status

</pre>

=== HSI vs. FTP ===
HSI syntax and usage is very similar to that of FTP. Please note the following information adapted from the HSI man page:

HSI supports several of the commonly used FTP commands, including "dir","get","ls","mdelete","mget","put","mput" and "prompt", with the following differences:

* The "dir" command is an alias for "ls" in HSI. The "ls" command supports an extensive set of options for displaying files, including wildcard pattern-matching, and the ability to recursively list a directory tree
* The "put" and "get" family of commands support recursion
* There are "conditional put" and "conditional" get commands (cput, cget)

* The syntax for renaming files during transfers with HSI is different from FTP. With HSI, the general format is always
<pre>
"local_file : repository_file"
</pre>
and multiple such pairs may be specified on a single command line.
With FTP, the local filename is specified first on a "put" command, and second on a "get" command.

For example, when using HSI to store the local file "file1" as "repository_file1" on the archival system, then retrieve it back to the local filesystem as "file1.bak", the following commands could be used:
<pre>
put file1 : repository_file1
get file1.bak : repository_file1
</pre>

* With FTP, the following commands could be used:
<pre>
put file1 repository_file1
get repository_file1 file1.bak
</pre>
* The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.

=== Other HSI Examples ===

* Creating tar archive on the fly by piping stdout:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HSI pathnames must be surrounded by whitespace (one or more space characters)

* Retrieve the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''HTAR''' ==
''' Please aggregate small files (<~100MB) into tarballs or htar files. '''

HTAR is a utility that is used for aggregating a set of files from the local filesystem directly into the repository, creating a file that conforms to the POSIX TAR specification.
It does this without having to first create an intermediate file on the local filesystem; instead, it uses a sophisticated multithreaded buffering scheme to write files directly into the repository, thereby achieving a high rate of performance.

=== '''CAUTION''' ===
* Files larger than 68 GB cannot be stored in an htar archive.
* HTAR archives cannot contain more than 1 million files.
* Check the HTAR exit code and log file before removing any files from the active filesystems.

=== HTAR Usage ===
* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

cd /scratch/$USER/recalled-from-repository/usr/local/bin/htar -xf finished-job3.tar

MARS

2011-06-23T19:00:49Z

Knecht: Add always-email and raising of exit status to all qsub examples.

= '''Massive Archive and Retrieval System''' =

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface which can be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the repository. It also creates a separate index file that can be accessed quickly.
* ISH is a TUI utility that can perform an inventory of the files and directories in your tarballs.

User access will be controlled by the job scheduling system of the GPC. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

== '''Guidelines''' ==
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== '''Access Through the Queue System''' ==
All access to the archive system is through the GPC queue system documented [[Moab|here]].
=== Scripted File Transfers ===
File transfers in and out of the archive should be scripted into jobs and submitted to the ''archive'' queue. Script will use either the HSI or HTAR commands documented below.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_put_file_in_repo
#PBS -j oe
#PBS -me

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
status=$?
if [ ! $status == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
exit $status
</pre>

The status of pending jobs can be monitored with showq specifying the archive queue:
<pre>
showq -w class=archive
</pre>

=== Recalling Data for Analysis ===

Typically, data will be recalled to the /scratch file system when it is needed for analysis. Job dependencies can be used to have analysis jobs wait in the queue for data recalls before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency:
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of the archive.
|-
| lcd,lls
| ''Local'' commands.
|}

Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz"
</pre>

More complex operations can be performed using a Here Document.
<pre>
hsi <<-EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
status=$?
if [ ! $status == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
exit status
</pre>

=== HSI Documentation ===
Complete documentation of HSI is available on the Gleicher Enterprises web site.
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Typical Usage ===
The most common interactions with the HPSS archive will ''putting'' data into the archive, examining the contents of the archive (ls), and ''getting'' data back onto one of the main filesystems for inspection or analysis.

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -me

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away
cd put-away
cput /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
cput /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF
status=$?
if [ ! $status == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
exit $status
</pre>

* sample '''data list'''
- This example performs a recursive list of all files in a user's portion of the namespace. The list is placed in a dated file in the directory /home/$USER/HPSS.ix that can be inspected from the gpc-devel nodes.
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -me

TODAY=$(date +%Y%m%d)
INDEX_DIR=/home/$USER/HPSS.ix
if [[ -! -e $INDEX_DIR ]];then
mkdir $INDEX_DIR
fi

/usr/local/bin/hsi <<EOF > $INDEX_DIR/contents-$TODAY
ls -lUR
EOF

</pre>

* sample '''data restore'''
- This example should be modified to emphasize that a single ''cget'' of multiple files (rather than several separate gets) allows HSI to do staging optimization.
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -me

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
cget /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
cget /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF
status=$?

exit $status

</pre>

=== HSI vs. FTP ===
HSI syntax and usage is very similar to that of FTP. Please note the following information adapted from the HSI man page:

HSI supports several of the commonly used FTP commands, including "dir","get","ls","mdelete","mget","put","mput" and "prompt", with the following differences:

* The "dir" command is an alias for "ls" in HSI. The "ls" command supports an extensive set of options for displaying files, including wildcard pattern-matching, and the ability to recursively list a directory tree
* The "put" and "get" family of commands support recursion
* There are "conditional put" and "conditional" get commands (cput, cget)

* The syntax for renaming local files when storing files to HPSS or retrieving files from HPSS is different than FTP. With HSI, the syntax is always
<pre>
"local_file : HPSS_file"
</pre>
and multiple such pairs may be specified on a single command line.
With FTP, the local filename is specified first on a "put" command, and second on a "get" command.

For example, when using HSI to store the local file "file1" as HPSS file "hpss_file1", then retrieve it back to the local filesystem as "file1.bak", the following commands could be used:
<pre>
put file1 : hpss_file1
get file1.bak : hpss_file1
</pre>

* With FTP, the following commands could be used:
<pre>
put file1 hpss_file1
get hpss_file1 file1.bak
</pre>
* The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.

=== Other HSI Examples ===

* Creating tar archive on the fly by piping stdout:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''HTAR''' ==
''' Please aggregate small files (<~100MB) into tarballs or htar files. '''

HTAR is a utility that is used for aggregating a set of files from the local file system directly into HPSS, creating a file that conforms to the POSIX TAR specification.
It does this without having to first create an intermediate file on the local filesystem; instead, it uses a sophisticated multithreaded buffering scheme to write files directly into HPSS, thereby achieving a high rate of performance.

=== '''CAUTION''' ===
* Files larger than 68 GB cannot be stored in an htar archive.
* HTAR archives cannot contain more than 1 million files.
* Check the HTAR exit code and log file before removing any files from the active filesystems.

=== HTAR Usage ===
* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

MARS

2011-06-23T16:36:24Z

Knecht:

= '''Massive Archive and Retrieval System''' =

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface which can be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the repository. It also creates a separate index file that can be accessed quickly.
* ISH is a TUI utility that can perform an inventory of the files and directories in your tarballs.

User access will be controlled by the job scheduling system of the GPC. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

== '''Guidelines''' ==
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== '''Access Through the Queue System''' ==
All access to the archive system is through the GPC queue system documented [[Moab|here]].
=== Scripted File Transfers ===
File transfers in and out of the archive should be scripted into jobs and submitted to the ''archive'' queue. Script will use either the HSI or HTAR commands documented below.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_put_file_in_repo
#PBS -j oe

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
</pre>

The status of pending jobs can be monitored with showq specifying the archive queue:
<pre>
showq -w class=archive
</pre>

=== Recalling Data for Analysis ===

Typically, data will be recalled to the /scratch file system when it is needed for analysis. Job dependencies can be used to have analysis jobs wait in the queue for data recalls before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency:
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of the archive.
|-
| lcd,lls
| ''Local'' commands.
|}

Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz"
</pre>

More complex operations can be performed using a Here Document.
<pre>
hsi <<-EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
if [ ! $? == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
</pre>

=== HSI Documentation ===
Complete documentation of HSI is available on the Gleicher Enterprises web site.
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Typical Usage ===
The most common interactions with the HPSS archive will ''putting'' data into the archive, examining the contents of the archive (ls), and ''getting'' data back onto one of the main filesystems for inspection or analysis.

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away
cd put-away
cput /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
cput /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF
if [ ! $? == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
</pre>

* sample '''data list'''
- This example performs a recursive list of all files in a user's portion of the namespace. The list is placed in a dated file in the directory /home/$USER/HPSS.ix that can be inspected from the gpc-devel nodes.
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe

TODAY=$(date +%Y%m%d)
INDEX_DIR=/home/$USER/HPSS.ix
if [[ -! -e $INDEX_DIR ]];then
mkdir $INDEX_DIR
fi

/usr/local/bin/hsi <<EOF > $INDEX_DIR/contents-$TODAY
ls -lUR
EOF

</pre>

* sample '''data restore'''
- This example should be modified to emphasize that a single ''cget'' of multiple files (rather than several separate gets) allows HSI to do staging optimization.
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
cget /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
cget /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

</pre>

=== HSI vs. FTP ===
HSI syntax and usage is very similar to that of FTP. Please note the following information adapted from the HSI man page:

HSI supports several of the commonly used FTP commands, including "dir","get","ls","mdelete","mget","put","mput" and "prompt", with the following differences:

* The "dir" command is an alias for "ls" in HSI. The "ls" command supports an extensive set of options for displaying files, including wildcard pattern-matching, and the ability to recursively list a directory tree
* The "put" and "get" family of commands support recursion
* There are "conditional put" and "conditional" get commands (cput, cget)

* The syntax for renaming local files when storing files to HPSS or retrieving files from HPSS is different than FTP. With HSI, the syntax is always
<pre>
"local_file : HPSS_file"
</pre>
and multiple such pairs may be specified on a single command line.
With FTP, the local filename is specified first on a "put" command, and second on a "get" command.

For example, when using HSI to store the local file "file1" as HPSS file "hpss_file1", then retrieve it back to the local filesystem as "file1.bak", the following commands could be used:
<pre>
put file1 : hpss_file1
get file1.bak : hpss_file1
</pre>

* With FTP, the following commands could be used:
<pre>
put file1 hpss_file1
get hpss_file1 file1.bak
</pre>
* The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.

=== Other HSI Examples ===

* Creating tar archive on the fly by piping stdout:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''HTAR''' ==
''' Please aggregate small files (<~100MB) into tarballs or htar files. '''

HTAR is a utility that is used for aggregating a set of files from the local file system directly into HPSS, creating a file that conforms to the POSIX TAR specification.
It does this without having to first create an intermediate file on the local filesystem; instead, it uses a sophisticated multithreaded buffering scheme to write files directly into HPSS, thereby achieving a high rate of performance.

=== '''CAUTION''' ===
* Files larger than 68 GB cannot be stored in an htar archive.
* HTAR archives cannot contain more than 1 million files.
* Check the HTAR exit code and log file before removing any files from the active filesystems.

=== HTAR Usage ===
* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

MARS

2011-06-23T16:34:08Z

Knecht: Move "typical" HSI examples to HSI section.

= '''Massive Archive and Retrieval System''' =

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface which can be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the repository. It also creates a separate index file that can be accessed quickly.
* ISH is a TUI utility that can perform an inventory of the files and directories in your tarballs.

User access will be controlled by the job scheduling system of the GPC. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

== '''Guidelines''' ==
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== '''Access Through the Queue System''' ==
All access to the archive system is through the GPC queue system documented [[Moab|here]].
=== Scripted File Transfers ===
File transfers in and out of the archive should be scripted into jobs and submitted to the ''archive'' queue. Script will use either the HSI or HTAR commands documented below.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_put_file_in_repo
#PBS -j oe

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
</pre>

The status of pending jobs can be monitored with showq specifying the archive queue:
<pre>
showq -w class=archive
</pre>

=== Recalling Data for Analysis ===

Typically, data will be recalled to the /scratch file system when it is needed for analysis. Job dependencies can be used to have analysis jobs wait in the queue for data recalls before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency:
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of the archive.
|-
| lcd,lls
| ''Local'' commands.
|}

Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz"
</pre>

More complex operations can be performed using a Here Document.
<pre>
hsi <<-EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
if [ ! $? == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
</pre>

=== HSI Documentation ===
Complete documentation of HSI is available on the Gleicher Enterprises web site.
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Typical Usage ===
The most common interactions with the HPSS archive will ''putting'' data into the archive, examining the contents of the archive (ls), and ''getting'' data back onto one of the main filesystems for inspection or analysis.

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away
cd put-away
cput /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
cput /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF
if [ ! $? == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
</pre>

* sample '''data list'''
- This example performs a recursive list of all files in a user's portion of the namespace. The list is placed in a dated file in the directory /home/$USER/HPSS.ix that can be inspected from the gpc-devel nodes.
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe

TODAY=$(date +%Y%m%d)
INDEX_DIR=/home/$USER/HPSS.ix
if [[ -! -e $INDEX_DIR ]];then
mkdir $INDEX_DIR
fi

/usr/local/bin/hsi <<EOF > $INDEX_DIR/contents-$TODAY
ls -lUR
EOF

</pre>

* sample '''data restore'''
- This example should be modified to emphasize that a single ''cget'' of multiple files (rather than several separate gets) allows HSI to do staging optimization.
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
cget /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
cget /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

</pre>

=== HSI vs. FTP ===
HSI syntax and usage is very similar to that of FTP. Please note the following information adapted from the HSI man page:

HSI supports several of the commonly used FTP commands, including "dir","get","ls","mdelete","mget","put","mput" and "prompt", with the following differences:

* The "dir" command is an alias for "ls" in HSI. The "ls" command supports an extensive set of options for displaying files, including wildcard pattern-matching, and the ability to recursively list a directory tree
* The "put" and "get" family of commands support recursion
* There are "conditional put" and "conditional" get commands (cput, cget)

* The syntax for renaming local files when storing files to HPSS or retrieving files from HPSS is different than FTP. With HSI, the syntax is always
<pre>
"local_file : HPSS_file"
</pre>
and multiple such pairs may be specified on a single command line.
With FTP, the local filename is specified first on a "put" command, and second on a "get" command.

For example, when using HSI to store the local file "file1" as HPSS file "hpss_file1", then retrieve it back to the local filesystem as "file1.bak", the following commands could be used:
<pre>
put file1 : hpss_file1
get file1.bak : hpss_file1
</pre>

* With FTP, the following commands could be used:
<pre>
put file1 hpss_file1
get hpss_file1 file1.bak
</pre>
* The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.

=== Other HSI Examples ===

* Creating tar archive on the fly by piping stdout:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''HTAR''' ==
''' Please aggregate small files (<~100MB) into tarballs or htar files. '''

HTAR is a utility that is used for aggregating a set of files from the local file system directly into HPSS, creating a file that conforms to the POSIX TAR specification.
It does this without having to first create an intermediate file on the local filesystem; instead, it uses a sophisticated multithreaded buffering scheme to write files directly into HPSS, thereby achieving a high rate of performance.

=== '''CAUTION''' ===
* Files larger than 68MB cannot be stored in an htar archive.
* HTAR archives cannot contain more than 1 million files.
* Check the HTAR exit code and log file before removing any files from the active filesystems.

=== HTAR Usage ===
* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

MARS

2011-06-23T16:03:04Z

Knecht: batch usage clarifications

= '''Massive Archive and Retrieval System''' =

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface which can be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the repository. It also creates a separate index file that can be accessed quickly.
* ISH is a TUI utility that can perform an inventory of the files and directories in your tarballs.

User access will be controlled by the job scheduling system of the GPC. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

== '''Guidelines''' ==
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== '''Access Through the Queue System''' ==
All access to the archive system is through the GPC queue system documented [[Moab|here]].
=== Scripted File Transfers ===
File transfers in and out of the archive should be scripted into jobs and submitted to the ''archive'' queue. Script will use either the HSI or HTAR commands documented below.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_put_file_in_repo
#PBS -j oe

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then
echo '!!! TRANSFER FAILED !!!'
fi
</pre>

The status of pending jobs can be monitored with showq specifying the archive queue:
<pre>
showq -w class=archive
</pre>

=== Recalling Data for Analysis ===

Typically, data will be recalled to the /scratch file system when it is needed for analysis. Job dependencies can be used to have analysis jobs wait in the queue for data recalls before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency:
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of the archive.
|-
| lcd,lls
| ''Local'' commands.
|}

Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz"
</pre>

More complex operations can be performed using a Here Document.
<pre>
hsi <<-EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
</pre>

=== HSI vs. FTP ===
HSI syntax and usage is very similar to that of FTP. Please note the following information adapted from the HSI man page:

HSI supports several of the commonly used FTP commands, including "dir","get","ls","mdelete","mget","put","mput" and "prompt", with the following differences:

* The "dir" command is an alias for "ls" in HSI. The "ls" command supports an extensive set of options for displaying files, including wildcard pattern-matching, and the ability to recursively list a directory tree
* The "put" and "get" family of commands support recursion
* There are "conditional put" and "conditional" get commands (cput, cget)

* The syntax for renaming local files when storing files to HPSS or retrieving files from HPSS is different than FTP. With HSI, the syntax is always
<pre>
"local_file : HPSS_file"
</pre>
and multiple such pairs may be specified on a single command line.
With FTP, the local filename is specified first on a "put" command, and second on a "get" command.

For example, when using HSI to store the local file "file1" as HPSS file "hpss_file1", then retrieve it back to the local filesystem as "file1.bak", the following commands could be used:
<pre>
put file1 : hpss_file1
get file1.bak : hpss_file1
</pre>

* With FTP, the following commands could be used:
<pre>
put file1 hpss_file1
get hpss_file1 file1.bak
</pre>
* The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.

=== HSI Documentation ===
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Other HSI Examples ===

* Creating tar archive on the fly by piping stdout:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''HTAR''' ==
''' Please aggregate small files (<~100MB) into tarballs or htar files. '''

HTAR is a utility that is used for aggregating a set of files from the local file system directly into HPSS, creating a file that conforms to the POSIX TAR specification.
It does this without having to first create an intermediate file on the local filesystem; instead, it uses a sophisticated multithreaded buffering scheme to write files directly into HPSS, thereby achieving a high rate of performance.

=== '''CAUTION''' ===
* Files larger than 68MB cannot be stored in an htar archive.
* HTAR archives cannot contain more than 1 million files.
* Check the HTAR exit code and log file before removing any files from the active filesystems.

=== HTAR Usage ===
* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==

<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away
cd put-away
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

MARS

2011-06-23T15:47:06Z

Knecht:

= '''Massive Archive and Retrieval System''' =

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface which can be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the repository. It also creates a separate index file that can be accessed quickly.
* ISH is a TUI utility that can perform an inventory of the files and directories in your tarballs.

User access will be controlled by the job scheduling system of the GPC. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

== '''Guidelines''' ==
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== '''Access Through the Queue System''' ==
All access to the archive system is through the queue system.
=== Scripted File Transfers ===
File transfers in and out of the archive should be scripted into jobs and submitted to the archive queue.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_file_transfer
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then

</pre>

=== Recalling Data for Analysis ===

Job dependencies can be used to make analysis jobs wait for data recalls before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency:
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of the archive.
|-
| lcd,lls
| ''Local'' commands.
|}

Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz"
</pre>

More complex operations can be performed using a Here Document.
<pre>
hsi <<-EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
</pre>

=== HSI vs. FTP ===
HSI syntax and usage is very similar to that of FTP. Please note the following information adapted from the HSI man page:

HSI supports several of the commonly used FTP commands, including "dir","get","ls","mdelete","mget","put","mput" and "prompt", with the following differences:

* The "dir" command is an alias for "ls" in HSI. The "ls" command supports an extensive set of options for displaying files, including wildcard pattern-matching, and the ability to recursively list a directory tree
* The "put" and "get" family of commands support recursion
* There are "conditional put" and "conditional" get commands (cput, cget)

* The syntax for renaming local files when storing files to HPSS or retrieving files from HPSS is different than FTP. With HSI, the syntax is always
<pre>
"local_file : HPSS_file"
</pre>
and multiple such pairs may be specified on a single command line.
With FTP, the local filename is specified first on a "put" command, and second on a "get" command.

For example, when using HSI to store the local file "file1" as HPSS file "hpss_file1", then retrieve it back to the local filesystem as "file1.bak", the following commands could be used:
<pre>
put file1 : hpss_file1
get file1.bak : hpss_file1
</pre>

* With FTP, the following commands could be used:
<pre>
put file1 hpss_file1
get hpss_file1 file1.bak
</pre>
* The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.

=== HSI Documentation ===
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Other HSI Examples ===

* Creating tar archive on the fly by piping stdout:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''HTAR''' ==
''' Please aggregate small files (<~100MB) into tarballs or htar files. '''

HTAR is a utility that is used for aggregating a set of files from the local file system directly into HPSS, creating a file that conforms to the POSIX TAR specification.
It does this without having to first create an intermediate file on the local filesystem; instead, it uses a sophisticated multithreaded buffering scheme to write files directly into HPSS, thereby achieving a high rate of performance.

=== '''CAUTION''' ===
* Files larger than 68MB cannot be stored in an htar archive.
* HTAR archives cannot contain more than 1 million files.
* Check the HTAR exit code and log file before removing any files from the active filesystems.

=== HTAR Usage ===
* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==

<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away
cd put-away
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

MARS

2011-06-20T17:53:12Z

Knecht:

== '''Massive Archive and Retrieval System''' ==

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface will be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the archive. It also creates a separate index file that can be accessed quickly.

User access will be controlled by the job scheduling system of the GPC. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

== Guidelines ==
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== Access Through the Queue System ==
All access to the archive system is through the queue system.
=== Scripted File Transfers ===
File transfers in and out of the archive should be scripted into jobs and submitted to the archive queue.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_file_transfer
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then

</pre>

=== Staging Data for Analysis ===

Job dependencies can be used to make analysis jobs wait for data staging before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency:
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of the archive.
|-
| lcd,lls
| ''Local'' commands.
|}

Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz"
</pre>

More complex operations can be performed using a Here Document.
<pre>
hsi <<-EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
</pre>

=== HSI vs. FTP ===
HSI syntax and usage is very similar to that of FTP. Please note the following information adapted from the HSI man page:

HSI supports several of the commonly used FTP commands, including "dir","get","ls","mdelete","mget","put","mput" and "prompt", with the following differences:

* The "dir" command is an alias for "ls" in HSI. The "ls" command supports an extensive set of options for displaying files, including wildcard pattern-matching, and the ability to recursively list a directory tree
* The "put" and "get" family of commands support recursion
* There are "conditional put" and "conditional" get commands (cput, cget)

* The syntax for renaming local files when storing files to HPSS or retrieving files from HPSS is different than FTP. With HSI, the syntax is always
<pre>
"local_file : HPSS_file"
</pre>
and multiple such pairs may be specified on a single command line.
With FTP, the local filename is specified first on a "put" command, and second on a "get" command.

For example, when using HSI to store the local file "file1" as HPSS file "hpss_file1", then retrieve it back to the local filesystem as "file1.bak", the following commands could be used:
<pre>
put file1 : hpss_file1
get file1.bak : hpss_file1
</pre>

* With FTP, the following commands could be used:
<pre>
put file1 hpss_file1
get hpss_file1 file1.bak
</pre>
* The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.

=== HSI Documentation ===
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Other HSI Examples ===

* Creating tar archive on the fly by piping stdout:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''HTAR''' ==
''' Please aggregate small files (<~100MB) into tarballs or htar files. '''

HTAR is a utility that is used for aggregating a set of files from the local file system directly into HPSS, creating a file that conforms to the POSIX TAR specification.
It does this without having to first create an intermediate file on the local filesystem; instead, it uses a sophisticated multithreaded buffering scheme to write files directly into HPSS, thereby achieving a high rate of performance.

=== '''CAUTION''' ===
* Files larger than 68MB cannot be stored in an htar archive.
* HTAR archives cannot contain more than 1 million files.
* Check the HTAR exit code and log file before removing any files from the active filesystems.

=== HTAR Usage ===
* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==

<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

MARS

2011-06-20T17:45:34Z

Knecht: more interactive references

== '''Massive Archive and Retrieval System''' ==

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface will be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the archive. It also creates a separate index file that can be accessed quickly.

User access will be controlled by the job scheduling system of the GPC. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

== Guidelines ==
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== Access Through the Queue System ==
All access to the archive system is through the queue system.
=== Scripted File Transfers ===
File transfers in and out of the archive should be scripted into jobs and submitted to the archive queue.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_file_transfer
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then

</pre>

=== Staging Data for Analysis ===

Job dependencies can be used to make analysis jobs wait for data staging before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency:
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of the archive.
|-
| lcd,lls
| ''Local'' commands.
|}

Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz
</pre>

More complex operations can be performed using a Here Document.
<pre>
hsi <<-EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
</pre>

=== HSI vs. FTP ===
HSI syntax and usage is very similar to that of FTP. Please note the following information adapted from the HSI man page:

HSI supports several of the commonly used FTP commands, including "dir","get","ls","mdelete","mget","put","mput" and "prompt", with the following differences:

* The "dir" command is an alias for "ls" in HSI. The "ls" command supports an extensive set of options for displaying files, including wildcard pattern-matching, and the ability to recursively list a directory tree
* The "put" and "get" family of commands support recursion
* There are "conditional put" and "conditional" get commands (cput, cget)

* The syntax for renaming local files when storing files to HPSS or retrieving files from HPSS is different than FTP. With HSI, the syntax is always
<pre>
"local_file : HPSS_file"
</pre>
and multiple such pairs may be specified on a single command line.
With FTP, the local filename is specified first on a "put" command, and second on a "get" command.

For example, when using HSI to store the local file "file1" as HPSS file "hpss_file1", then retrieve it back to the local filesystem as "file1.bak", the following commands could be used:
<pre>
put file1 : hpss_file1
get file1.bak : hpss_file1
</pre>

* With FTP, the following commands could be used:
<pre>
put file1 hpss_file1
get hpss_file1 file1.bak
</pre>
* The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.

=== HSI Documentation ===
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Other Examples ===

* Creating tar archive on the fly by piping stdout:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''HTAR''' ==
''' Please aggregate small file (<~100MB) into tarballs or htar files. '''

HTAR is a utility that is used for aggregating a set of files from the local file system directly into HPSS, creating a file that conforms to the POSIX TAR specification.
It does this without having to first create an intermediate file on the local filesystem; instead, it uses a sophisticated multithreaded buffering scheme to write files directly into HPSS, thereby achieving a high rate of performance.

=== '''CAUTION''' ===
* Files larger than 68MB cannot be stored in an htar archive.
* HTAR archives cannot contain more than 1 million files.
* Check the HTAR exit code and log file before removing any files from the active filesystems.

=== HTAR Usage ===
* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==

<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

MARS

2011-06-20T17:42:16Z

Knecht: htar notes

== '''Massive Archive and Retrieval System''' ==

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface will be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the archive. It also creates a separate index file that can be accessed quickly.

User access will be controlled by the job scheduling system of the GPC. An interactive session can be requested that will allow a user to list, rearrange or remove files with the HSI client. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

== Guidelines ==
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== Access Through the Queue System ==
All access to the archive system is through the queue system.
=== Scripted File Transfers ===
File transfers in and out of the archive should be scripted into jobs and submitted to the archive queue.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_file_transfer
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then

</pre>

=== Staging Data for Analysis ===

Job dependencies can be used to make analysis jobs wait for data staging before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency:
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of the archive.
|-
| lcd,lls
| ''Local'' commands.
|}

Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz
</pre>

More complex operations can be performed using a Here Document.
<pre>
hsi <<-EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
</pre>

=== HSI vs. FTP ===
HSI syntax and usage is very similar to that of FTP. Please note the following information adapted from the HSI man page:

HSI supports several of the commonly used FTP commands, including "dir","get","ls","mdelete","mget","put","mput" and "prompt", with the following differences:

* The "dir" command is an alias for "ls" in HSI. The "ls" command supports an extensive set of options for displaying files, including wildcard pattern-matching, and the ability to recursively list a directory tree
* The "put" and "get" family of commands support recursion
* There are "conditional put" and "conditional" get commands (cput, cget)

* The syntax for renaming local files when storing files to HPSS or retrieving files from HPSS is different than FTP. With HSI, the syntax is always
<pre>
"local_file : HPSS_file"
</pre>
and multiple such pairs may be specified on a single command line.
With FTP, the local filename is specified first on a "put" command, and second on a "get" command.

For example, when using HSI to store the local file "file1" as HPSS file "hpss_file1", then retrieve it back to the local filesystem as "file1.bak", the following commands could be used:
<pre>
put file1 : hpss_file1
get file1.bak : hpss_file1
</pre>

* With FTP, the following commands could be used:
<pre>
put file1 hpss_file1
get hpss_file1 file1.bak
</pre>
* The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.

=== HSI Documentation ===
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Other Examples ===

* Creating tar archive on the fly by piping stdout:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''HTAR''' ==
''' Please aggregate small file (<~100MB) into tarballs or htar files. '''

HTAR is a utility that is used for aggregating a set of files from the local file system directly into HPSS, creating a file that conforms to the POSIX TAR specification.
It does this without having to first create an intermediate file on the local filesystem; instead, it uses a sophisticated multithreaded buffering scheme to write files directly into HPSS, thereby achieving a high rate of performance.

=== '''CAUTION''' ===
* Files larger than 68MB cannot be stored in an htar archive.
* HTAR archives cannot contain more than 1 million files.
* Check the HTAR exit code and log file before removing any files from the active filesystems.

=== HTAR Usage ===
* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==

<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

MARS

2011-06-20T17:35:15Z

Knecht:

== '''Massive Archive and Retrieval System''' ==

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface will be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the archive. It also creates a separate index file that can be accessed quickly.

User access will be controlled by the job scheduling system of the GPC. An interactive session can be requested that will allow a user to list, rearrange or remove files with the HSI client. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

== Guidelines ==
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== Access Through the Queue System ==
All access to the archive system is through the queue system.
=== Scripted File Transfers ===
File transfers in and out of the archive should be scripted into jobs and submitted to the archive queue.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_file_transfer
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then

</pre>

=== Staging Data for Analysis ===

Job dependencies can be used to make analysis jobs wait for data staging before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency:
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of the archive.
|-
| lcd,lls
| ''Local'' commands.
|}

Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz
</pre>

More complex operations can be performed using a Here Document.
<pre>
hsi <<-EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
</pre>

=== HSI vs. FTP ===
HSI syntax and usage is very similar to that of FTP. Please note the following information adapted from the HSI man page:

HSI supports several of the commonly used FTP commands, including "dir","get","ls","mdelete","mget","put","mput" and "prompt", with the following differences:

* The "dir" command is an alias for "ls" in HSI. The "ls" command supports an extensive set of options for displaying files, including wildcard pattern-matching, and the ability to recursively list a directory tree
* The "put" and "get" family of commands support recursion
* There are "conditional put" and "conditional" get commands (cput, cget)

* The syntax for renaming local files when storing files to HPSS or retrieving files from HPSS is different than FTP. With HSI, the syntax is always
<pre>
"local_file : HPSS_file"
</pre>
and multiple such pairs may be specified on a single command line.
With FTP, the local filename is specified first on a "put" command, and second on a "get" command.

For example, when using HSI to store the local file "file1" as HPSS file "hpss_file1", then retrieve it back to the local filesystem as "file1.bak", the following commands could be used:
<pre>
put file1 : hpss_file1
get file1.bak : hpss_file1
</pre>

* With FTP, the following commands could be used:
<pre>
put file1 hpss_file1
get hpss_file1 file1.bak
</pre>
* The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.

=== HSI Documentation ===
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Other Examples ===

* Creating tar archive on the fly by piping stdout:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''HTAR''' ==
''' Please aggregate small file (<~100MB) into tarballs or htar files. '''

HTAR is a utility that is used for aggregating a set of files from the local file system directly into HPSS, creating a file that conforms to the POSIX TAR specification.
It does this without having to first create an intermediate file on the local filesystem; instead, it uses a sophisticated multithreaded buffering scheme to write files directly into HPSS, thereby achieving a high rate of performance.

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==
[[GPC_Quickstart|Submitting_A_Batch_Job]]
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

MARS

2011-06-20T17:29:04Z

Knecht: /* Examples */

== '''Massive Archive and Retrieval System''' ==

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface will be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the archive. It also creates a separate index file that can be accessed quickly.

User access will be controlled by the job scheduling system of the GPC. An interactive session can be requested that will allow a user to list, rearrange or remove files with the HSI client. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

== Guidelines ==
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== Access Through the Queue System ==
All access to the archive system is through the queue system.
=== Scripted File Transfers ===
File transfers in and out of the archive should be scripted into jobs and submitted to the archive queue.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_file_transfer
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then

</pre>

=== Staging Data for Analysis ===

Job dependencies can be used to make analysis jobs wait for data staging before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency:
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of the archive.
|-
| lcd,lls
| ''Local'' commands.
|}

Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz
</pre>

More complex operations can be performed using a Here Document.
<pre>
hsi <<-EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
</pre>

=== HSI vs. FTP ===
HSI syntax and usage is very similar to that of FTP. Please note the following information adapted from the HSI man page:

HSI supports several of the commonly used FTP commands, including "dir","get","ls","mdelete","mget","put","mput" and "prompt", with the following differences:

* The "dir" command is an alias for "ls" in HSI. The "ls" command supports an extensive set of options for displaying files, including wildcard pattern-matching, and the ability to recursively list a directory tree
* The "put" and "get" family of commands support recursion
* There are "conditional put" and "conditional" get commands (cput, cget)

* The syntax for renaming local files when storing files to HPSS or retrieving files from HPSS is different than FTP. With HSI, the syntax is always
<pre>
"local_file : HPSS_file"
</pre>
and multiple such pairs may be specified on a single command line.
With FTP, the local filename is specified first on a "put" command, and second on a "get" command.

For example, when using HSI to store the local file "file1" as HPSS file "hpss_file1", then retrieve it back to the local filesystem as "file1.bak", the following commands could be used:
<pre>
put file1 : hpss_file1
get file1.bak : hpss_file1
</pre>

* With FTP, the following commands could be used:
<pre>
put file1 hpss_file1
get hpss_file1 file1.bak
</pre>
* The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.

=== HSI Documentation ===
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Other Examples ===

* Creating tar archive on the fly by piping stdout:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''Using HTAR''' ==

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==
[[GPC_Quickstart|Submitting_A_Batch_Job]]
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

MARS

2011-06-20T17:27:38Z

Knecht: cut interactive batch access section

== '''Massive Archive and Retrieval System''' ==

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface will be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the archive. It also creates a separate index file that can be accessed quickly.

User access will be controlled by the job scheduling system of the GPC. An interactive session can be requested that will allow a user to list, rearrange or remove files with the HSI client. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

== Guidelines ==
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== Access Through the Queue System ==
All access to the archive system is through the queue system.
=== Scripted File Transfers ===
File transfers in and out of the archive should be scripted into jobs and submitted to the archive queue.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_file_transfer
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then

</pre>

=== Staging Data for Analysis ===

Job dependencies can be used to make analysis jobs wait for data staging before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency:
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of the archive.
|-
| lcd,lls
| ''Local'' commands.
|}

Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz
</pre>

More complex operations can be performed using a Here Document.
<pre>
hsi <<-EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
</pre>

=== HSI vs. FTP ===
HSI syntax and usage is very similar to that of FTP. Please note the following information adapted from the HSI man page:

HSI supports several of the commonly used FTP commands, including "dir","get","ls","mdelete","mget","put","mput" and "prompt", with the following differences:

* The "dir" command is an alias for "ls" in HSI. The "ls" command supports an extensive set of options for displaying files, including wildcard pattern-matching, and the ability to recursively list a directory tree
* The "put" and "get" family of commands support recursion
* There are "conditional put" and "conditional" get commands (cput, cget)

* The syntax for renaming local files when storing files to HPSS or retrieving files from HPSS is different than FTP. With HSI, the syntax is always
<pre>
"local_file : HPSS_file"
</pre>
and multiple such pairs may be specified on a single command line.
With FTP, the local filename is specified first on a "put" command, and second on a "get" command.

For example, when using HSI to store the local file "file1" as HPSS file "hpss_file1", then retrieve it back to the local filesystem as "file1.bak", the following commands could be used:
<pre>
put file1 : hpss_file1
get file1.bak : hpss_file1
</pre>

* With FTP, the following commands could be used:
<pre>
put file1 hpss_file1
get hpss_file1 file1.bak
</pre>
* The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.

=== HSI Documentation ===
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Examples ===

* Save a "tar file" of C source programs and header files:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''Using HTAR''' ==

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==
[[GPC_Quickstart|Submitting_A_Batch_Job]]
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

MARS

2011-06-20T17:27:08Z

Knecht: purge interactive examples

== '''Massive Archive and Retrieval System''' ==

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface will be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the archive. It also creates a separate index file that can be accessed quickly.

User access will be controlled by the job scheduling system of the GPC. An interactive session can be requested that will allow a user to list, rearrange or remove files with the HSI client. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

== Guidelines ==
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== Access Through the Queue System ==
All access to the archive system is through the queue system.
=== Interactive Access ===
To get an interactive session and use the HSI client, use the -I option of qsub.
<pre>
gpc04 $ qsub -q archive -I
hpss-archive01:~ $ hsi ls
******************************************************************
* Welcome to the Massive Archive and Restore System @ SciNet *
* *
* Contact Information: support@scinet.utoronto.ca *
* NOTE: do not transfer SMALL FILES with HSI. Use HTAR instead *
******************************************************************
[HSI]/archive/group/user->
</pre>

=== Scripted File Transfers ===
File transfers in and out of the archive should be scripted into jobs and submitted to the archive queue.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_file_transfer
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then

</pre>

=== Staging Data for Analysis ===

Job dependencies can be used to make analysis jobs wait for data staging before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency:
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of the archive.
|-
| lcd,lls
| ''Local'' commands.
|}

Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz
</pre>

More complex operations can be performed using a Here Document.
<pre>
hsi <<-EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
</pre>

=== HSI vs. FTP ===
HSI syntax and usage is very similar to that of FTP. Please note the following information adapted from the HSI man page:

HSI supports several of the commonly used FTP commands, including "dir","get","ls","mdelete","mget","put","mput" and "prompt", with the following differences:

* The "dir" command is an alias for "ls" in HSI. The "ls" command supports an extensive set of options for displaying files, including wildcard pattern-matching, and the ability to recursively list a directory tree
* The "put" and "get" family of commands support recursion
* There are "conditional put" and "conditional" get commands (cput, cget)

* The syntax for renaming local files when storing files to HPSS or retrieving files from HPSS is different than FTP. With HSI, the syntax is always
<pre>
"local_file : HPSS_file"
</pre>
and multiple such pairs may be specified on a single command line.
With FTP, the local filename is specified first on a "put" command, and second on a "get" command.

For example, when using HSI to store the local file "file1" as HPSS file "hpss_file1", then retrieve it back to the local filesystem as "file1.bak", the following commands could be used:
<pre>
put file1 : hpss_file1
get file1.bak : hpss_file1
</pre>

* With FTP, the following commands could be used:
<pre>
put file1 hpss_file1
get hpss_file1 file1.bak
</pre>
* The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.

=== HSI Documentation ===
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Examples ===

* Save a "tar file" of C source programs and header files:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''Using HTAR''' ==

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==
[[GPC_Quickstart|Submitting_A_Batch_Job]]
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

MARS

2011-06-20T17:25:48Z

Knecht: /* HSI vs. FTP */

== '''Massive Archive and Retrieval System''' ==

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface will be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the archive. It also creates a separate index file that can be accessed quickly.

User access will be controlled by the job scheduling system of the GPC. An interactive session can be requested that will allow a user to list, rearrange or remove files with the HSI client. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

== Guidelines ==
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== Access Through the Queue System ==
All access to the archive system is through the queue system.
=== Interactive Access ===
To get an interactive session and use the HSI client, use the -I option of qsub.
<pre>
gpc04 $ qsub -q archive -I
hpss-archive01:~ $ hsi ls
******************************************************************
* Welcome to the Massive Archive and Restore System @ SciNet *
* *
* Contact Information: support@scinet.utoronto.ca *
* NOTE: do not transfer SMALL FILES with HSI. Use HTAR instead *
******************************************************************
[HSI]/archive/group/user->
</pre>

=== Scripted File Transfers ===
File transfers in and out of the archive should be scripted into jobs and submitted to the archive queue.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_file_transfer
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then

</pre>

=== Staging Data for Analysis ===

Job dependencies can be used to make analysis jobs wait for data staging before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency:
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of the archive.
|-
| lcd,lls
| ''Local'' commands.
|}

Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz
</pre>

More complex operations can be performed using a Here Document.
<pre>
hsi <<-EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
</pre>

=== HSI vs. FTP ===
HSI syntax and usage is very similar to that of FTP. Please note the following information adapted from the HSI man page:

HSI supports several of the commonly used FTP commands, including "dir","get","ls","mdelete","mget","put","mput" and "prompt", with the following differences:

* The "dir" command is an alias for "ls" in HSI. The "ls" command supports an extensive set of options for displaying files, including wildcard pattern-matching, and the ability to recursively list a directory tree
* The "put" and "get" family of commands support recursion
* There are "conditional put" and "conditional" get commands (cput, cget)

* The syntax for renaming local files when storing files to HPSS or retrieving files from HPSS is different than FTP. With HSI, the syntax is always
<pre>
"local_file : HPSS_file"
</pre>
and multiple such pairs may be specified on a single command line.
With FTP, the local filename is specified first on a "put" command, and second on a "get" command.

For example, when using HSI to store the local file "file1" as HPSS file "hpss_file1", then retrieve it back to the local filesystem as "file1.bak", the following commands could be used:
<pre>
put file1 : hpss_file1
get file1.bak : hpss_file1
</pre>

* With FTP, the following commands could be used:
<pre>
put file1 hpss_file1
get hpss_file1 file1.bak
</pre>
* The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.

=== HSI Documentation ===
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Examples ===

* Put a subdirectory ''LargeFiles'' and all its contents recursively. You may use '-u' option to resume a previously disrupted session (as rsync would do).
<pre>
hsi "prompt; mput -R -u LargeFiles"
</pre>

* Interactively descend into the ''Source'' directory and move all files which end in ".h" into a sibling directory (ie, a directory at the same level in the tree as "Source") named "Include":
<pre>
hsi <RETURN>
[HSI] cd Source
[HSI] mv *.h ../Include
</pre>

* Delete all files beginning with "m" and ending with 9101 (note that this is an interactive request, not a one-liner request, so the wildcard path does not need quotes to preserve it):
<pre>
hsi <RETURN>
[HSI] delete m*9101
</pre>

* Interactively delete all files beginning with H and ending with a digit, and ask for verification before deleting each such file.
<pre>
hsi <RETURN>
[HSI] mdel H*[0-9]
</pre>

* From a shell, save your local files that begin with the letter "c" (let the UN*X shell resolve the wild-card path pattern in terms of your local files by not enclosing it in quotes):
<pre>
hsi mput c*
</pre>

* From a shell, get all files in the subdirectory ''subdirA'' which begin with the letters "b" or "c" (surrounding the wildcard path in single quotes prevents shells on UNIX systems from processing the wild card pattern):
<pre>
hsi get ’subdirA/[bc]*’
</pre>

* Save a "tar file" of C source programs and header files:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''Using HTAR''' ==

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==
[[GPC_Quickstart|Submitting_A_Batch_Job]]
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

MARS

2011-06-20T17:24:49Z

Knecht: copy in ftp vs. hsi notes

== '''Massive Archive and Retrieval System''' ==

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface will be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the archive. It also creates a separate index file that can be accessed quickly.

User access will be controlled by the job scheduling system of the GPC. An interactive session can be requested that will allow a user to list, rearrange or remove files with the HSI client. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

== Guidelines ==
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== Access Through the Queue System ==
All access to the archive system is through the queue system.
=== Interactive Access ===
To get an interactive session and use the HSI client, use the -I option of qsub.
<pre>
gpc04 $ qsub -q archive -I
hpss-archive01:~ $ hsi ls
******************************************************************
* Welcome to the Massive Archive and Restore System @ SciNet *
* *
* Contact Information: support@scinet.utoronto.ca *
* NOTE: do not transfer SMALL FILES with HSI. Use HTAR instead *
******************************************************************
[HSI]/archive/group/user->
</pre>

=== Scripted File Transfers ===
File transfers in and out of the archive should be scripted into jobs and submitted to the archive queue.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_file_transfer
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then

</pre>

=== Staging Data for Analysis ===

Job dependencies can be used to make analysis jobs wait for data staging before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency:
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of the archive.
|-
| lcd,lls
| ''Local'' commands.
|}

Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz
</pre>

More complex operations can be performed using a Here Document.
<pre>
hsi <<-EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
</pre>

=== HSI vs. FTP ===
Please note the following information adapted from the HSI man page:

HSI supports several of the commonly used FTP commands, including "dir","get","ls","mdelete","mget","put","mput" and "prompt", with the following differences:

* The "dir" command is an alias for "ls" in HSI. The "ls" command supports an extensive set of options for displaying files, including wildcard pattern-matching, and the ability to recursively list a directory tree
* The "put" and "get" family of commands support recursion
* There are "conditional put" and "conditional" get commands (cput, cget)

* The syntax for renaming local files when storing files to HPSS or retrieving files from HPSS is different than FTP. With HSI, the syntax is always
<pre>
"local_file : HPSS_file"
</pre>
and multiple such pairs may be specified on a single command line.
With FTP, the local filename is specified first on a "put" command, and second on a "get" command.

For example, when using HSI to store the local file "file1" as HPSS file "hpss_file1", then retrieve it back to the local filesystem as "file1.bak", the following commands could be used:
<pre>
put file1 : hpss_file1
get file1.bak : hpss_file1
</pre>

* With FTP, the following commands could be used:
<pre>
put file1 hpss_file1
get hpss_file1 file1.bak
</pre>
* The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.

=== HSI Documentation ===
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Examples ===

* Put a subdirectory ''LargeFiles'' and all its contents recursively. You may use '-u' option to resume a previously disrupted session (as rsync would do).
<pre>
hsi "prompt; mput -R -u LargeFiles"
</pre>

* Interactively descend into the ''Source'' directory and move all files which end in ".h" into a sibling directory (ie, a directory at the same level in the tree as "Source") named "Include":
<pre>
hsi <RETURN>
[HSI] cd Source
[HSI] mv *.h ../Include
</pre>

* Delete all files beginning with "m" and ending with 9101 (note that this is an interactive request, not a one-liner request, so the wildcard path does not need quotes to preserve it):
<pre>
hsi <RETURN>
[HSI] delete m*9101
</pre>

* Interactively delete all files beginning with H and ending with a digit, and ask for verification before deleting each such file.
<pre>
hsi <RETURN>
[HSI] mdel H*[0-9]
</pre>

* From a shell, save your local files that begin with the letter "c" (let the UN*X shell resolve the wild-card path pattern in terms of your local files by not enclosing it in quotes):
<pre>
hsi mput c*
</pre>

* From a shell, get all files in the subdirectory ''subdirA'' which begin with the letters "b" or "c" (surrounding the wildcard path in single quotes prevents shells on UNIX systems from processing the wild card pattern):
<pre>
hsi get ’subdirA/[bc]*’
</pre>

* Save a "tar file" of C source programs and header files:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''Using HTAR''' ==

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==
[[GPC_Quickstart|Submitting_A_Batch_Job]]
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

MARS

2011-06-20T17:17:04Z

Knecht: staging updated

== '''Massive Archive and Retrieval System''' ==

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface will be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the archive. It also creates a separate index file that can be accessed quickly.

User access will be controlled by the job scheduling system of the GPC. An interactive session can be requested that will allow a user to list, rearrange or remove files with the HSI client. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

== Guidelines ==
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== Access Through the Queue System ==
All access to the archive system is through the queue system.
=== Interactive Access ===
To get an interactive session and use the HSI client, use the -I option of qsub.
<pre>
gpc04 $ qsub -q archive -I
hpss-archive01:~ $ hsi ls
******************************************************************
* Welcome to the Massive Archive and Restore System @ SciNet *
* *
* Contact Information: support@scinet.utoronto.ca *
* NOTE: do not transfer SMALL FILES with HSI. Use HTAR instead *
******************************************************************
[HSI]/archive/group/user->
</pre>

=== Scripted File Transfers ===
File transfers in and out of the archive should be scripted into jobs and submitted to the archive queue.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_file_transfer
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then

</pre>

=== Staging Data for Analysis ===

Job dependencies can be used to make analysis jobs wait for data staging before starting. The qsub flag is
<pre>
-W depend=afterok:<JOBID>
</pre>
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency:
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of the archive.
|-
| lcd,lls
| ''Local'' commands.
|}

Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz
</pre>

More complex operations can be performed using a Here Document.
<pre>
hsi <<-EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
</pre>

=== HSI Documentation ===
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Examples ===

* Put a subdirectory ''LargeFiles'' and all its contents recursively. You may use '-u' option to resume a previously disrupted session (as rsync would do).
<pre>
hsi "prompt; mput -R -u LargeFiles"
</pre>

* Interactively descend into the ''Source'' directory and move all files which end in ".h" into a sibling directory (ie, a directory at the same level in the tree as "Source") named "Include":
<pre>
hsi <RETURN>
[HSI] cd Source
[HSI] mv *.h ../Include
</pre>

* Delete all files beginning with "m" and ending with 9101 (note that this is an interactive request, not a one-liner request, so the wildcard path does not need quotes to preserve it):
<pre>
hsi <RETURN>
[HSI] delete m*9101
</pre>

* Interactively delete all files beginning with H and ending with a digit, and ask for verification before deleting each such file.
<pre>
hsi <RETURN>
[HSI] mdel H*[0-9]
</pre>

* From a shell, save your local files that begin with the letter "c" (let the UN*X shell resolve the wild-card path pattern in terms of your local files by not enclosing it in quotes):
<pre>
hsi mput c*
</pre>

* From a shell, get all files in the subdirectory ''subdirA'' which begin with the letters "b" or "c" (surrounding the wildcard path in single quotes prevents shells on UNIX systems from processing the wild card pattern):
<pre>
hsi get ’subdirA/[bc]*’
</pre>

* Save a "tar file" of C source programs and header files:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''Using HTAR''' ==

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==
[[GPC_Quickstart|Submitting_A_Batch_Job]]
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

MARS

2011-06-20T17:08:54Z

Knecht: /* Using HSI */

== '''Massive Archive and Retrieval System''' ==

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface will be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the archive. It also creates a separate index file that can be accessed quickly.

User access will be controlled by the job scheduling system of the GPC. An interactive session can be requested that will allow a user to list, rearrange or remove files with the HSI client. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

== Guidelines ==
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== Access Through the Queue System ==
All access to the archive system is through the queue system.
=== Interactive Access ===
To get an interactive session and use the HSI client, use the -I option of qsub.
<pre>
gpc04 $ qsub -q archive -I
hpss-archive01:~ $ hsi ls
******************************************************************
* Welcome to the Massive Archive and Restore System @ SciNet *
* *
* Contact Information: support@scinet.utoronto.ca *
* NOTE: do not transfer SMALL FILES with HSI. Use HTAR instead *
******************************************************************
[HSI]/archive/group/user->
</pre>

=== Scripted File Transfers ===
File transfers in and out of the archive should be scripted into jobs and submitted to the archive queue.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_file_transfer
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then

</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of the archive.
|-
| lcd,lls
| ''Local'' commands.
|}

Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz
</pre>

More complex operations can be performed using a Here Document.
<pre>
hsi <<-EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/$USER/examples/
cput -R -u *
EOF
</pre>

=== HSI Documentation ===
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Examples ===

* Put a subdirectory ''LargeFiles'' and all its contents recursively. You may use '-u' option to resume a previously disrupted session (as rsync would do).
<pre>
hsi "prompt; mput -R -u LargeFiles"
</pre>

* Interactively descend into the ''Source'' directory and move all files which end in ".h" into a sibling directory (ie, a directory at the same level in the tree as "Source") named "Include":
<pre>
hsi <RETURN>
[HSI] cd Source
[HSI] mv *.h ../Include
</pre>

* Delete all files beginning with "m" and ending with 9101 (note that this is an interactive request, not a one-liner request, so the wildcard path does not need quotes to preserve it):
<pre>
hsi <RETURN>
[HSI] delete m*9101
</pre>

* Interactively delete all files beginning with H and ending with a digit, and ask for verification before deleting each such file.
<pre>
hsi <RETURN>
[HSI] mdel H*[0-9]
</pre>

* From a shell, save your local files that begin with the letter "c" (let the UN*X shell resolve the wild-card path pattern in terms of your local files by not enclosing it in quotes):
<pre>
hsi mput c*
</pre>

* From a shell, get all files in the subdirectory ''subdirA'' which begin with the letters "b" or "c" (surrounding the wildcard path in single quotes prevents shells on UNIX systems from processing the wild card pattern):
<pre>
hsi get ’subdirA/[bc]*’
</pre>

* Save a "tar file" of C source programs and header files:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''Using HTAR''' ==

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==
[[GPC_Quickstart|Submitting_A_Batch_Job]]
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

* sample '''analysis''' (depends on previous data-restore.sh execution)
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

MARS

2011-06-20T17:06:39Z

Knecht: /* Using HSI */

== '''Massive Archive and Retrieval System''' ==

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface will be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the archive. It also creates a separate index file that can be accessed quickly.

User access will be controlled by the job scheduling system of the GPC. An interactive session can be requested that will allow a user to list, rearrange or remove files with the HSI client. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

== Guidelines ==
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== Access Through the Queue System ==
All access to the archive system is through the queue system.
=== Interactive Access ===
To get an interactive session and use the HSI client, use the -I option of qsub.
<pre>
gpc04 $ qsub -q archive -I
hpss-archive01:~ $ hsi ls
******************************************************************
* Welcome to the Massive Archive and Restore System @ SciNet *
* *
* Contact Information: support@scinet.utoronto.ca *
* NOTE: do not transfer SMALL FILES with HSI. Use HTAR instead *
******************************************************************
[HSI]/archive/group/user->
</pre>

=== Scripted File Transfers ===
File transfers in and out of the archive should be scripted into jobs and submitted to the archive queue.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_file_transfer
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then

</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of the archive.
|-
| lcd,lls
| ''Local'' commands.
|}

Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz
</pre>

More complex operations can be performed using a Here Document.
<pre>
hsi <<-EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/knecht/examples/
cput -R -u *
EOF
</pre>

=== HSI Documentation ===
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Examples ===

* Put a subdirectory ''LargeFiles'' and all its contents recursively. You may use '-u' option to resume a previously disrupted session (as rsync would do).
<pre>
hsi "prompt; mput -R -u LargeFiles"
</pre>

* Interactively descend into the ''Source'' directory and move all files which end in ".h" into a sibling directory (ie, a directory at the same level in the tree as "Source") named "Include":
<pre>
hsi <RETURN>
[HSI] cd Source
[HSI] mv *.h ../Include
</pre>

* Delete all files beginning with "m" and ending with 9101 (note that this is an interactive request, not a one-liner request, so the wildcard path does not need quotes to preserve it):
<pre>
hsi <RETURN>
[HSI] delete m*9101
</pre>

* Interactively delete all files beginning with H and ending with a digit, and ask for verification before deleting each such file.
<pre>
hsi <RETURN>
[HSI] mdel H*[0-9]
</pre>

* From a shell, save your local files that begin with the letter "c" (let the UN*X shell resolve the wild-card path pattern in terms of your local files by not enclosing it in quotes):
<pre>
hsi mput c*
</pre>

* From a shell, get all files in the subdirectory ''subdirA'' which begin with the letters "b" or "c" (surrounding the wildcard path in single quotes prevents shells on UNIX systems from processing the wild card pattern):
<pre>
hsi get ’subdirA/[bc]*’
</pre>

* Save a "tar file" of C source programs and header files:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''Using HTAR''' ==

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==
[[GPC_Quickstart|Submitting_A_Batch_Job]]
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

* sample '''analysis''' (depends on previous data-restore.sh execution)
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

MARS

2011-06-10T21:37:13Z

Knecht: /* Using HSI */

== '''Massive Archive and Retrieval System''' ==

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface will be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the archive. It also creates a separate index file that can be accessed quickly.

User access will be controlled by the job scheduling system of the GPC. An interactive session can be requested that will allow a user to list, rearrange or remove files with the HSI client. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

== Guidelines ==
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== Access Through the Queue System ==
All access to the archive system is through the queue system.
=== Interactive Access ===
To get an interactive session and use the HSI client, use the -I option of qsub.
<pre>
gpc04 $ qsub -q archive -I
hpss-archive01:~ $ hsi ls
******************************************************************
* Welcome to the Massive Archive and Restore System @ SciNet *
* *
* Contact Information: support@scinet.utoronto.ca *
* NOTE: do not transfer SMALL FILES with HSI. Use HTAR instead *
******************************************************************
[HSI]/archive/group/user->
</pre>

=== Scripted File Transfers ===
File transfers in and out of the archive should be scripted into jobs and submitted to the archive queue.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_file_transfer
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then

</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| cd
|
|-
| mkdir
|
|-
| ls
| list
|-
| rm
| remove
|}

Simple commands can be executed on a single line.
<pre>
hsi "mkdir examples; cd examples; cput example_data.tgz
</pre>

More complex operations can be performed using a Here Document.
<pre>
hsi <<-EOF
mkdir -p examples/201106
cd examples
mv example_data.tgz 201106/
lcd /scratch/knecht/examples/
cput -R -u *
EOF
</pre>

=== HSI Documentation ===
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Examples ===

* Put a subdirectory ''LargeFiles'' and all its contents recursively. You may use '-u' option to resume a previously disrupted session (as rsync would do).
<pre>
hsi "prompt; mput -R -u LargeFiles"
</pre>

* Interactively descend into the ''Source'' directory and move all files which end in ".h" into a sibling directory (ie, a directory at the same level in the tree as "Source") named "Include":
<pre>
hsi <RETURN>
[HSI] cd Source
[HSI] mv *.h ../Include
</pre>

* Delete all files beginning with "m" and ending with 9101 (note that this is an interactive request, not a one-liner request, so the wildcard path does not need quotes to preserve it):
<pre>
hsi <RETURN>
[HSI] delete m*9101
</pre>

* Interactively delete all files beginning with H and ending with a digit, and ask for verification before deleting each such file.
<pre>
hsi <RETURN>
[HSI] mdel H*[0-9]
</pre>

* From a shell, save your local files that begin with the letter "c" (let the UN*X shell resolve the wild-card path pattern in terms of your local files by not enclosing it in quotes):
<pre>
hsi mput c*
</pre>

* From a shell, get all files in the subdirectory ''subdirA'' which begin with the letters "b" or "c" (surrounding the wildcard path in single quotes prevents shells on UNIX systems from processing the wild card pattern):
<pre>
hsi get ’subdirA/[bc]*’
</pre>

* Save a "tar file" of C source programs and header files:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''Using HTAR''' ==

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==
[[GPC_Quickstart|Submitting_A_Batch_Job]]
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

* sample '''analysis''' (depends on previous data-restore.sh execution)
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

MARS

2011-06-10T19:38:32Z

Knecht: /* Using HSI */

== '''Massive Archive and Retrieval System''' ==

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface will be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the archive. It also creates a separate index file that can be accessed quickly.

User access will be controlled by the job scheduling system of the GPC. An interactive session can be requested that will allow a user to list, rearrange or remove files with the HSI client. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

== Guidelines ==
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== Access Through the Queue System ==
All access to the archive system is through the queue system.
=== Interactive Access ===
To get an interactive session and use the HSI client, use the -I option of qsub.
<pre>
gpc04 $ qsub -q archive -I
hpss-archive01:~ $ hsi ls
******************************************************************
* Welcome to the Massive Archive and Restore System @ SciNet *
* *
* Contact Information: support@scinet.utoronto.ca *
* NOTE: do not transfer SMALL FILES with HSI. Use HTAR instead *
******************************************************************
[HSI]/archive/group/user->
</pre>

=== Scripted File Transfers ===
File transfers in and out of the archive should be scripted into jobs and submitted to the archive queue.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_file_transfer
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then

</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| cd
|
|-
| mkdir
|
|-
| ls
| list
|-
| rm
| remove
|}

Simple commands can be executed on a single line:
<pre>
hsi "mkdir examples; cd examples;

=== HSI Documentation ===
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Examples ===

* Put a subdirectory ''LargeFiles'' and all its contents recursively. You may use '-u' option to resume a previously disrupted session (as rsync would do).
<pre>
hsi "prompt; mput -R -u LargeFiles"
</pre>

* Interactively descend into the ''Source'' directory and move all files which end in ".h" into a sibling directory (ie, a directory at the same level in the tree as "Source") named "Include":
<pre>
hsi <RETURN>
[HSI] cd Source
[HSI] mv *.h ../Include
</pre>

* Delete all files beginning with "m" and ending with 9101 (note that this is an interactive request, not a one-liner request, so the wildcard path does not need quotes to preserve it):
<pre>
hsi <RETURN>
[HSI] delete m*9101
</pre>

* Interactively delete all files beginning with H and ending with a digit, and ask for verification before deleting each such file.
<pre>
hsi <RETURN>
[HSI] mdel H*[0-9]
</pre>

* From a shell, save your local files that begin with the letter "c" (let the UN*X shell resolve the wild-card path pattern in terms of your local files by not enclosing it in quotes):
<pre>
hsi mput c*
</pre>

* From a shell, get all files in the subdirectory ''subdirA'' which begin with the letters "b" or "c" (surrounding the wildcard path in single quotes prevents shells on UNIX systems from processing the wild card pattern):
<pre>
hsi get ’subdirA/[bc]*’
</pre>

* Save a "tar file" of C source programs and header files:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''Using HTAR''' ==

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==
[[GPC_Quickstart|Submitting_A_Batch_Job]]
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

* sample '''analysis''' (depends on previous data-restore.sh execution)
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

MARS

2011-06-08T21:25:59Z

Knecht: Change heading levels.

== '''Massive Archive and Retrieval System''' ==

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface will be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the archive. It also creates a separate index file that can be accessed quickly.

User access will be controlled by the job scheduling system of the GPC. An interactive session can be requested that will allow a user to list, rearrange or remove files with the HSI client. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

== Guidelines ==
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

== Access Through the Queue System ==
All access to the archive system is through the queue system.
=== Interactive Access ===
To get an interactive session and use the HSI client, use the -I option of qsub.
<pre>
gpc04 $ qsub -q archive -I
hpss-archive01:~ $ hsi ls
******************************************************************
* Welcome to the Massive Archive and Restore System @ SciNet *
* *
* Contact Information: support@scinet.utoronto.ca *
* NOTE: do not transfer SMALL FILES with HSI. Use HTAR instead *
******************************************************************
[HSI]/archive/group/user->
</pre>

=== Scripted File Transfers ===
File transfers in and out of the archive should be scripted into jobs and submitted to the archive queue.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_file_transfer
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then

</pre>

== '''Using HSI''' ==

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| ls
| list
|-
| rm
| remove
|}

=== HSI Documentation ===
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

=== Examples ===

* Interactively put a subdirectory ''LargeFiles'' and all its contents recursively. You may use '-u' option to resume a previously disrupted session (as rsync would do).
<pre>
hsi <RETURN>
[HSI] prompt
[HSI] mput -R -u LargeFiles</pre>

* Same as above, but from a shell
<pre>
hsi "prompt; mput -R -u LargeFiles"
</pre>

* Interactively descend into the ''Source'' directory and move all files which end in ".h" into a sibling directory (ie, a directory at the same level in the tree as "Source") named "Include":
<pre>
hsi <RETURN>
[HSI] cd Source
[HSI] mv *.h ../Include
</pre>

* Delete all files beginning with "m" and ending with 9101 (note that this is an interactive request, not a one-liner request, so the wildcard path does not need quotes to preserve it):
<pre>
hsi <RETURN>
[HSI] delete m*9101
</pre>

* Interactively delete all files beginning with H and ending with a digit, and ask for verification before deleting each such file.
<pre>
hsi <RETURN>
[HSI] mdel H*[0-9]
</pre>

* From a shell, save your local files that begin with the letter "c" (let the UN*X shell resolve the wild-card path pattern in terms of your local files by not enclosing it in quotes):
<pre>
hsi mput c*
</pre>

* From a shell, get all files in the subdirectory ''subdirA'' which begin with the letters "b" or "c" (surrounding the wildcard path in single quotes prevents shells on UNIX systems from processing the wild card pattern):
<pre>
hsi get ’subdirA/[bc]*’
</pre>

* Save a "tar file" of C source programs and header files:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

== '''Using HTAR''' ==

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

== '''More detailed examples''' ==
[[GPC_Quickstart|Submitting_A_Batch_Job]]
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

* sample '''analysis''' (depends on previous data-restore.sh execution)
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

MARS

2011-06-08T21:15:38Z

Knecht: /* Using HSI */

=== '''Massive Archive and Retrieval System''' ===

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface will be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the archive. It also creates a separate index file that can be accessed quickly.

User access will be controlled by the job scheduling system of the GPC. An interactive session can be requested that will allow a user to list, rearrange or remove files with the HSI client. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

=== Guidelines ===
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

=== Access Through the Queue System ===
All access to the archive system is through the queue system.
== Interactive Access ==
To get an interactive session and use the HSI client, use the -I option of qsub.
<pre>
gpc04 $ qsub -q archive -I
hpss-archive01:~ $ hsi ls
******************************************************************
* Welcome to the Massive Archive and Restore System @ SciNet *
* *
* Contact Information: support@scinet.utoronto.ca *
* NOTE: do not transfer SMALL FILES with HSI. Use HTAR instead *
******************************************************************
[HSI]/archive/group/user->
</pre>

== Scripted File Transfers ==
File transfers in and out of the archive should be scripted into jobs and submitted to the archive queue.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_file_transfer
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then

</pre>

=== '''Using HSI''' ===

HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally stores a file only if the HPSS file does not exist
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
|-
| ls
| list
|-
| rm
| remove
|}

== HSI Documentation ==
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]

= Examples =

* Interactively put a subdirectory ''LargeFiles'' and all its contents recursively. You may use '-u' option to resume a previously disrupted session (as rsync would do).
<pre>
hsi <RETURN>
[HSI] prompt
[HSI] mput -R -u LargeFiles</pre>

* Same as above, but from a shell
<pre>
hsi "prompt; mput -R -u LargeFiles"
</pre>

* Interactively descend into the ''Source'' directory and move all files which end in ".h" into a sibling directory (ie, a directory at the same level in the tree as "Source") named "Include":
<pre>
hsi <RETURN>
[HSI] cd Source
[HSI] mv *.h ../Include
</pre>

* Delete all files beginning with "m" and ending with 9101 (note that this is an interactive request, not a one-liner request, so the wildcard path does not need quotes to preserve it):
<pre>
hsi <RETURN>
[HSI] delete m*9101
</pre>

* Interactively delete all files beginning with H and ending with a digit, and ask for verification before deleting each such file.
<pre>
hsi <RETURN>
[HSI] mdel H*[0-9]
</pre>

* From a shell, save your local files that begin with the letter "c" (let the UN*X shell resolve the wild-card path pattern in terms of your local files by not enclosing it in quotes):
<pre>
hsi mput c*
</pre>

* From a shell, get all files in the subdirectory ''subdirA'' which begin with the letters "b" or "c" (surrounding the wildcard path in single quotes prevents shells on UNIX systems from processing the wild card pattern):
<pre>
hsi get ’subdirA/[bc]*’
</pre>

* Save a "tar file" of C source programs and header files:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

=== '''Using HTAR''' ===

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

=== '''More detailed examples''' ===
[[GPC_Quickstart|Submitting_A_Batch_Job]]
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

* sample '''analysis''' (depends on previous data-restore.sh execution)
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

MARS

2011-06-08T18:59:05Z

Knecht:

=== '''Massive Archive and Retrieval System''' ===

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface will be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the archive. It also creates a separate index file that can be accessed quickly.

User access will be controlled by the job scheduling system of the GPC. An interactive session can be requested that will allow a user to list, rearrange or remove files with the HSI client. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

=== Guidelines ===
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

=== Access Through the Queue System ===
All access to the archive system is through the queue system.
== Interactive Access ==
To get an interactive session and use the HSI client, use the -I option of qsub.
<pre>
gpc04 $ qsub -q archive -I
hpss-archive01:~ $ hsi ls
******************************************************************
* Welcome to the Massive Archive and Restore System @ SciNet *
* *
* Contact Information: support@scinet.utoronto.ca *
* NOTE: do not transfer SMALL FILES with HSI. Use HTAR instead *
******************************************************************
[HSI]/archive/group/user->
</pre>

== Scripted File Transfers ==
File transfers in and out of the archive should be scripted into jobs and submitted to the archive queue.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_file_transfer
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
if [ ! $? == 0 ];then

</pre>

=== '''Using HSI''' ===

* Link to docs at Gleicher Ent.

* Interactively put a subdirectory ''LargeFiles'' and all its contents recursively. You may use '-u' option to resume a previously disrupted session (as rsync would do).
<pre>
hsi <RETURN>
[HSI] prompt
[HSI] mput -R -u LargeFiles</pre>

* Same as above, but from a shell
<pre>
hsi "prompt; mput -R -u LargeFiles"
</pre>

* Interactively descend into the ''Source'' directory and move all files which end in ".h" into a sibling directory (ie, a directory at the same level in the tree as "Source") named "Include":
<pre>
hsi <RETURN>
[HSI] cd Source
[HSI] mv *.h ../Include
</pre>

* Delete all files beginning with "m" and ending with 9101 (note that this is an interactive request, not a one-liner request, so the wildcard path does not need quotes to preserve it):
<pre>
hsi <RETURN>
[HSI] delete m*9101
</pre>

* Interactively delete all files beginning with H and ending with a digit, and ask for verification before deleting each such file.
<pre>
hsi <RETURN>
[HSI] mdel H*[0-9]
</pre>

* From a shell, save your local files that begin with the letter "c" (let the UN*X shell resolve the wild-card path pattern in terms of your local files by not enclosing it in quotes):
<pre>
hsi mput c*
</pre>

* From a shell, get all files in the subdirectory ''subdirA'' which begin with the letters "b" or "c" (surrounding the wildcard path in single quotes prevents shells on UNIX systems from processing the wild card pattern):
<pre>
hsi get ’subdirA/[bc]*’
</pre>

* Save a "tar file" of C source programs and header files:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

=== '''Using HTAR''' ===

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

=== '''More detailed examples''' ===
[[GPC_Quickstart|Submitting_A_Batch_Job]]
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

* sample '''analysis''' (depends on previous data-restore.sh execution)
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

MARS

2011-06-08T18:54:44Z

Knecht:

=== '''Massive Archive and Retrieval System''' ===

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchicle storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.

Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
following utilities:
* HSI is a client with an ftp-like interface will be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
* HTAR is a utility that creates tar format archives resident in the archive. It also creates a separate index file that can be accessed quickly.

User access will be controlled by the job scheduling system of the GPC. An interactive session can be requested that will allow a user to list, rearrange or remove files with the HSI client. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.

=== Guidelines ===
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
* Make sure to check the application return code and check the log file for errors after all data transfers.
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

=== Access Through the Queue System ===
All access to the archive system is through the queue system. To get an interactive session and use the HSI client, use the -I option of qsub.
== Interactive Access ==
<pre>
gpc04 $ qsub -q archive -I
hpss-archive01:~ $ hsi ls
******************************************************************
* Welcome to the Massive Archive and Restore System @ SciNet *
* *
* Contact Information: support@scinet.utoronto.ca *
* NOTE: do not transfer SMALL FILES with HSI. Use HTAR instead *
******************************************************************
[HSI]/archive/group/user->
</pre>

== Scripted File Transfers ==
File transfers in and out of the archive should be scripted into jobs and submitted to the archive queue.
<pre>
#!/bin/env bash
#PBS -q archive
#PBS -N hsi_file_transfer
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

/usr/local/bin/hsi -v <<EOF
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
EOF
</pre>
if [ ! $? == 0 ];then

=== '''Using HSI''' ===

* Link to docs at Gleicher Ent.

* Interactively put a subdirectory ''LargeFiles'' and all its contents recursively. You may use '-u' option to resume a previously disrupted session (as rsync would do).
<pre>
hsi <RETURN>
[HSI] prompt
[HSI] mput -R -u LargeFiles</pre>

* Same as above, but from a shell
<pre>
hsi "prompt; mput -R -u LargeFiles"
</pre>

* Interactively descend into the ''Source'' directory and move all files which end in ".h" into a sibling directory (ie, a directory at the same level in the tree as "Source") named "Include":
<pre>
hsi <RETURN>
[HSI] cd Source
[HSI] mv *.h ../Include
</pre>

* Delete all files beginning with "m" and ending with 9101 (note that this is an interactive request, not a one-liner request, so the wildcard path does not need quotes to preserve it):
<pre>
hsi <RETURN>
[HSI] delete m*9101
</pre>

* Interactively delete all files beginning with H and ending with a digit, and ask for verification before deleting each such file.
<pre>
hsi <RETURN>
[HSI] mdel H*[0-9]
</pre>

* From a shell, save your local files that begin with the letter "c" (let the UN*X shell resolve the wild-card path pattern in terms of your local files by not enclosing it in quotes):
<pre>
hsi mput c*
</pre>

* From a shell, get all files in the subdirectory ''subdirA'' which begin with the letters "b" or "c" (surrounding the wildcard path in single quotes prevents shells on UNIX systems from processing the wild card pattern):
<pre>
hsi get ’subdirA/[bc]*’
</pre>

* Save a "tar file" of C source programs and header files:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

=== '''Using HTAR''' ===

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

=== '''More detailed examples''' ===
[[GPC_Quickstart|Submitting_A_Batch_Job]]
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

* sample '''analysis''' (depends on previous data-restore.sh execution)
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

MARS

2011-06-07T19:49:50Z

Knecht: /* Using the batch queue */

=== '''Massive Archive and Retrieval System''' ===

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

'''NOTE: PLEASE TAKE THE TIME TO AT LEAST READ THIS PAGE IN FULL.'''

The '''MARS''' deployment at SciNet is an effort to offer a more efficient way to offoad/archive data from the most active file systems (scratch and project) than our current TSM-HSM solution, still without having to deal directly with the tape library or "tape commands"

The system is a combination of the underlaying hardware infrastructure, 3 software components, ''HPSS'', ''HSI'' and ''HTAR'', plus some environment customization.

* '''HPSS''': the main software component, best described as a very scalable engine running on a "blackbox" made of disks and tapes, to support the Archive and Retrieve operations. [http://www.hpss-collaboration.org/index.shtml High Performance Storage System - HPSS] is the result of over a decade of collaboration among five Department of Energy laboratories and IBM, with significant contributions by universities and other laboratories worldwide. For now the best way for SciNet users to [https://support.scinet.utoronto.ca/wiki/index.php/HPSS_compared_to_HSM-TSM understand HPSS] may be to compare it with our existing HSM-TSM implementation.

* '''HSI''': it may be best understood as a supercharged ftp like interface, specially designed by [http://www.mgleicher.us/GEL/hsi/ Gleicher Enterprises] to act as a front-end for HPSS, gathering some of the best features you would encounter on a shell, rsync and GridFTP (and a few more). It enables users to transfer whole directory trees from /project and /scratch, therefore freeing up space. HSI is most suitable when those directory trees do not contain too many small files to start with, or when you already have a series of tarballs.

* '''HTAR''': similarly, htar is sort of a "super-tar" application, also specially designed by [http://www.mgleicher.us/GEL/htar/ Gleicher Enterprises] to interact with HPSS, allowing users to build and automatically transfer tarballs to HPSS on the fly. HTAR is most suitable to aggregate whole directory trees. When HTAR creates the TAR file, it also builds an index file, with a ".idx" suffix added, which is stored in the same directory as the TAR file.

=== '''General guide lines''' ===
* IN/OUT transfers to HPSS using HSI is bound to maximum of about '''4 files/second'''. Therefore do not attempt to transfer directories with too many small files inside. Not only it will take "forever", it will induce a lot of ware and tear on the library's robot mechanism as well as the tapes themselves in case of recalls. Instead use HTAR, so the files are aggregated while being sent to HPSS
* The maximum size that an individual file can have inside an HTAR is '''68GB'''. Please be sure to identify and "fish out" those files that are larger than 68GB from the directories and transfer them with HSI
* The maximum size of a tar file that HPSS will take is '''1TB'''. Please do not generate tarballs that large.
* The maximum number of files in a htar is '''1 million'''. Please, break-up your htar segments as required.
* These guide lines may be strict, but for as long as they are followed the system will perform reasonably well

=== '''Performance considerations''' ===
* Files are kept on disk-cache for as long as possible, so as to avoid tape operations during recalls.
* Average transfer rates with '''HSI'''
No small files, average > 10MB/file:
* write: 100-130MB/s
* read: 450-600MB/s (IF no recall from tapes required)

* '''NOTE: do not use HSI with small files (< 1MB/file). It would take over 1 week to transfer 1 TB. If we find that you are abusing the system we'll suspend your privileges'''

* Average transfer rates with '''HTAR'''
Average files size > 1MB
* write: 120MB/s
* read: 480MB/s (IF no recall from tapes required)

Not too many small files, average > 100KB/file:
* write: 64MB/s
* read: 170MB/s (IF no recall from tapes required)

* Average transfer rates from '''tapes''', if stage is required (add to the above estimates)
* read: 80-100MB/s per tape drive.
* maximum of 4 drives may be used per hsi/htar session

=== '''Quick Reference''' ===

* Users must request authorization to access MARS@SciNet. To run HSI or HTAR with HPSS please login to the '''gpc-archive01''' node.
* To browse the contents of your HPSS archive just type '''hsi''' on a shell to get the ''hsi prompt''. Then use simple commands such as '''ls''', '''pwd''', '''cd''' to navigate your way around. You may also use [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''help'''] from the hsi prompt.
* Files are organized inside HPSS in the same fashion as in /project. Users in the same group have read permissions to each other's archives.
<pre>
[HSI]/archive/<group>/<user>
</pre>
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

=== '''Using HSI''' ===

* Interactively put a subdirectory ''LargeFiles'' and all its contents recursively. You may use '-u' option to resume a previously disrupted session (as rsync would do).
<pre>
hsi <RETURN>
[HSI] prompt
[HSI] mput -R -u LargeFiles</pre>

* Same as above, but from a shell
<pre>
hsi "prompt; mput -R -u LargeFiles"
</pre>

* Interactively descend into the ''Source'' directory and move all files which end in ".h" into a sibling directory (ie, a directory at the same level in the tree as "Source") named "Include":
<pre>
hsi <RETURN>
[HSI] cd Source
[HSI] mv *.h ../Include
</pre>

* Delete all files beginning with "m" and ending with 9101 (note that this is an interactive request, not a one-liner request, so the wildcard path does not need quotes to preserve it):
<pre>
hsi <RETURN>
[HSI] delete m*9101
</pre>

* Interactively delete all files beginning with H and ending with a digit, and ask for verification before deleting each such file.
<pre>
hsi <RETURN>
[HSI] mdel H*[0-9]
</pre>

* From a shell, save your local files that begin with the letter "c" (let the UN*X shell resolve the wild-card path pattern in terms of your local files by not enclosing it in quotes):
<pre>
hsi mput c*
</pre>

* From a shell, get all files in the subdirectory ''subdirA'' which begin with the letters "b" or "c" (surrounding the wildcard path in single quotes prevents shells on UNIX systems from processing the wild card pattern):
<pre>
hsi get ’subdirA/[bc]*’
</pre>

* Save a "tar file" of C source programs and header files:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

=== '''Using HTAR''' ===

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

=== '''Using the batch queue''' ===
[[GPC_Quickstart|Submitting_A_Batch_Job]]
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

* sample '''analysis''' (depends on previous data-restore.sh execution)
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

MARS

2011-06-07T19:13:02Z

Knecht: /* Massive Archive and Retrieval System */

=== '''Massive Archive and Retrieval System''' ===

(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)

'''NOTE: PLEASE TAKE THE TIME TO AT LEAST READ THIS PAGE IN FULL.'''

The '''MARS''' deployment at SciNet is an effort to offer a more efficient way to offoad/archive data from the most active file systems (scratch and project) than our current TSM-HSM solution, still without having to deal directly with the tape library or "tape commands"

The system is a combination of the underlaying hardware infrastructure, 3 software components, ''HPSS'', ''HSI'' and ''HTAR'', plus some environment customization.

* '''HPSS''': the main software component, best described as a very scalable engine running on a "blackbox" made of disks and tapes, to support the Archive and Retrieve operations. [http://www.hpss-collaboration.org/index.shtml High Performance Storage System - HPSS] is the result of over a decade of collaboration among five Department of Energy laboratories and IBM, with significant contributions by universities and other laboratories worldwide. For now the best way for SciNet users to [https://support.scinet.utoronto.ca/wiki/index.php/HPSS_compared_to_HSM-TSM understand HPSS] may be to compare it with our existing HSM-TSM implementation.

* '''HSI''': it may be best understood as a supercharged ftp like interface, specially designed by [http://www.mgleicher.us/GEL/hsi/ Gleicher Enterprises] to act as a front-end for HPSS, gathering some of the best features you would encounter on a shell, rsync and GridFTP (and a few more). It enables users to transfer whole directory trees from /project and /scratch, therefore freeing up space. HSI is most suitable when those directory trees do not contain too many small files to start with, or when you already have a series of tarballs.

* '''HTAR''': similarly, htar is sort of a "super-tar" application, also specially designed by [http://www.mgleicher.us/GEL/htar/ Gleicher Enterprises] to interact with HPSS, allowing users to build and automatically transfer tarballs to HPSS on the fly. HTAR is most suitable to aggregate whole directory trees. When HTAR creates the TAR file, it also builds an index file, with a ".idx" suffix added, which is stored in the same directory as the TAR file.

=== '''General guide lines''' ===
* IN/OUT transfers to HPSS using HSI is bound to maximum of about '''4 files/second'''. Therefore do not attempt to transfer directories with too many small files inside. Not only it will take "forever", it will induce a lot of ware and tear on the library's robot mechanism as well as the tapes themselves in case of recalls. Instead use HTAR, so the files are aggregated while being sent to HPSS
* The maximum size that an individual file can have inside an HTAR is '''68GB'''. Please be sure to identify and "fish out" those files that are larger than 68GB from the directories and transfer them with HSI
* The maximum size of a tar file that HPSS will take is '''1TB'''. Please do not generate tarballs that large.
* The maximum number of files in a htar is '''1 million'''. Please, break-up your htar segments as required.
* These guide lines may be strict, but for as long as they are followed the system will perform reasonably well

=== '''Performance considerations''' ===
* Files are kept on disk-cache for as long as possible, so as to avoid tape operations during recalls.
* Average transfer rates with '''HSI'''
No small files, average > 10MB/file:
* write: 100-130MB/s
* read: 450-600MB/s (IF no recall from tapes required)

* '''NOTE: do not use HSI with small files (< 1MB/file). It would take over 1 week to transfer 1 TB. If we find that you are abusing the system we'll suspend your privileges'''

* Average transfer rates with '''HTAR'''
Average files size > 1MB
* write: 120MB/s
* read: 480MB/s (IF no recall from tapes required)

Not too many small files, average > 100KB/file:
* write: 64MB/s
* read: 170MB/s (IF no recall from tapes required)

* Average transfer rates from '''tapes''', if stage is required (add to the above estimates)
* read: 80-100MB/s per tape drive.
* maximum of 4 drives may be used per hsi/htar session

=== '''Quick Reference''' ===

* Users must request authorization to access MARS@SciNet. To run HSI or HTAR with HPSS please login to the '''gpc-archive01''' node.
* To browse the contents of your HPSS archive just type '''hsi''' on a shell to get the ''hsi prompt''. Then use simple commands such as '''ls''', '''pwd''', '''cd''' to navigate your way around. You may also use [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''help'''] from the hsi prompt.
* Files are organized inside HPSS in the same fashion as in /project. Users in the same group have read permissions to each other's archives.
<pre>
[HSI]/archive/<group>/<user>
</pre>
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

=== '''Using HSI''' ===

* Interactively put a subdirectory ''LargeFiles'' and all its contents recursively. You may use '-u' option to resume a previously disrupted session (as rsync would do).
<pre>
hsi <RETURN>
[HSI] prompt
[HSI] mput -R -u LargeFiles</pre>

* Same as above, but from a shell
<pre>
hsi "prompt; mput -R -u LargeFiles"
</pre>

* Interactively descend into the ''Source'' directory and move all files which end in ".h" into a sibling directory (ie, a directory at the same level in the tree as "Source") named "Include":
<pre>
hsi <RETURN>
[HSI] cd Source
[HSI] mv *.h ../Include
</pre>

* Delete all files beginning with "m" and ending with 9101 (note that this is an interactive request, not a one-liner request, so the wildcard path does not need quotes to preserve it):
<pre>
hsi <RETURN>
[HSI] delete m*9101
</pre>

* Interactively delete all files beginning with H and ending with a digit, and ask for verification before deleting each such file.
<pre>
hsi <RETURN>
[HSI] mdel H*[0-9]
</pre>

* From a shell, save your local files that begin with the letter "c" (let the UN*X shell resolve the wild-card path pattern in terms of your local files by not enclosing it in quotes):
<pre>
hsi mput c*
</pre>

* From a shell, get all files in the subdirectory ''subdirA'' which begin with the letters "b" or "c" (surrounding the wildcard path in single quotes prevents shells on UNIX systems from processing the wild card pattern):
<pre>
hsi get ’subdirA/[bc]*’
</pre>

* Save a "tar file" of C source programs and header files:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.

=== '''Using HTAR''' ===

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cf subdirA.tar subdirA
</pre>

* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

=== '''Using the batch queue''' ===
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
showq -w class=archive

qsub -I -q archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

# This script is named: data-offload.sh

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

# individual tarballs already exist
/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

# create a tarball on-the-fly of the finished-job3 directory
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

# This script is named: data-list.sh

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/$USER/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

cd /scratch/$USER/restored-from-MARS
/usr/local/bin/htar -xf finished-job3.tar

date
</pre>

* sample '''analysis''' (depends on previous data-restore.sh execution)
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

MARS

2011-05-26T20:25:40Z

Knecht: /* Using the batch queue */

=== '''Massive Archive and Restore System''' ===

(Pilot usage phase to start in May/2011 with a select group of users. Deployment and configuration are still a work in progress)

The '''MARS''' deployment at SciNet is an effort to offer a more efficient way to offoad/archive data from the most active file systems (scratch and project) than our current TSM-HSM solution, still without having to deal directly with the tape library or "tape commands"

The system is a combination of the underlaying hardware infrastructure, 3 software components, ''HPSS'', ''HSI'' and ''HTAR'', plus some environment customization.

* '''HPSS''': the main component, best described as a very scalable "blackbox" running in the background to support the Archive and Restore operations. [http://www.hpss-collaboration.org/index.shtml High Performance Storage System - HPSS] is the result of over a decade of collaboration among five Department of Energy laboratories and IBM, with significant contributions by universities and other laboratories worldwide. For now the best way for SciNet users to [https://support.scinet.utoronto.ca/wiki/index.php/HPSS_compared_to_HSM-TSM understand HPSS] may be to compare it with our existing HSM-TSM implementation.

* '''HSI''': it may be best understood as a supercharged ftp interface, specially designed by [http://www.mgleicher.us/GEL/hsi/ Gleicher Enterprises] to act as a front-end for HPSS, gathering some of the best features you would encounter on a shell, rsync and GridFTP (and a few more). It enables users to transfer whole directory trees from /project and /scratch, therefore freeing up space. HSI is most suitable when those directory trees do not contain too many small files to start with, or when you already have a series of tarballs.

* '''HTAR''': similarly, htar is sort of a "super-tar" application, also specially designed by [http://www.mgleicher.us/GEL/htar/ Gleicher Enterprises] to interact with HPSS, allowing users to build and automatically transfer tarballs to HPSS on the fly. HTAR is most suitable to aggregate whole directory trees. When HTAR creates the TAR file, it also builds an index file, with a ".idx" suffix added, which is stored in the same directory as the TAR file.

=== '''General guide lines''' ===
* IN/OUT transfers to HPSS using HSI is bound to maximum of about '''4 files/second'''. Therefore do not attempt to transfer directories with too many small files inside. Not only it will take "forever", it will induce a lot of ware and tear on the library's robot mechanism as well as the tapes themselves in case of recalls. Instead use HTAR, so the files are aggregated while being sent to HPSS
* The maximum size that an individual file can have inside an HTAR is '''68GB'''. Please be sure to identify and "fish out" those files that are larger than 68GB from the directories and transfer them with HSI
* The maximum size of a tar file that HPSS will take is '''1TB'''. Please do not generate tarballs that large.
* The maximum number of files in a htar is '''1 million'''. Please, break-up your htar segments as required.
* These guide lines may be strict, but for as long as they are followed the system will perform reasonably well

=== '''Performance considerations''' ===
* Files are kept on disk-cache for as long as possible, so as to avoid tape operations during recalls.
* Average transfer rates with '''HSI'''
No small files, average > 1MB/file:
* write: 100-130MB/s
* read: 450-600MB/s (IF no recall from tapes required)

* '''NOTE: do not use HSI with small files (< 1MB/file). It would take over 1 week to transfer 1 TB. If we find that you are abusing the system we'll suspend your privileges'''

* Average transfer rates with '''HTAR'''
Average files size > 1MB
* write: 120MB/s
* read: 480MB/s (IF no recall from tapes required)

Not too many small files, average > 100KB/file:
* write: 64MB/s
* read: 170MB/s (IF no recall from tapes required)

* Average transfer rates from '''tapes''', if stage is required (add to the above estimates)
* read: 80MB/s per tape drive.
* maximum of 4 drives may be used per hsi/htar session

=== '''Quick Reference''' ===

* To use HSI or HTAR and access HPSS please login to the '''gpc-archive01''' node.
* AUTHENTICATION: done automatically
* To browse the contents of your HPSS archive just type '''hsi''' on a shell to get the hsi prompt. Then use simple commands such as '''ls''', '''pwd''', '''cd''' to navigate your way around. You may also use [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''help'''] from the hsi prompt.
* Files are organized inside HPSS in the same fashion as in /project. Users in the same group have read permissions to each other's archives.
<pre>
[HSI]/archive/<group>/<user>
</pre>
* There is also provision to have files migrated to a "mirrored set of tapes" (the 2nd set may be kept off-site). You'll have request and justify the need. The default is a "single tape set".
<pre>
[HSI]/archive-dual-copy/<group>/<user>
</pre>
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

=== '''Using HSI''' ===

* Interactively put a subdirectory subdirb and all its contents recursively. You may use '-u' option to resume a previously disrupted session (as rsync would do).
<pre>
hsi <RETURN>
[HSI] prompt
[HSI] mput -R -u subdirb
</pre>

* Interactively descend into the "Source" directory and move all files which end in ".h" into a sibling directory (ie, a directory at the same level in the tree as "Source") named "Include":
<pre>
hsi <RETURN>
[HSI] cd Source
[HSI] mv *.h ../Include
</pre>

* Delete all files beginning with "m" and ending with 9101 (note that this is an interactive request, not a one-liner request, so the wildcard path does not need quotes to preserve it):
<pre>
hsi <RETURN>
[HSI] delete m*9101
</pre>

* Interactively delete all files beginning with H and ending with a digit, and ask for verification before deleting each such file.
<pre>
hsi <RETURN>
[HSI] mdel H*[0-9]
</pre>

* Save your local files that begin with the letter "c" (let the UN*X shell resolve the wild-card path pattern in terms of your local files by not enclosing it in quotes:
<pre>
hsi put c*
</pre>

* Get all files in the subdirectory subdira which begin with the letters "b" or "c" (surrounding the wildcard path in single quotes prevents shells on UNIX systems from processing the wild card pattern):
<pre>
hsi get ’subdira/[bc]*’
</pre>

* Save a "tar file" of C source programs and header files:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* Using the "mirrored set of tapes" provision (you'll need authorization and a directory placement in /archive-dual-copy)
<pre>
hsi put source.tar : /archive-dual-copy/<group>/<user>/source.tar
OR
hsi <RETURN>
[HSI] cd /archive-dual-copy/<group>/<user>
[HSI] put source.tar
</pre>
* For more details please check the [http://www.mgleicher.us/GEL/hsi/ HSI Introduction] or the [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page] online

=== '''Using HTAR''' ===

* To write the file1 and file2 files to a new archive called "files.tar" in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write the file1 and file2 files to a new archive called "files.tar" on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the project1/src directory in the Archive file called proj1.tar, and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the out.tar archive file within the HPSS home directory, enter:
<pre>
htar -vtf out.tar
</pre>

* Using the "mirrored set of tapes" provision (you'll need authorization and a directory placement in /archive-dual-copy)
<pre>
htar -cf /archive-dual-copy/<group>/<user>/files.tar file1 file2
</pre>

For more details please check the [http://www.mgleicher.us/GEL/htar/ HTAR - Introduction] or the [http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page] online

=== '''Using the batch queue''' ===
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
qsub -I -q archive
showq -w class=archive
</pre>

* sample '''data offload'''
<pre>
#!/bin/bash

#PBS -q archive
#PBS -N offload
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

/usr/local/bin/hsi -v <<EOF
mkdir put-away-and-forget
cd put-away-and-forget
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
EOF

date
</pre>
* sample '''data list'''
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>

* sample '''data restore'''
<pre>
#!/bin/bash

# This script is named: data-restore.sh

#PBS -q archive
#PBS -N restore
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/knecht/restored-from-MARS

/usr/local/bin/hsi -v << EOF
get /scratch/knecht/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
get /scratch/knecht/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
EOF

date
</pre>

* sample analysis
<pre>
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
</pre>

MARS

2011-05-26T18:53:43Z

Knecht: /* Using the batch queue */

=== '''Massive Archive and Restore System''' ===

(Pilot usage phase to start in May/2011 with a select group of users. Deployment and configuration are still a work in progress)

The '''MARS''' deployment at SciNet is an effort to offer a more efficient way to offoad/archive data from the most active file systems (scratch and project) than our current TSM-HSM solution, still without having to deal directly with the tape library or "tape commands"

The system is a combination of the underlaying hardware infrastructure, 3 software components, ''HPSS'', ''HSI'' and ''HTAR'', plus some environment customization.

* '''HPSS''': the main component, best described as a very scalable "blackbox" running in the background to support the Archive and Restore operations. [http://www.hpss-collaboration.org/index.shtml High Performance Storage System - HPSS] is the result of over a decade of collaboration among five Department of Energy laboratories and IBM, with significant contributions by universities and other laboratories worldwide. For now the best way for SciNet users to [https://support.scinet.utoronto.ca/wiki/index.php/HPSS_compared_to_HSM-TSM understand HPSS] may be to compare it with our existing HSM-TSM implementation.

* '''HSI''': it may be best understood as a supercharged ftp interface, specially designed by [http://www.mgleicher.us/GEL/hsi/ Gleicher Enterprises] to act as a front-end for HPSS, gathering some of the best features you would encounter on a shell, rsync and GridFTP (and a few more). It enables users to transfer whole directory trees from /project and /scratch, therefore freeing up space. HSI is most suitable when those directory trees do not contain too many small files to start with, or when you already have a series of tarballs.

* '''HTAR''': similarly, htar is sort of a "super-tar" application, also specially designed by [http://www.mgleicher.us/GEL/htar/ Gleicher Enterprises] to interact with HPSS, allowing users to build and automatically transfer tarballs to HPSS on the fly. HTAR is most suitable to aggregate whole directory trees. When HTAR creates the TAR file, it also builds an index file, with a ".idx" suffix added, which is stored in the same directory as the TAR file.

=== '''General guide lines''' ===
* IN/OUT transfers to HPSS using HSI is bound to maximum of about '''4 files/second'''. Therefore do not attempt to transfer directories with too many small files inside. Not only it will take "forever", it will induce a lot of ware and tear on the library's robot mechanism as well as the tapes themselves in case of recalls. Instead use HTAR, so the files are aggregated while being sent to HPSS
* The maximum size that an individual file can have inside an HTAR is '''68GB'''. Please be sure to identify and "fish out" those files that are larger than 68GB from the directories and transfer them with HSI
* The maximum size of a tar file that HPSS will take is '''1TB'''. Please do not generate tarballs that large.
* The maximum number of files in a htar is '''1 million'''. Please, break-up your htar segments as required.
* These guide lines may be strict, but for as long as they are followed the system will perform reasonably well

=== '''Performance considerations''' ===
* Files are kept on disk-cache for as long as possible, so as to avoid tape operations during recalls.
* Average transfer rates with '''HSI'''
No small files, average > 1MB/file:
* write: 100-130MB/s
* read: 450-600MB/s (IF no recall from tapes required)

* '''NOTE: do not use HSI with small files (< 1MB/file). It would take over 1 week to transfer 1 TB. If we find that you are abusing the system we'll suspend your privileges'''

* Average transfer rates with '''HTAR'''
Average files size > 1MB
* write: 120MB/s
* read: 480MB/s (IF no recall from tapes required)

Not too many small files, average > 100KB/file:
* write: 64MB/s
* read: 170MB/s (IF no recall from tapes required)

* Average transfer rates from '''tapes''', if stage is required (add to the above estimates)
* read: 80MB/s per tape drive.
* maximum of 4 drives may be used per hsi/htar session

=== '''Quick Reference''' ===

* To use HSI or HTAR and access HPSS please login to the '''gpc-archive01''' node.
* AUTHENTICATION: done automatically
* To browse the contents of your HPSS archive just type '''hsi''' on a shell to get the hsi prompt. Then use simple commands such as '''ls''', '''pwd''', '''cd''' to navigate your way around. You may also use [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''help'''] from the hsi prompt.
* Files are organized inside HPSS in the same fashion as in /project. Users in the same group have read permissions to each other's archives.
<pre>
[HSI]/archive/<group>/<user>
</pre>
* There is also provision to have files migrated to a "mirrored set of tapes" (the 2nd set may be kept off-site). You'll have request and justify the need. The default is a "single tape set".
<pre>
[HSI]/archive-dual-copy/<group>/<user>
</pre>
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

=== '''Using HSI''' ===

* Interactively put a subdirectory subdirb and all its contents recursively. You may use '-u' option to resume a previously disrupted session (as rsync would do).
<pre>
hsi <RETURN>
[HSI] prompt
[HSI] mput -R -u subdirb
</pre>

* Interactively descend into the "Source" directory and move all files which end in ".h" into a sibling directory (ie, a directory at the same level in the tree as "Source") named "Include":
<pre>
hsi <RETURN>
[HSI] cd Source
[HSI] mv *.h ../Include
</pre>

* Delete all files beginning with "m" and ending with 9101 (note that this is an interactive request, not a one-liner request, so the wildcard path does not need quotes to preserve it):
<pre>
hsi <RETURN>
[HSI] delete m*9101
</pre>

* Interactively delete all files beginning with H and ending with a digit, and ask for verification before deleting each such file.
<pre>
hsi <RETURN>
[HSI] mdel H*[0-9]
</pre>

* Save your local files that begin with the letter "c" (let the UN*X shell resolve the wild-card path pattern in terms of your local files by not enclosing it in quotes:
<pre>
hsi put c*
</pre>

* Get all files in the subdirectory subdira which begin with the letters "b" or "c" (surrounding the wildcard path in single quotes prevents shells on UNIX systems from processing the wild card pattern):
<pre>
hsi get ’subdira/[bc]*’
</pre>

* Save a "tar file" of C source programs and header files:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* Using the "mirrored set of tapes" provision (you'll need authorization and a directory placement in /archive-dual-copy)
<pre>
hsi put source.tar : /archive-dual-copy/<group>/<user>/source.tar
OR
hsi <RETURN>
[HSI] cd /archive-dual-copy/<group>/<user>
[HSI] put source.tar
</pre>
* For more details please check the [http://www.mgleicher.us/GEL/hsi/ HSI Introduction] or the [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page] online

=== '''Using HTAR''' ===

* To write the file1 and file2 files to a new archive called "files.tar" in the default HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
OR
htar -cf /archive/<group>/<user>/files.tar file1 file2
</pre>

* To write the file1 and file2 files to a new archive called "files.tar" on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the project1/src directory in the Archive file called proj1.tar, and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the out.tar archive file within the HPSS home directory, enter:
<pre>
htar -vtf out.tar
</pre>

* Using the "mirrored set of tapes" provision (you'll need authorization and a directory placement in /archive-dual-copy)
<pre>
htar -cf /archive-dual-copy/<group>/<user>/files.tar file1 file2
</pre>

For more details please check the [http://www.mgleicher.us/GEL/htar/ HTAR - Introduction] or the [http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page] online

=== '''Using the batch queue''' ===
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
qsub -I -q archive
showq -w class=archive
</pre>
* sample data import
<pre>
#!/bin/bash

#PBS -q archive
#PBS -N ingest
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

/usr/local/bin/hsi -v <<EOF
mkdir HItest
cd HItest
put /scratch/knecht/workarea/SwapTests/A/DBRelease-13.2.1.tar.gz : DBRelease-13.2.1.tar.gz
put /scratch/knecht/workarea/SwapTests/A/RDO.215791._003065.pool.root.1 : RDO.215791._003065.pool.root.1
EOF

date
</pre>
* sample data list
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>
* sample stage-in
<pre>
#!/bin/bash

#PBS -q archive
#PBS -N stage
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date

mkdir -p /scratch/knecht/workarea/SwapTests/stagetest

/usr/local/bin/hsi -v << EOF
get /scratch/knecht/workarea/SwapTests/stagetest/RDO.215791._003065.pool.root.1 : HItest/RDO.215791._003065.pool.root.1
get /scratch/knecht/workarea/SwapTests/stagetest/DBRelease-13.2.1.tar.gz : HItest/DBRelease-13.2.1.tar.gz :
EOF

date
</pre>
* sample analysis
<pre>
gpc04#> qsub $(qsub stage.job | awk -F '.' '{print "-W depend=afterok:"$1}') hiReco.sh
</pre>

MARS

2011-05-26T16:36:51Z

Knecht: /* Using the batch queue */

=== '''Massive Archive and Restore System''' ===

(Pilot usage phase to start in May/2011 with a select group of users. Deployment and configuration are still a work in progress)

The '''MARS''' deployment at SciNet is an effort to offer a more efficient way to offoad/archive data from the most active file systems (scratch and project) than our current TSM-HSM solution, still without having to deal directly with the tape library or "tape commands"

The system is a combination of the underlaying hardware infrastructure, 3 software components, ''HPSS'', ''HSI'' and ''HTAR'', plus some environment customization.

* '''HPSS''': the main component, best described as a very scalable "blackbox" running in the background to support the Archive and Restore operations. [http://www.hpss-collaboration.org/index.shtml High Performance Storage System - HPSS] is the result of over a decade of collaboration among five Department of Energy laboratories and IBM, with significant contributions by universities and other laboratories worldwide. For now the best way for SciNet users to [https://support.scinet.utoronto.ca/wiki/index.php/HPSS_compared_to_HSM-TSM understand HPSS] may be to compare it with our existing HSM-TSM implementation.

* '''HSI''': it may be best understood as a supercharged ftp interface, specially designed by [http://www.mgleicher.us/GEL/hsi/ Gleicher Enterprises] to act as a front-end for HPSS, gathering some of the best features you would encounter on a shell, rsync and GridFTP (and a few more). It enables users to transfer whole directory trees from /project and /scratch, therefore freeing up space. HSI is most suitable when those directory trees do not contain too many small files to start with, or when you already have a series of tarballs.

* '''HTAR''': similarly, htar is sort of a "super-tar" application, also specially designed by [http://www.mgleicher.us/GEL/htar/ Gleicher Enterprises] to interact with HPSS, allowing users to build and automatically transfer tarballs to HPSS on the fly. HTAR is most suitable to aggregate whole directory trees. When HTAR creates the TAR file, it also builds an index file, with a ".idx" suffix added, which is stored in the same directory as the TAR file.

=== '''General guide lines''' ===
* IN/OUT transfers to HPSS using HSI is bound to maximum of about '''4 files/second'''. Therefore do not attempt to transfer directories with too many small files inside. Not only it will take "forever", it will induce a lot of ware and tear on the library's robot mechanism as well as the tapes themselves in case of recalls. Instead use HTAR, so the files are aggregated while being sent to HPSS
* The maximum size that an individual file can have inside an HTAR is '''68GB'''. Please be sure to identify and "fish out" those files that are larger than 68GB from the directories and transfer them with HSI
* The maximum size of a tar file that HPSS will take is '''1TB'''. Please do not generate tarballs that large.
* The maximum number of files in a htar is '''1 million'''. Please, break-up your htar segments as required.
* These guide lines may be strict, but for as long as they are followed the system will perform reasonably well

=== '''Performance considerations''' ===
* Files are kept on disk-cache for as long as possible, so as to avoid tape operations during recalls.
* Average transfer rates with '''HSI'''
No small files, average > 1MB/file:
* write: 100-130MB/s
* read: 450-600MB/s (IF no recall from tapes required)

* '''NOTE: do not use HSI with small files (< 1MB/file). It would take over 1 week to transfer 1 TB. If we find that you are abusing the system we'll suspend your privileges'''

* Average transfer rates with '''HTAR'''
Average files size > 1MB
* write: 120MB/s
* read: 480MB/s (IF no recall from tapes required)

Not too many small files, average > 100KB/file:
* write: 64MB/s
* read: 170MB/s (IF no recall from tapes required)

* Average transfer rates from '''tapes''', if stage is required (add to the above estimates)
* read: 80MB/s per tape drive.
* maximum of 4 drives may be used per hsi/htar session

=== '''Quick Reference''' ===

* To use HSI or HTAR and access HPSS please login to the '''gpc-archive01''' node.
* AUTHENTICATION: done automatically
* To browse the contents of your HPSS archive just type '''hsi''' on a shell to get the hsi prompt. Then use simple commands such as '''ls''', '''pwd''', '''cd''' to navigate your way around. You may also use [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''help'''] from the hsi prompt.
* Files are organized inside HPSS in the same fashion as in /project. Users in the same group have read permissions to each other's archives.
<pre>
[HSI]/archive/<group>/<user>
</pre>
* There is also provision to have files migrated to a "mirrored set of tapes" Class of Service (COS). You'll have request and justify the need. The default is a "single tape" COS.
<pre>
[HSI]/archive-dual-copy/<group>/<user>
</pre>
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

=== '''Using HSI''' ===

* Interactively put a subdirectory subdirb and all its contents recursively. You may use '-u' option to resume a previously disrupted session (as rsync would do).
<pre>
hsi <RETURN>
[HSI] prompt
[HSI] mput -R -u subdirb
</pre>

* Interactively descend into the "Source" directory and move all files which end in ".h" into a sibling directory (ie, a directory at the same level in the tree as "Source") named "Include":
<pre>
hsi <RETURN>
[HSI] cd Source
[HSI] mv *.h ../Include
</pre>

* Delete all files beginning with "m" and ending with 9101 (note that this is an interactive request, not a one-liner request, so the wildcard path does not need quotes to preserve it):
<pre>
hsi <RETURN>
[HSI] delete m*9101
</pre>

* Interactively delete all files beginning with H and ending with a digit, and ask for verification before deleting each such file.
<pre>
hsi <RETURN>
[HSI] mdel H*[0-9]
</pre>

* Save your local files that begin with the letter "c" (let the UN*X shell resolve the wild-card path pattern in terms of your local files by not enclosing it in quotes:
<pre>
hsi put c*
</pre>

* Get all files in the subdirectory subdira which begin with the letters "b" or "c" (surrounding the wildcard path in single quotes prevents shells on UNIX systems from processing the wild card pattern):
<pre>
hsi get ’subdira/[bc]*’
</pre>

* Save a "tar file" of C source programs and header files:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* Using the copy to mirror tapes provision (you'll need authorization and a directory placement in /archive-dual-copy)
<pre>
hsi put source.tar : /archive-dual-copy/<group>/<user>/source.tar
OR
hsi <RETURN>
[HSI] cd /archive-dual-copy/<group>/<user>
[HSI] put source.tar
</pre>
* For more details please check the [http://www.mgleicher.us/GEL/hsi/ HSI Introduction] or the [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page] online

=== '''Using HTAR''' ===

* To write the file1 and file2 files to a new archive called "files.tar" in the current HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
</pre>

* To write the file1 and file2 files to a new archive called "files.tar" on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter:
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the project1/src directory in the Archive file called proj1.tar, and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the out.tar archive file within the HPSS home directory, enter:
<pre>
htar -vtf out.tar
</pre>
For more details please check the [http://www.mgleicher.us/GEL/htar/ HTAR - Introduction] or the [http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page] online

=== '''Using the batch queue''' ===
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
qsub -I -q archive
</pre>
* sample data import
<pre>
#!/bin/bash

#PBS -q archive
#PBS -N ingest
#PBS -j oe
#PBS -o logs/

# module load dcap
# module load

hostname
date

/usr/local/bin/hsi -v <<EOF
mkdir HItest
cd HItest
put /scratch/knecht/workarea/SwapTests/A/DBRelease-13.2.1.tar.gz : DBRelease-13.2.1.tar.gz
put /scratch/knecht/workarea/SwapTests/A/RDO.215791._003065.pool.root.1 : RDO.215791._003065.pool.root.1
EOF

date
</pre>
* sample data list
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>
* sample stage-in
<pre>
#!/bin/bash

#PBS -q archive
#PBS -N stage
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

# module load dcap
# module load

date

mkdir -p /scratch/knecht/workarea/SwapTests/stagetest

/usr/local/bin/hsi -v << EOF
get /scratch/knecht/workarea/SwapTests/stagetest/RDO.215791._003065.pool.root.1 : HItest/RDO.215791._003065.pool.root.1
get /scratch/knecht/workarea/SwapTests/stagetest/DBRelease-13.2.1.tar.gz : HItest/DBRelease-13.2.1.tar.gz :
EOF

date
</pre>
* sample analyis
<pre>
gpc04#> qsub $(qsub stage.job | awk -F '.' '{print "-W depend=afterok:"$1}') hiReco.sh
</pre>

MARS

2011-05-26T16:30:41Z

Knecht: /* Using the batch queue */

=== '''Massive Archive and Restore System''' ===

(Pilot usage phase to start in May/2011 with a select group of users. Deployment and configuration are still a work in progress)

The '''MARS''' deployment at SciNet is an effort to offer a more efficient way to offoad/archive data from the most active file systems (scratch and project) than our current TSM-HSM solution, still without having to deal directly with the tape library or "tape commands"

The system is a combination of the underlaying hardware infrastructure, 3 software components, ''HPSS'', ''HSI'' and ''HTAR'', plus some environment customization.

* '''HPSS''': the main component, best described as a very scalable "blackbox" running in the background to support the Archive and Restore operations. [http://www.hpss-collaboration.org/index.shtml High Performance Storage System - HPSS] is the result of over a decade of collaboration among five Department of Energy laboratories and IBM, with significant contributions by universities and other laboratories worldwide. For now the best way for SciNet users to [https://support.scinet.utoronto.ca/wiki/index.php/HPSS_compared_to_HSM-TSM understand HPSS] may be to compare it with our existing HSM-TSM implementation.

* '''HSI''': it may be best understood as a supercharged ftp interface, specially designed by [http://www.mgleicher.us/GEL/hsi/ Gleicher Enterprises] to act as a front-end for HPSS, gathering some of the best features you would encounter on a shell, rsync and GridFTP (and a few more). It enables users to transfer whole directory trees from /project and /scratch, therefore freeing up space. HSI is most suitable when those directory trees do not contain too many small files to start with, or when you already have a series of tarballs.

* '''HTAR''': similarly, htar is sort of a "super-tar" application, also specially designed by [http://www.mgleicher.us/GEL/htar/ Gleicher Enterprises] to interact with HPSS, allowing users to build and automatically transfer tarballs to HPSS on the fly. HTAR is most suitable to aggregate whole directory trees. When HTAR creates the TAR file, it also builds an index file, with a ".idx" suffix added, which is stored in the same directory as the TAR file.

=== '''General guide lines''' ===
* IN/OUT transfers to HPSS using HSI is bound to maximum of about '''4 files/second'''. Therefore do not attempt to transfer directories with too many small files inside. Not only it will take "forever", it will induce a lot of ware and tear on the library's robot mechanism as well as the tapes themselves in case of recalls. Instead use HTAR, so the files are aggregated while being sent to HPSS
* The maximum size that an individual file can have inside an HTAR is '''68GB'''. Please be sure to identify and "fish out" those files that are larger than 68GB from the directories and transfer them with HSI
* The maximum size of a tar file that HPSS will take is '''1TB'''. Please do not generate tarballs that large.
* The maximum number of files in a htar is '''1 million'''. Please, break-up your htar segments as required.
* These guide lines may be strict, but for as long as they are followed the system will perform reasonably well

=== '''Performance considerations''' ===
* Files are kept on disk-cache for as long as possible, so as to avoid tape operations during recalls.
* Average transfer rates with '''HSI'''
No small files, average > 1MB/file:
* write: 100-130MB/s
* read: 450-600MB/s (IF no recall from tapes required)

* '''NOTE: do not use HSI with small files (< 1MB/file). It would take over 1 week to transfer 1 TB. If we find that you are abusing the system we'll suspend your privileges'''

* Average transfer rates with '''HTAR'''
Average files size > 1MB
* write: 120MB/s
* read: 480MB/s (IF no recall from tapes required)

Not too many small files, average > 100KB/file:
* write: 64MB/s
* read: 170MB/s (IF no recall from tapes required)

* Average transfer rates from '''tapes''', if stage is required (add to the above estimates)
* read: 80MB/s per tape drive.
* maximum of 4 drives may be used per hsi/htar session

=== '''Quick Reference''' ===

* To use HSI or HTAR and access HPSS please login to the '''gpc-archive01''' node.
* AUTHENTICATION: done automatically
* To browse the contents of your HPSS archive just type '''hsi''' on a shell to get the hsi prompt. Then use simple commands such as '''ls''', '''pwd''', '''cd''' to navigate your way around. You may also use [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''help'''] from the hsi prompt.
* Files are organized inside HPSS in the same fashion as in /project. Users in the same group have read permissions to each other's archives.
<pre>
[HSI]/archive/<group>/<user>
</pre>
* There is also provision to have files migrated to a "mirrored set of tapes" Class of Service (COS). You'll have request and justify the need. The default is a "single tape" COS.
<pre>
[HSI]/archive-dual-copy/<group>/<user>
</pre>
* '''Pilot users:''' DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project

=== '''Using HSI''' ===

* Interactively put a subdirectory subdirb and all its contents recursively. You may use '-u' option to resume a previously disrupted session (as rsync would do).
<pre>
hsi <RETURN>
[HSI] prompt
[HSI] mput -R -u subdirb
</pre>

* Interactively descend into the "Source" directory and move all files which end in ".h" into a sibling directory (ie, a directory at the same level in the tree as "Source") named "Include":
<pre>
hsi <RETURN>
[HSI] cd Source
[HSI] mv *.h ../Include
</pre>

* Delete all files beginning with "m" and ending with 9101 (note that this is an interactive request, not a one-liner request, so the wildcard path does not need quotes to preserve it):
<pre>
hsi <RETURN>
[HSI] delete m*9101
</pre>

* Interactively delete all files beginning with H and ending with a digit, and ask for verification before deleting each such file.
<pre>
hsi <RETURN>
[HSI] mdel H*[0-9]
</pre>

* Save your local files that begin with the letter "c" (let the UN*X shell resolve the wild-card path pattern in terms of your local files by not enclosing it in quotes:
<pre>
hsi put c*
</pre>

* Get all files in the subdirectory subdira which begin with the letters "b" or "c" (surrounding the wildcard path in single quotes prevents shells on UNIX systems from processing the wild card pattern):
<pre>
hsi get ’subdira/[bc]*’
</pre>

* Save a "tar file" of C source programs and header files:
<pre>
tar cf - *.[ch] | hsi put - : source.tar
</pre>
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)

* Restore the tar file source kept above and extract all files:
<pre>
hsi get - : source.tar | tar xf -
</pre>

* The commands below are equivalent:
<pre>
hsi put source.tar
hsi put source.tar : /archive/<group>/<user>/source.tar
</pre>

* Using the copy to mirror tapes provision (you'll need authorization and a directory placement)
<pre>
hsi put source.tar : /archive-dual-copy/<group>/<user>/source.tar
</pre>
* For more details please check the [http://www.mgleicher.us/GEL/hsi/ HSI Introduction] or the [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page] online

=== '''Using HTAR''' ===

* To write the file1 and file2 files to a new archive called "files.tar" in the current HPSS home directory, enter:
<pre>
htar -cf files.tar file1 file2
</pre>

* To write the file1 and file2 files to a new archive called "files.tar" on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter:
<pre>
htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
</pre>

* To extract all files from the project1/src directory in the Archive file called proj1.tar, and use the time of extraction as the modification time, enter:
<pre>
htar -xm -f proj1.tar project1/src
</pre>

* To display the names of the files in the out.tar archive file within the HPSS home directory, enter:
<pre>
htar -vtf out.tar
</pre>
For more details please check the [http://www.mgleicher.us/GEL/htar/ HTAR - Introduction] or the [http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page] online

=== '''Using the batch queue''' ===
* gpc-archive01 is part of the gpc queuing system under torque/moab
* Currently it is setup to share the node with up to 12 jobs at one time
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
<pre>
qsub -I -q archive
</pre>
* sample data import
<pre>
#!/bin/bash

#PBS -q archive
#PBS -N ingest
#PBS -j oe
#PBS -o logs/

# module load dcap
# module load

hostname
date

/usr/local/bin/hsi -v <<EOF
mkdir HItest
cd HItest
put /scratch/knecht/workarea/SwapTests/A/DBRelease-13.2.1.tar.gz : DBRelease-13.2.1.tar.gz
put /scratch/knecht/workarea/SwapTests/A/RDO.215791._003065.pool.root.1 : RDO.215791._003065.pool.root.1
EOF

date
</pre>
* sample data list
- Very painful without interactive browsing
-Tentative solution: dump all user files to log file and use that as file index
<pre>
#!/bin/bash

#PBS -q archive
#PBS -N hpss_dump
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

date
echo ===========
echo
/usr/local/bin/hsi -v <<EOF
ls -lUR
EOF
echo
echo ===========
date
</pre>
* sample stage-in
<pre>
#!/bin/bash

#PBS -q archive
#PBS -N stage
#PBS -j oe
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID

# module load dcap
# module load

date

mkdir -p /scratch/knecht/workarea/SwapTests/stagetest

/usr/local/bin/hsi -v << EOF
get HItest/RDO.215791._003065.pool.root.1 : /scratch/knecht/workarea/SwapTests/stagetest/RDO.215791._003065.pool.root.1
get HItest/DBRelease-13.2.1.tar.gz : /scratch/knecht/workarea/SwapTests/stagetest/DBRelease-13.2.1.tar.gz
EOF

date
</pre>
* sample analyis
<pre>
gpc04#> qsub $(qsub stage.job | awk -F '.' '{print "-W depend=afterok:"$1}') hiReco.sh
</pre>

Software and Libraries

2010-06-10T21:40:58Z

Knecht: /* GPC Software */

All the software listed on this page is accessed using a [[GPC_Quickstart#Environment_Variables | modules]] system. This means that much of the software is not
accessible by default but has to be loaded using the module command. The
reason is that
* it allows us to easily keep multiple versions of software for different users on the system;
* it allows users to easily switch between versions.
The module system works similarly on the GPC and the TCS.

To use particular software, just load the module (the last column in the following table) as follows.
<pre>
module load [module-name]
</pre>
It is recommended to load frequently used modules in the file .bashrc in your home directory.

Modules that load libraries, define environment variables pointing to the location of library files and
include files for use Makefiles. These environment variables follow the naming convention
<pre>
SCINET_[module-name]_{LIB,INC,BASE}
</pre>
So to compile and link the library, you will have to add <tt>-I$SCINET_[module-name]_INC</tt> and <tt>-L$SCINET_[module-name]_LIB</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

For a full list of available software packages that may be accessed with the modules system, use the command
<pre>
module avail
</pre>

To see the list of available versions of a specific software package, use the command
<pre>
module avail [module-name]
</pre>

For a list of the currently loaded modules in your shell, use the command
<pre>
module list
</pre>

For a description of a particular module, use the command
<pre>
module help [module-name]
</pre>

== Other software and libraries ==

If you want to use a piece of software or a library that is not on the list, you can in principle install it yourself in you <tt>/home</tt> directory.
Note however that building libraries and software from source often uses a lot of files. To avoid running out of disk space, building software is therefore best done from the <tt>/scratch</tt>, from which
you can copy/install only the libraries, header files and binaries to your <tt>/home</tt> directory.

If you suspect that a particular piece of software or a library would be of use to other users of SciNet as well, contact us, and we will consider adding it to the system.

== GPC Software ==

{| border="1" cellpadding="10" cellspacing="0"
! Software
! Version
! Comments
! Command/Library
! Module Name
|- style='background: lightgray'
|Intel Compiler
| 11.1
|
| <tt>icpc,icc,ifort, includes MKL library</tt>
| <tt>intel</tt>
|- style='background: lightgray'
| GCC Compiler
| 4.4.0
|
| <tt>gcc,g++,gfortran</tt>
| <tt>gcc</tt>
|- style='background: lightgray'
| IntelMPI
| 3.2.2
|
| <tt>mpicc,mpiCC,mpif77,mpif90</tt>
| <tt>intelmpi</tt>
|- style='background: lightgray'
| OpenMPI
| 1.4.1
|
| <tt>mpicc,mpiCC,mpif77,mpif90</tt>
| <tt>openmpi</tt>
|- style='background: white'
| Emacs
| 23.1
| New version of popular text editor
| <tt>emacs</tt>
| <tt>emacs</tt>
|- style='background: white'
| Git
| 1.6.3
| Revision control system
| <tt>git,gitk</tt>
| <tt>git</tt>
|- style='background: white'
| Valgrind
| 3.4.1
| Memory checking utility
| <tt>valgrind,cachegrind</tt>
| <tt>valgrind</tt>
|- style='background: lightgray'
| grace
| 5.22.1
| Plotting utility
| <tt>xmgrace</tt>
| <tt>graphics</tt>
|- style='background: lightgray'
| Gnuplot
| 4.2.6
| Plotting utility
| <tt>gnuplot</tt>
| <tt>graphics</tt>
|- style='background: lightgray'
| VMD
| 1.8.6
| Visualization and analysis utility
| <tt>vmd</tt>
| <tt>vmd</tt>
|- style='background: lightgray'
| ferret
| 6.4
| Plotting utility
| <tt>ferret</tt>
| <tt>ferret</tt>
|- style='background: lightgray'
| ncl/ncarg
| 5.1.1
| NCARG graphics and ncl utilities
| <tt>ncl</tt>
| <tt>ncl</tt>
|- style='background: lightgray'
| ROOT
| 5.26.00
| ROOT Analysis Framework from CERN
| <tt>root</tt>
| <tt>ROOT</tt>
|- style='background: lightgray'
| ParaView
| 3.6.1
| Scientific visualization, server only
| <tt>pvserver,pvbatch,pvpython</tt>
| <tt>visualization</tt>
|- style='background: white'
| NetCDF
| 4.0.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf</tt>
|- style='background: white'
| ncview
| 1.93g
| Visualization for NetCDF files
| <tt>ncview</tt>
| <tt>graphics/ncview</tt>
|- style='background: white'
| NCO
| 3.9.9
| NCO utilities to manipulate netCDF files
| <tt>ncap, ncap2, ncatted, etc.</tt>
| <tt>nco</tt>
|- style='background: white'
| udunits
| 2.1.11
| unit conversion utilities
| <tt>libudunits2</tt>
| <tt>udunits</tt>
|- style='background: white'
| HDF4
| 4.2r4
| Scientific data storage and retrieval
| <tt>h4fc,hdiff,...,libdf,libsz</tt>
| <tt>hdf4</tt>
|- style='background: white'
| HDF5
| 1.8.4-v18
| Scientific data storage and retrieval, parallel I/O
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5</tt>
|- style='background: lightgray'
| [[gamess|GAMESS (US)]]
| January 12, 2009 R3
| General Atomic and Molecular Electronic Structure System
| <tt>rungms</tt>
| <tt>gamess</tt>
|- style='background: lightgray'
| [http://blast.ncbi.nlm.nih.gov BLAST]
| 2.2.23+
| Basic Local Alignment Search Tool
| <tt>blastn,blastp,blastx,psiblast,tblastn...</tt>
| <tt>blast</tt>
|- style='background: lightgray'
| [[amber|AMBER 10]]
| Amber 10 + Amber Tools 1.3
| Amber Molecular Dynamics Package
| <tt>sander, sander.MPI</tt>
| <tt>amber10</tt>
|- style='background: lightgray'
| [http://www.gdal.org/ GDAL]
| 1.7.1
| Geospatial Data Abstraction Library
| <tt>gdal_contour,gdal_rasterize,gdal_grid, libgdal</tt>
| <tt>gdal</tt>
|- style='background: lightgray'
| [http://www.mcs.anl.gov/petsc/petsc-as/ PETSc ]
|
| Portable, Extensible Tolkit for Scientific Computation (PETSc)
| <tt>libpetsc, etc.. </tt>
| <tt>petsc</tt>
|- style='background: lightgray'
| Matlab/IDL/Commercial software
|
| Little to none. See [[FAQ#How_can_I_run_Matlab_.2F_IDL_.2F_my_favourite_commercial_software_on_SciNet.3F | the FAQ]]
|
|
|- style='background: white'
| gsl
| 1.13
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|- style='background: white'
| fftw
| 3.2.2
| fftw fast Fourier transform library
| <tt>libfftw3</tt>
| <tt>fftw</tt>
|- style='background: white'
| extras
|
| Full set of X11 libraries and others not installed on compute nodes
| <tt>bc, dmidecode, gv, iostat, lsof, tkdiff, zip, libXaw,...,libjpeg</tt>
| <tt>extras</tt>
|}

== TCS Software ==
The software listed below uses the same module system as described [[ #GPC_Software | above for the GPC]].
{| border="1" cellpadding="10" cellspacing="0"
! Software
! Version
! Comments
! Command/Library
! Module Name
|-
| antlr
| 2.7.7
| ANother Tool for Language Recognition
| <tt>antlr, antlr-config libantlr, antlr.jar, antlr.py</tt>
| <tt>antlr</tt>
|-
| gsl
| 1.13
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| HDF4
| 4.2.5
| Scientific data storage and retrieval
| <tt>h4fc, hdiff, ..., libdf, libsz</tt>
| <tt>hdf4</tt>
|-
| mpe 2
| 1.0.6
| Performance Visualization for Parallel Programs
| <tt>libmpe</tt>
| <tt>mpe</tt>
|-
| NetCDF + ncview
| 4.0.1
| Scientific data storage and retrieval
| <tt>ncdump, ncgen, libnetcdf, ncview</tt>
| <tt>netcdf</tt>
|-
| NCL
| 5.1.1
| NCAR Command Language
| <tt>ncl, libncl, ...</tt>
| <tt>ncl</tt>
|-
| NCO
| 3.9.6
| NCO utilities to manipulate netCDF files
| <tt>ncap, ncap2, ncatted, </tt> etc.
| <tt>nco</tt>
|-
| parallel netCDF
| 1.1.1
| Scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
| scalasca
| 1.2
| SCalable performance Analysis of LArge SCale Applications
| <tt>scalasca, ...</tt>
| <tt>scalasca</tt>
|-
| upc
| 1.2
| Unified Parallel C
| <tt>xlupc</tt>
| <tt>upc</tt>
|-
| extras
|
| Adds paths to a fuller set of libraries to your user environment compile with <tt>-I$SCINET_EXTRAS_INC</tt> link with <tt>-L$SCINET_EXTRAS_LIB</tt>
| <tt>libfftw, libfftw_mpi, libfftw3, libhdf5, liblapack, ...</tt>
| <tt>extras</tt>
|-|}