Difference between revisions of "MARS"

From oldwiki.scinet.utoronto.ca
Jump to navigation Jump to search
m
 
(19 intermediate revisions by one other user not shown)
Line 1: Line 1:
−
= '''Massive Archive and Retrieval System''' =
+
{| style="border-spacing: 8px; width:100%"
−
 
+
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|
−
(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)
+
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''
−
 
 
−
The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.
 
−
 
 
−
 
 
−
Migration of data into and out of the repository will be under the control of the user who will interact with the system using one or both of the
 
−
following utilities:
 
−
* HSI is a client with an ftp-like interface which can be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
 
−
* HTAR is a utility that creates tar format archives resident in the repository. It also creates a separate index file that can be accessed quickly.
 
−
* ISH is a TUI utility that can perform an inventory of the files and directories in your tarballs.
 
−
 
 
−
User access will be controlled by the job scheduling system of the GPC. Transfer of data into or out of the repository is expected to be scripted and submitted as a batch job.
 
−
 
 
−
== '''Guidelines''' ==
 
−
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
 
−
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
 
−
* Make sure to check the application return code and check the  log file for errors after all data transfers.
 
−
* '''Pilot users:''' <span style="color:#CC0000">DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project</span>
 
−
 
 
−
== '''Access Through the Queue System'''  ==
 
−
All access to the archive system is through the GPC queue system documented [[Moab|here]].
 
−
=== Scripted File Transfers ===
 
−
File transfers in and out of the archive should be scripted into jobs and submitted to the ''archive'' queue. Script will use either the HSI or HTAR commands documented below.
 
−
<pre>
 
−
#!/bin/env bash
 
−
#PBS -q archive
 
−
#PBS -N hsi_put_file_in_repo
 
−
#PBS -j oe
 
−
#PBS -me
 
−
 
 
−
/usr/local/bin/hsi  -v <<EOF
 
−
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
 
−
EOF
 
−
status=$?
 
−
if [ ! $status == 0 ];then
 
−
  echo '!!! TRANSFER FAILED !!!'
 
−
fi
 
−
exit $status
 
−
</pre>
 
−
 
 
−
The status of pending jobs can be monitored with showq specifying the archive queue:
 
−
<pre>
 
−
showq -w class=archive
 
−
</pre>
 
−
 
 
−
=== Recalling Data for Analysis ===
 
−
 
 
−
Typically, data will be recalled to the /scratch file system when it is needed for analysis. Job dependencies can be used to have analysis jobs wait in the queue for data recalls before starting. The qsub flag is
 
−
<pre>
 
−
-W depend=afterok:<JOBID>
 
−
</pre>
 
−
where JOBID is the job number of the staging job that must finish successfully before the analysis job can start.
 
−
 
 
−
Here is a short cut for generating the dependency:
 
−
<pre>
 
−
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
 
−
</pre>
 
−
 
 
−
== '''Using HSI''' ==
 
−
 
 
−
HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
 
−
{|border="1" cellpadding="10" cellspacing="0"
 
−
|-
 
−
  | cput
 
−
  | Conditionally stores a file only if the HPSS file does not exist
 
−
|-
 
−
  | cget
 
−
  | Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
 
−
|-
 
−
  | cd,mkdir,ls,rm,mv
 
−
  | Operate as one would expect on the contents of the archive.
 
−
|-
 
−
  | lcd,lls
 
−
  | ''Local'' commands.
 
 
|}
 
|}
  
−
Simple commands can be executed on a single line.
+
MARS is not more. [https://support.scinet.utoronto.ca/wiki/index.php/HPSS Follow this link]
−
<pre>
 
−
  hsi "mkdir examples; cd examples; cput example_data.tgz"
 
−
</pre>
 
−
 
 
−
More complex operations can be performed using a Here Document.
 
−
<pre>
 
−
hsi <<-EOF
 
−
  mkdir -p examples/201106
 
−
  cd examples
 
−
  mv example_data.tgz 201106/
 
−
  lcd /scratch/$USER/examples/
 
−
  cput -R -u *
 
−
EOF
 
−
status=$?
 
−
if [ ! $status == 0 ];then
 
−
  echo '!!! TRANSFER FAILED !!!'
 
−
fi
 
−
exit status
 
−
</pre>
 
−
 
 
−
=== HSI Documentation ===
 
−
Complete documentation of HSI is available on the Gleicher Enterprises web site.
 
−
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
 
−
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]
 
−
 
 
−
=== Typical Usage ===
 
−
The most common interactions with the HPSS archive will be ''putting'' data into the archive, examining the contents of the archive (ls), and ''getting'' data back onto one of the active filesystems for inspection or analysis.
 
−
 
 
−
* sample '''data offload'''
 
−
<pre>
 
−
#!/bin/bash
 
−
 
 
−
# This script is named: data-offload.sh
 
−
 
 
−
#PBS -q archive
 
−
#PBS -N offload
 
−
#PBS -j oe
 
−
#PBS -me
 
−
 
 
−
date
 
−
 
 
−
# individual tarballs already exist
 
−
/usr/local/bin/hsi  -v <<EOF
 
−
mkdir put-away
 
−
cd put-away
 
−
cput /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
 
−
cput /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
 
−
EOF
 
−
status=$?
 
−
if [ ! $status == 0 ];then
 
−
  echo '!!! TRANSFER FAILED !!!'
 
−
fi
 
−
exit $status
 
−
</pre>
 
−
 
 
−
* sample '''data list'''
 
−
    - This example performs a recursive list of all files in a user's portion of the namespace. The list is placed in a dated file in the directory /home/$USER/HPSS.ix that can be inspected from the gpc-devel nodes.
 
−
<pre>
 
−
#!/bin/bash
 
−
 
 
−
# This script is named: data-list.sh
 
−
 
 
−
#PBS -q archive
 
−
#PBS -N hpss_dump
 
−
#PBS -j oe
 
−
#PBS -me
 
−
 
 
−
TODAY=$(date +%Y%m%d)
 
−
INDEX_DIR=/home/$USER/HPSS.ix
 
−
if [[ -! -e $INDEX_DIR ]];then
 
−
  mkdir $INDEX_DIR
 
−
fi
 
−
 
 
−
/usr/local/bin/hsi  <<EOF > $INDEX_DIR/contents-$TODAY
 
−
ls -lUR
 
−
EOF
 
−
 
 
−
</pre>
 
−
 
 
−
* sample '''data restore'''
 
−
  - This example should be modified to emphasize that a single ''cget'' of multiple files (rather than several separate gets) allows HSI to do staging optimization.
 
−
<pre>
 
−
#!/bin/bash
 
−
 
 
−
# This script is named: data-restore.sh
 
−
 
 
−
#PBS -q archive
 
−
#PBS -N restore
 
−
#PBS -j oe
 
−
#PBS -me
 
−
 
 
−
 
 
−
mkdir -p /scratch/$USER/restored-from-MARS
 
−
 
 
−
/usr/local/bin/hsi  -v << EOF
 
−
cget /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
 
−
cget /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
 
−
EOF
 
−
status=$?
 
−
 
 
−
 
 
−
exit $status
 
−
 
 
−
</pre>
 
−
 
 
−
=== HSI vs. FTP ===
 
−
HSI syntax and usage is very similar to that of FTP. Please note the following information adapted from the HSI man page:
 
−
 
 
−
HSI supports several of the commonly used FTP commands, including "dir","get","ls","mdelete","mget","put","mput" and "prompt", with the following differences:
 
−
 
 
−
* The "dir" command is an alias for "ls" in HSI. The "ls" command supports an extensive set of options for displaying files, including wildcard pattern-matching, and the ability to recursively list a directory tree
 
−
* The "put" and "get" family of commands support recursion
 
−
* There are "conditional put" and "conditional" get commands (cput, cget)
 
−
 
 
−
* The syntax for renaming local files when storing files to HPSS or retrieving files from HPSS is different than FTP. With HSI, the syntax is always
 
−
<pre>
 
−
    "local_file : HPSS_file"
 
−
</pre>
 
−
and multiple such pairs may be specified on a single command line.
 
−
With FTP, the local filename is specified first on a "put" command, and second on a "get" command.
 
−
 
 
−
For example, when using HSI to store the local file "file1" as HPSS file "hpss_file1", then retrieve it back to the local filesystem as "file1.bak", the following commands could be used:
 
−
<pre>
 
−
    put file1 : hpss_file1
 
−
    get file1.bak : hpss_file1
 
−
</pre>
 
−
 
 
−
 
 
−
* With FTP, the following commands could be used:
 
−
<pre>
 
−
    put file1 hpss_file1
 
−
    get hpss_file1 file1.bak
 
−
</pre>
 
−
* The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.
 
−
 
 
−
 
 
−
=== Other HSI Examples ===
 
−
 
 
−
* Creating tar archive on the fly by piping stdout:
 
−
<pre>
 
−
  tar cf - *.[ch] | hsi put - : source.tar
 
−
</pre>
 
−
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)
 
−
 
 
−
* Restore the tar file source kept above and extract all files:
 
−
<pre>
 
−
    hsi get - : source.tar | tar xf -
 
−
</pre>
 
−
 
 
−
* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
 
−
<pre>
 
−
    hsi put source.tar
 
−
    hsi put source.tar : /archive/<group>/<user>/source.tar
 
−
</pre>
 
−
 
 
−
* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.
 
−
 
 
−
== '''HTAR''' ==
 
−
''' Please aggregate small files (<~100MB) into tarballs or htar files. '''
 
−
 
 
−
HTAR is a utility that is used for aggregating a set of files from the local file system directly into HPSS, creating a file that conforms to the POSIX TAR specification.
 
−
It does this without having to first create an intermediate file on the local filesystem; instead, it uses a sophisticated multithreaded buffering scheme to write files directly into HPSS, thereby achieving a high rate of performance.
 
−
 
 
−
=== '''CAUTION''' ===
 
−
* Files larger than 68 GB cannot be stored in an htar archive.
 
−
* HTAR archives cannot contain more than 1 million files.
 
−
* Check the HTAR exit code and log file before removing any files from the active filesystems.
 
−
 
 
−
 
 
−
=== HTAR Usage ===
 
−
* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
 
−
<pre>
 
−
    htar -cf files.tar file1 file2
 
−
OR
 
−
    htar -cf /archive/<group>/<user>/files.tar file1 file2
 
−
</pre>
 
−
 
 
−
* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
 
−
<pre>
 
−
    htar -cf subdirA.tar subdirA
 
−
</pre>
 
−
 
 
−
* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
 
−
<pre>
 
−
    htar -xm -f proj1.tar project1/src
 
−
</pre>
 
−
 
 
−
* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
 
−
<pre>
 
−
    htar -vtf out.tar
 
−
</pre>
 
−
 
 
−
For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online
 
−
 
 
−
== '''More detailed examples''' ==
 
−
 
 
−
 
 
−
 
 
−
# create a tarball on-the-fly of the finished-job3 directory
 
−
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/
 
−
 
 
−
 
 
−
cd /scratch/$USER/restored-from-MARS
 
−
/usr/local/bin/htar -xf finished-job3.tar
 

Latest revision as of 19:37, 31 August 2018

WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to https://docs.scinet.utoronto.ca

MARS is not more. Follow this link

Retrieved from ‘https://oldwiki.scinet.utoronto.ca/index.php?title=MARS&oldid=9413’