Difference between revisions of "MARS"

From oldwiki.scinet.utoronto.ca
Jump to navigation Jump to search
m
 
(43 intermediate revisions by 2 users not shown)
Line 1: Line 1:
−
== '''Massive Archive and Retrieval System''' ==
+
{| style="border-spacing: 8px; width:100%"
−
 
+
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|
−
(Pilot usage phase to start in Jun/2011 with a select group of users. Deployment and configuration are still a work in progress)
+
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''
−
 
 
−
The SciNet Massive Archive and Retrieval System (MARS) is a tape backed hierarchical storage management system that will provide a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data will be returned to the active GPFS filesystem when it is needed.
 
−
 
 
−
 
 
−
Migration of data into and out of the repository will be under the control of the user who will interact with system using one or both of the
 
−
following utilities:
 
−
* HSI is a client with an ftp-like interface will be used to archive and retrieve large files. It is also useful for browsing the contents of the repository.
 
−
* HTAR is a utility that creates tar format archives resident in the archive. It also creates a separate index file that can be accessed quickly.
 
−
 
 
−
 
 
−
User access will be controlled by the job scheduling system of the GPC. An interactive session can be requested that will allow a user to list, rearrange or remove files with the HSI client. Transfer of data into or out of the archive is expected to be scripted and submitted as a batch job.
 
−
 
 
−
 
 
−
== Guidelines ==
 
−
* HPSS storage space is provided on tape -- a media that is not suited for storing small files. Files smaller than ~100MB should be grouped into archive files with tar or htar.
 
−
* The maximum size of a file that can be transferred into the archive is 1TB. However, optimal performance is obtained with file sizes <= 100 GB.
 
−
* Make sure to check the application return code and check the  log file for errors after all data transfers.
 
−
* '''Pilot users:''' <span style="color:#CC0000">DURING THE TESTING PHASE DO NOT DELETE THE ORIGINAL FILES FROM /scratch OR /project</span>
 
−
 
 
−
== Access Through the Queue System  ==
 
−
All access to the archive system is through the queue system.
 
−
=== Interactive Access ===
 
−
To get an interactive session and use the HSI client, use the -I option of qsub.
 
−
<pre>
 
−
gpc04 $ qsub -q archive -I
 
−
hpss-archive01:~ $ hsi ls
 
−
******************************************************************
 
−
*  Welcome to the Massive Archive and Restore System @ SciNet  *
 
−
*                                                                *
 
−
*        Contact Information: support@scinet.utoronto.ca         *
 
−
*  NOTE: do not transfer SMALL FILES with HSI. Use HTAR instead  *
 
−
******************************************************************
 
−
[HSI]/archive/group/user->
 
−
</pre>
 
−
 
 
−
=== Scripted File Transfers ===
 
−
File transfers in and out of the archive should be scripted into jobs and submitted to the archive queue.
 
−
<pre>
 
−
#!/bin/env bash
 
−
#PBS -q archive
 
−
#PBS -N hsi_file_transfer
 
−
#PBS -j oe
 
−
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID
 
−
 
 
−
/usr/local/bin/hsi  -v <<EOF
 
−
cput -p /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
 
−
EOF
 
−
if [ ! $? == 0 ];then
 
−
 
 
−
 
 
−
</pre>
 
−
 
 
−
 
 
−
 
 
−
== '''Using HSI''' ==
 
−
 
 
−
HSI is the primary client with which a user will interact with the archive system. It provides an ftp-like interface for archiving and retrieving files. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents of the archive. The most commonly used commands will be:
 
−
{|border="1" cellpadding="10" cellspacing="0"
 
−
|-
 
−
  | cput
 
−
  | Conditionally stores a file only if the HPSS file does not exist
 
−
|-
 
−
  | cget
 
−
  | Conditionally retrieves a copy of a file from HPSS to your local file space on the host system only if a local copy does not already exist.
 
−
|-
 
−
  | cd,mkdir,ls,rm,mv
 
−
  | Operate as one would expect on the contents of the archive.
 
−
|-
 
−
  | lcd,lls
 
−
  | ''Local'' commands.
 
 
|}
 
|}
  
−
Simple commands can be executed on a single line.
+
MARS is not more. [https://support.scinet.utoronto.ca/wiki/index.php/HPSS Follow this link]
−
<pre>
 
−
  hsi "mkdir examples; cd examples; cput example_data.tgz
 
−
</pre>
 
−
 
 
−
More complex operations can be performed using a Here Document.
 
−
<pre>
 
−
hsi <<-EOF
 
−
  mkdir -p examples/201106
 
−
  cd examples
 
−
  mv example_data.tgz 201106/
 
−
  lcd /scratch/knecht/examples/
 
−
  cput -R -u *
 
−
EOF
 
−
</pre>
 
−
 
 
−
 
 
−
=== HSI Documentation ===
 
−
* [http://www.mgleicher.us/GEL/hsi/hsi_man_page.html man hsi]
 
−
* [http://www.mgleicher.us/GEL/hsi/hsi-exit-codes.html exit codes]
 
−
 
 
−
=== Examples ===
 
−
 
 
−
* Put a subdirectory ''LargeFiles'' and all its contents recursively. You may use '-u' option to resume a previously disrupted session (as rsync would do).
 
−
<pre>
 
−
    hsi "prompt; mput -R -u LargeFiles"
 
−
</pre>
 
−
 
 
−
* Interactively descend into the ''Source'' directory and move all files which end in ".h" into a sibling directory (ie, a directory at the same level in the tree as "Source") named "Include":
 
−
<pre>
 
−
    hsi <RETURN>
 
−
    [HSI] cd Source
 
−
    [HSI] mv *.h ../Include
 
−
</pre>
 
−
 
 
−
* Delete all files beginning with "m" and ending with 9101 (note that this is an interactive request, not a one-liner request, so the wildcard path does not need quotes to preserve it):
 
−
<pre>
 
−
    hsi <RETURN>
 
−
    [HSI] delete m*9101
 
−
</pre>
 
−
 
 
−
* Interactively delete all files beginning with H and ending with a digit, and ask for verification before deleting each such file.
 
−
<pre>
 
−
    hsi <RETURN>
 
−
    [HSI] mdel H*[0-9]
 
−
</pre>
 
−
 
 
−
* From a shell, save your local files that begin with the letter "c" (let the UN*X shell resolve the wild-card path pattern in terms of your local files by not enclosing it in quotes):
 
−
<pre>
 
−
    hsi mput c*
 
−
</pre>
 
−
 
 
−
* From a shell, get all files in the subdirectory ''subdirA'' which begin with the letters "b" or "c" (surrounding the wildcard path in single quotes prevents shells on UNIX systems from processing the wild card pattern):
 
−
<pre>
 
−
    hsi get ’subdirA/[bc]*’
 
−
</pre>
 
−
 
 
−
* Save a "tar file" of C source programs and header files:
 
−
<pre>
 
−
  tar cf - *.[ch] | hsi put - : source.tar
 
−
</pre>
 
−
Note: the ":" operator which separates the local and HPSS pathnames must be surrounded by whitespace (one or more space characters)
 
−
 
 
−
* Restore the tar file source kept above and extract all files:
 
−
<pre>
 
−
    hsi get - : source.tar | tar xf -
 
−
</pre>
 
−
 
 
−
* The commands below are equivalent (the default HSI directory placement is /archive/<group>/<user>/):
 
−
<pre>
 
−
    hsi put source.tar
 
−
    hsi put source.tar : /archive/<group>/<user>/source.tar
 
−
</pre>
 
−
 
 
−
* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' online or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help'''] from the hsi prompt.
 
−
 
 
−
== '''Using HTAR''' ==
 
−
 
 
−
* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, enter:
 
−
<pre>
 
−
    htar -cf files.tar file1 file2
 
−
OR
 
−
    htar -cf /archive/<group>/<user>/files.tar file1 file2
 
−
</pre>
 
−
 
 
−
* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
 
−
<pre>
 
−
    htar -cf subdirA.tar subdirA
 
−
</pre>
 
−
 
 
−
*  To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' on a remote FTP server called "blue.pacific.llnl.gov", creating the tar file in the user’s remote FTP home directory, enter (bonus HTAR functionality to sites outside SciNet):
 
−
<pre>
 
−
    htar -cf files.tar -F blue.pacific.llnl.gov file1 file2
 
−
</pre>
 
−
 
 
−
* To extract all files from the ''project1/src'' directory in the Archive file called ''proj1.tar'', and use the time of extraction as the modification time, enter:
 
−
<pre>
 
−
    htar -xm -f proj1.tar project1/src
 
−
</pre>
 
−
 
 
−
* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
 
−
<pre>
 
−
    htar -vtf out.tar
 
−
</pre>
 
−
 
 
−
For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online
 
−
 
 
−
== '''More detailed examples''' ==
 
−
[[GPC_Quickstart|Submitting_A_Batch_Job]]
 
−
* gpc-archive01 is part of the gpc queuing system under torque/moab
 
−
* Currently it is setup to share the node with up to 12 jobs at one time
 
−
* default parameters ( -l nodes=1:ppn=1,walltime=48:00:00)
 
−
<pre>
 
−
showq -w class=archive
 
−
 
 
−
qsub -I -q archive
 
−
</pre>
 
−
 
 
−
* sample '''data offload'''
 
−
<pre>
 
−
#!/bin/bash
 
−
 
 
−
# This script is named: data-offload.sh
 
−
 
 
−
#PBS -q archive
 
−
#PBS -N offload
 
−
#PBS -j oe
 
−
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID
 
−
 
 
−
date
 
−
 
 
−
# individual tarballs already exist
 
−
/usr/local/bin/hsi  -v <<EOF
 
−
mkdir put-away-and-forget
 
−
cd put-away-and-forget
 
−
put /scratch/$USER/workarea/finished-job1.tar.gz : finished-job1.tar.gz
 
−
put /scratch/$USER/workarea/finished-job2.tar.gz : finished-job2.tar.gz
 
−
EOF
 
−
 
 
−
# create a tarball on-the-fly of the finished-job3 directory
 
−
/usr/local/bin/htar -cf finished-job3.tar /scratch/$USER/workarea/finished-job3/
 
−
 
 
−
date
 
−
</pre>
 
−
* sample '''data list'''
 
−
  - Very painful without interactive browsing
 
−
      -Tentative solution: dump all user files to log file and use that as file index
 
−
<pre>
 
−
#!/bin/bash
 
−
 
 
−
# This script is named: data-list.sh
 
−
 
 
−
#PBS -q archive
 
−
#PBS -N hpss_dump
 
−
#PBS -j oe
 
−
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID
 
−
 
 
−
date
 
−
echo ===========
 
−
echo
 
−
/usr/local/bin/hsi  -v <<EOF
 
−
ls -lUR
 
−
EOF
 
−
echo
 
−
echo ===========
 
−
date
 
−
</pre>
 
−
 
 
−
* sample '''data restore'''
 
−
<pre>
 
−
#!/bin/bash
 
−
 
 
−
# This script is named: data-restore.sh
 
−
 
 
−
#PBS -q archive
 
−
#PBS -N restore
 
−
#PBS -j oe
 
−
#PBS -o hpsslogs/$PBS_JOBNAME.$PBS_JOBID
 
−
 
 
−
date
 
−
 
 
−
mkdir -p /scratch/$USER/restored-from-MARS
 
−
 
 
−
/usr/local/bin/hsi  -v << EOF
 
−
get /scratch/$USER/restored-from-MARS/Jan-2010-jobs.tar.gz : forgotten-from-2010/Jan-2010-jobs.tar.gz
 
−
get /scratch/$USER/restored-from-MARS/Feb-2010-jobs.tar.gz : forgotten-from-2010/Feb-2010-jobs.tar.gz
 
−
EOF
 
−
 
 
−
cd /scratch/$USER/restored-from-MARS
 
−
/usr/local/bin/htar -xf finished-job3.tar
 
−
 
 
−
date
 
−
</pre>
 
−
 
 
−
* sample '''analysis''' (depends on previous data-restore.sh execution)
 
−
<pre>
 
−
gpc04 $ qsub $(qsub data-restore.sh | awk -F '.' '{print "-W depend=afterok:"$1}') job-to-work-on-restored-data.sh
 
−
</pre>
 

Latest revision as of 19:37, 31 August 2018

WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to https://docs.scinet.utoronto.ca

MARS is not more. Follow this link

Retrieved from ‘https://oldwiki.scinet.utoronto.ca/index.php?title=MARS&oldid=9413’