Difference between revisions of "MARS"

Latest revision as of 19:37, 31 August 2018

WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to https://docs.scinet.utoronto.ca

MARS is not more. Follow this link

@@ Line 1: / Line 1: @@
-=== Massive Archive and Restore System ===
+{| style="border-spacing: 8px; width:100%"
+| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|
+'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''
+|}
-(Pilot usage phase to start in May/2011 with a select group of users. Deployment and configuration are still a work in progress)
+MARS is not more. [https://support.scinet.utoronto.ca/wiki/index.php/HPSS Follow this link]
-The '''MARS''' deployment at SciNet is an effort to offer a more efficient way to offoad/archive data from the most active file systems (scratch and project) than our current TSM-HSM solution, still without having to deal directly with the tape library or "tape commands"
-The system is a combination of the underlaying hardware infrastructure, 3 software components, HPSS, HSI and HTAR, plus some environment customization.
-'''HPSS''': the main component, best described as a very scalable "blackbox" engine running in the background to support the Archive and Restore operations. [http://www.hpss-collaboration.org/index.shtml High Performance Storage System - HPSS] is the result of over a decade of collaboration among five Department of Energy laboratories and IBM, with significant contributions by universities and other laboratories worldwide. For now the best way for SciNet users to [https://support.scinet.utoronto.ca/wiki/index.php/HPSS_compared_to_HSM-TSM understand HPSS] may be to compare it with our existing HSM-TSM implementation.
-'''HSI''': it may be best understood as a supercharged ftp interface, specially designed by [http://www.mgleicher.us/GEL/hsi/ Gleicher Enterprises] to act as a front-end for HPSS, gathering some of the best features you would encounter on a shell, rsync and GridFTP (and a few more). It enables users to transfer whole directory trees from /project and /scratch, therefore freeing up space. HSI is most suitable when those directory trees do not contain too many small files to start with, or when you already have a series of tarballs.
-'''HTAR''': similarly, htar is sort of a "super-tar" application, also specially designed by [http://www.mgleicher.us/GEL/htar/ Gleicher Enterprises] to interact with HPSS, allowing users to build and automatically transfer tarballs to HPSS on the fly. HTAR is most suitable to aggregate whole directory trees.
-== '''Quick Reference''' ==
-Files are organized inside HPSS in the same fashion as in /project. Users in the same group have read permissions to each other's archives.
-<pre>
-/archive/<group>/<user>
-</pre>
-== '''Using HSI''' ==
-<pre>
-</pre>
-== '''Using HTAR''' ==
-<pre>
-</pre>
-== '''Performance/Limits considerations''' ==
-* IN/OUT transfers to HPSS using HSI is bound to maximum of about '''4 files/second'''. Therefore do not attempt to transfer directories with too many (small) files inside. Instead use HTAR.
-* The maximum size that an individual file can have inside an HTAR is 68GB. Please be sure to fish out those files from the directories and transfer them with  HSI
-* The maximum size of a tar/htar file that HPSS will take is 1TB. Please do not generate tarballs that large.

Difference between revisions of "MARS"

Latest revision as of 19:37, 31 August 2018

Navigation menu

Search