Difference between revisions of "MARS"
| Line 3: | Line 3: | ||
The '''MARS''' deployment at SciNet is a combination of 3 software components, HPSS, HSI and HTAR, plus some customization done to our environment. | The '''MARS''' deployment at SciNet is a combination of 3 software components, HPSS, HSI and HTAR, plus some customization done to our environment. | ||
| − | '''HPSS''': the main component | + | '''HPSS''': the main component, best described as a "blackbox" engine running in the background to support the Archive and Restore operations. |
'''HSI''': it may be best understood as a supercharged ftp interface, specially designed to act as a front-end for HPSS, gathering some of the best features you would encounter on a shell, rsync or GridFTP for example. In general terms it enables user to transfer whole directory trees from our most active file systems, therefore freeing up space in /project and /scratch. HSI is most suitable when those directory trees do not contain too many small files to start with, or when you already have a series of tarballs. | '''HSI''': it may be best understood as a supercharged ftp interface, specially designed to act as a front-end for HPSS, gathering some of the best features you would encounter on a shell, rsync or GridFTP for example. In general terms it enables user to transfer whole directory trees from our most active file systems, therefore freeing up space in /project and /scratch. HSI is most suitable when those directory trees do not contain too many small files to start with, or when you already have a series of tarballs. | ||
| − | '''HTAR''': similarly, htar is sort of a "super-tar" application, also specially designed to interact with HPSS, allowing users to auto-magically build | + | '''HTAR''': similarly, htar is sort of a "super-tar" application, also specially designed to interact with HPSS, allowing users to auto-magically build and transfer larger tarballs to/from HPSS. HTAR is most suitable to aggregate whole directory trees, provided that no individual files exceed 68GB. The maximum size of any htar file should not exceed 1T either. |
Revision as of 12:00, 27 April 2011
Massive Archive and Restore System
The MARS deployment at SciNet is a combination of 3 software components, HPSS, HSI and HTAR, plus some customization done to our environment.
HPSS: the main component, best described as a "blackbox" engine running in the background to support the Archive and Restore operations.
HSI: it may be best understood as a supercharged ftp interface, specially designed to act as a front-end for HPSS, gathering some of the best features you would encounter on a shell, rsync or GridFTP for example. In general terms it enables user to transfer whole directory trees from our most active file systems, therefore freeing up space in /project and /scratch. HSI is most suitable when those directory trees do not contain too many small files to start with, or when you already have a series of tarballs.
HTAR: similarly, htar is sort of a "super-tar" application, also specially designed to interact with HPSS, allowing users to auto-magically build and transfer larger tarballs to/from HPSS. HTAR is most suitable to aggregate whole directory trees, provided that no individual files exceed 68GB. The maximum size of any htar file should not exceed 1T either.