MARS

Massive Archive and Restore System

The MARS deployment at SciNet is a combination of 3 software components, HPSS, HSI and HTAR, plus some customization done to our environment.

HPSS: the main component, best described as a "blackbox" engine running in the background to support the Archive and Restore operations.

HSI: it may be best understood as a supercharged ftp interface, specially designed to act as a front-end for HPSS, gathering some of the best features you would encounter on a shell, rsync and GridFTP. In general terms it enables user to transfer whole directory trees from our most active file systems, therefore freeing up space in /project and /scratch. HSI is most suitable when those directory trees do not contain too many small files to start with, or when you already have a series of tarballs.

HTAR: similarly, htar is sort of a "super-tar" application, also specially designed to interact with HPSS, allowing users to auto-magically build and transfer larger tarballs to/from HPSS. HTAR is most suitable to aggregate whole directory trees, provided that no individual files exceed 68GB. The maximum size of any htar file should not exceed 1T either.