ISH

ish - inventory shell

Introduction

Browse content if directories, tarball and purging lists from a local inventory file (or index).

ish is located in /scinet/gpc/bin/ish.

Current version is 0.98 (a beta version).

Ramses van Zon - SciNet/Toronto/Canada/July 8, 2011

Usage

From the command line

   ish -h|--help                      show this help
   ish --version                      show version number
   ish [INDEX]                        interactive shell for file INDEX
   ish [INDEX] COMMAND                perform single COMMAND on file INDEX

   When INDEX is omitted, hpss.igz is loaded if available.

Shell Commands

 Most common
   ls [-lr]   [DIR/[FILES]] ...       list FILES from DIR in index
   cd         DIR                     set current directory in index
   find       PATTERN ...             find files following PATTERN in index
   index      DIR|TARFILE             make index for folder/tarfile
   exit                               exit the ish shell

 More
   avail      [-a]                    list (all) available index files
   colour     1|0                     set colour usage
   help       [COMMAND]               show help on (all) commands
   register   [DIR]                   set new index file location 
   use        [INDEX]                 use INDEX or list available ones
   info                               show properties of the index file
   pwd                                show current directory
   settings                           show settings (colour, etc.)
   tar [-zcf] TARFILE DIR[/FILES] ... tar and make index
   check [-n] [COMMENT]               exit ish if error in last command
   !COMMAND   [ARGS]                  local commands (ls, cd, pwd only)
   pindex     [FILE]                  make index from a purge listing

 Only in archive queue (hpss)
   hindex     DIR|TARFILE             make index for hpss folder/tarfile
   htar [-cf] TARFILE DIR[/FILES] ... htar and make index

Command line examples

 Local tar and gzip directory 'code': ish tar -z code.tgz code
 Long list of its content:            ish code.tgz.igz ls -l 'code/*'
 Make index file for existing tar:    ish index another.tar
 List its top level content:          ish another.tar.igz ls
 List all of its content:             ish another.tar.igz ls -r
 Find file 'hello.txt' in it:         ish another.tar.igz find hello.txt
 Make index hspp.igz for hpss tree:   ish hindex
 Create index file for an htar file:  ish hindex data.tar
 Create htar and index file:          ish htar -cpf data.tar data/

Ish commands

help - ish command to show help on ish commands

Usage

   help [COMMAND]

If no COMMAND is given, a list of all ish commands, with a brief description, is given.

ls - ish command to list directory contents

Usage

   ls [OPTION] [ PATTERN [PATTERN ... ] ]

Lists files in the index according to one or more patterns. In interactive mode, the files are displayed in colour. The list is sorted alphabetically by name.

Patterns are of the form [PATH[/]][FILES] and may contain wildcards * and ?. Without FILES, the form PATH only lists the directory name, while PATH/ lists the files in the directory.

When no patterns are given, ls lists files in the current directory in the current index file (as given on the command line or set with use).

Paths are relative to the current directory (as set with 'cd'). To specify a path from the root of the index, an initial colon (:) should be put in front of the path. If the index contains absolute paths, the root can be indicated by an initial slash (/) as well.

The optional argument can be

 -l  list in long format, displaying information on file sizes,
     modification times, and other metadata present in the index.
 -r  lists recursively into subdirectories.

To give both options, they have to be combined into one, e.g. -lr.

cd - ish command to set current directory in index

Usage

   cd [PATH]

Change the current directory within the index to the directory specified by PATH.

The new path is relative to the previous current directory, unless preceded by a colon (:), in which case it specifies a path from the root of the index. If the index contains absolute paths, the root can be indicated by an initial slash (/) as well.

If no PATH is given, the current directory is set to the root directory of the index file.

The current directory is always displayed in the ish prompt.

find - ish command to recursively find files in index

Usage

   find PATTERN ...

Recursively searches the directory tree in the index from the current directory (set by 'cd') for files following one or more patterns. PATTERN may be of the form PATH/FILE or just FILE, and can contain the wildcards * and ?.

exit - ish command to exit ish

Usage

   exit 

Ish will exit with the exit status of the last command that was run.

When input is redirected to be read from a file or here document, the end-of-file also exits ish, so in that case exit is optional.

index - ish command to make an index for a directory or tarfile

Usage

   index PATH|TARFILE

Makes an index file of a directory (PATH) or tar file (TARFILE). The PATH, or the paths stored in the tar file, can be either absolute or relative (not both). The tar file can be in compressed format, provided the local tar installation supports it.

The index files are stored in the register directory (~/.ish_register by default). See 'register' for details.

For consistency, if the parent directory of any directory in PATH or the TARFILE is not in that PATH or in that TARFILE, it gets added to the index as a stub (i.e., not with its full content). For example, 'index /home/rzon' will include /home as a stub, which will seem to only contain the directory rzon. Similarly, if test.tar was created (not from within ish) with 'tar cf test.tar /home/rzon', then 'index test.tar' will add a stub for /home.

Ish automatically assigns a name to the index file, following this naming convention

• A directory index stored in the current directory will be called PATH.igz, with any slashes replaced by underscores.
• A directory index stored in the registry will be called ABSPATH.igz, where ABSPATH is the absolute path to the directory PATH, again, with slashes replaced by underscores.
• A tar index will be called TARFILE.igz, with slashes replaced with underscores.
• When running an index command again, or running an index command that would result in the name of an existing index that would be overwritten, the existing index file NAME.igz gets renamed to NAME_DATE_TIME.igz, where date is the date of the existing index file. The exception to this rule is that if the new index file has identical content to the old one, the old one is removed.

Once the indexing is done, the index file becomes the current one in the ish shell.

The index file for a tar file can be created when the tar is made, using the ish command tar (see 'tar').

colour - ish command to set colour usage

Usage

   colour ARG

Switches the usage of colour in listings on if ARG=1 and off if ARG=0. By default, colour is on in interactive sessions, and off in single command mode.

The current colour setting can be found with the 'settings' command.

pwd - ish command to show current directory

Usage

   pwd 

Shows the current directory within the index (as set by 'cd').

This command should be rarely needed, as the current directory is always displayed as part of the ish prompt.

register - ish command to set new index file location

Usage

   register [DIR]

Sets the location where index files are stored. This location is called the register. DIR must be an existing directory on the file system. If DIR is omitted, the register is set to the default location.

The default location of the register is the (hidden) directory ~/.ish_register, or is set by the environment variable ISHREGISTER.

Index files in the current directory will be found by 'use' as well, regardless of the register setting. But to store an index file in the current directory (using 'index', 'hindex', or 'pindex'), the register has to be set to '.' first.

The current setting of the register can be found with 'settings'.

settings - ish command to list settings

Usage

   settings 

Lists settings such as colour and register location.

tar - ish command to tar and make index

Usage

   tar [OPTION] TARFILE [DIR][FILES] ...

Tars a directory or files list as usual, AND creates the corresponding index file. It should be equivalent to running the same tar command from the shell (i.e. not within ish), followed by "ish index TARFILE".

Common tar options

 -c  create the file.
 -z  gzip the tarball. 
 -j  bzip2 the tarball.
 -f  should be the last option, and indicates that output should go to
     the tarfile. Has to be the last option.

Other options can be found from the tar man page. Multiple options have to be combined into one, e.g. -czf.

avail - ish command to list available index files

Usage

   avail [-a]

Displays a list of index files the register directory (see 'register') or the current directory. These index files can be accessed with the 'use' command.

The list contains the creation date and time of the igz file. Not that this need not be the same as the creation date of the corresponding tar.

By default, older versions of the igz files (which are automatically saved by index and hindex) are not shown.

The optional argument can be

 -a  When given, older saved versions of the igz files are also listed.

See 'index', 'hindex' and 'pindex' on how to create index files, and on their naming scheme.

use - ish command to set the index file

Usage

   use INDEX

Sets the current index file to INDEX. This file should reside in the register directory (see 'register') or the current directory. INDEX should include the index file extension '.igz'.

Note that the index file can also be set when starting ish with the name of the index file as the first argument (e.g. ish data.tgz.igz). Omitting the index file when starting ish will cause if to look for hpss.igz.

See 'index', 'hindex' and 'pindex' on how to create index files, and on their naming scheme.

The current index file is always displayed in the ish prompt.

!COMMAND - ish command to local commands (ls, cd, pwd only)

Usage

   !COMMAND [ARGS]

Runs a local command. This is intended to allow the local directory to be changed and listed. Any arguments will be passed to the command.

Only the commands ls, cd and pwd will be accepted to run locally.

hindex - ish command to make index for hpss folder/tarfile (hpss only)

Usage

   hindex [PATH|TARFILE]

Makes an index file of a directory (PATH) or tar file (TARFILE) on hpss. The distinction is made on whether the argument has the extension .tar, .tar.gz, .tgz, .tar.bz2, .tbz2, or tb2.

The PATH, or the paths stored in the tar file, can be either absolute or relative (not both). The tar file cannot be in compressed format, and ideally should have been created with htar.

If neither a PATH nor a TARFILE is given, hindex will generate an index for your whole hpss directory, and store the index in hpss.igz.

For consistency, if the parent directory of any directory in PATH or the TARFILE is not in that PATH or in that TARFILE, it gets added to the index as a stub (i.e., not with its full content). For example, 'hindex /archive/scinet/rzon' will include /archive and /archive/scinet as a stubs, which will seem to only contain one directory. Similarly, if test.tar was created (not from within ish) with 'htar cf test.tar /home/rzon', then 'hindex test.tar' will add a stub for /home.

Ish automatically assigns a name to the index file, following this naming convention

• The directory index will be called PATH.igz, with any slashes replaced by underscores.
• A tar index will be called TARFILE.igz, with slashes replaced with underscores.
• The index will be called hpss.igz when hindex is called without argument.
• When running an hindex command again, or running an hindex command that would result in an existing index getting overwritten, the existing index file NAME.igz gets renamed to NAME_DATE_TIME.igz, where DATE and TIME are of the existing index file. The exception to this rule is that if the new index file has identical content to the old one, the old one is removed.

Ish calls the hsi application under the hood to get the directory listings. This means that this command has to be run on a system that has hpss with hsi. This may mean you have to submit a job for building the index. With the index file, you can then locally traverse the listings in the index and see the modification times and file sizes.

If the tar file was created with htar (outside of ish), then the index is created from the remote index file generated by a call to htar. So this command may have to be run on a system that has hpss with htar.

If the hpss-resident tar file was created with tar (outside of ish), then no remote index file exists and ish will first request htar to generate one. Because this may require the tar to be reclaimed from tape, this is an expensive operation. So it is recommended to 1) use htar, or, 2) at the point of tarring, use ish's tar command, or, 3) if the tar is already created and still available locally, to use the ish command 'index'.

Once the indexing is done, the index file becomes the current one in the ish shell.

The index files are stored in the register directory (~/.ish_register by default). See 'register' for details.

Note that the htarring and the creation of a local index file can be done with one command, using the 'htar' ish command.

htar - ish command to htar and make index (hpss only)

Usage

   htar OPTIONS TARFILE [DIR][FILES] ...

Htars a directory or set of files list as usual, AND creates the corresponding local index file. It should be equivalent to running the same htar command from the shell (i.e. not from within ish), followed by "ish hindex TARFILE".

Common htar options

 -c  create the file. 
 -p  Preserve time stamps.
 -f  always the last option; indicates that output should go to a file. 

To give multiple options, they have to be combined into one, e.g. -cpf. Other options can be found from the htar man page.

info - ish command to show information on the current index file

Usage

   info 

Lists information such as creation date, ish version, how it was created.

check - ish command to exit ish if the previous command had an error

Usage

   check [-n] [COMMENT]

Ish will print the COMMENT, then the error message of the last command, and will exit with its exit code. Very useful and recommended when doing multiple nontrivial actions in one ish session.

If the optional argument -n is given, ish does not exit.

pindex - ish command to make an index for a purge file

Usage

   pindex [FILE]

Makes an index file from the monthly scratch purge file generated for you at Scinet. (Obviously, this is a very SciNet-specific command). This list only contains files, so for consistency, stubs are added for directories.

If no FILE is given, ish searches for your most current purge list.

The name of the index file is composed of the file name of the purge file, without underscores, and prepended by a path with slashes replaced by underscores. The index file is stored in the register directory (~/.ish_register by default). See 'register' for details.

Once the indexing is done, the index file becomes the current one in the ish shell.

BACK TO HPSS