Difference between revisions of "ISH"

From oldwiki.scinet.utoronto.ca
Jump to navigation Jump to search
m
Line 1: Line 1:
 
'''ish - inventory shell'''
 
'''ish - inventory shell'''
  
 +
==Introduction==
 
Browse directories or tarball content from local inventory files.
 
Browse directories or tarball content from local inventory files.
  
 
ish is located in /scinet/gpc/bin/ish.
 
ish is located in /scinet/gpc/bin/ish.
  
−
Ramses van Zon - SciNet/Toronto/Canada/24-06-2011
+
Current version is 0.9 (a beta version).
 +
 
 +
==Usage==
 +
 
 +
===From the command line===
 +
    ish -h|--help                      show this help
 +
    ish --version                      show version number
 +
    ish [INDEX]                        interactive shell for file INDEX
 +
    ish [INDEX] COMMAND                perform single COMMAND on file INDEX
 +
 
 +
===Shell Commands===
 +
 
 +
Most common
 +
    ls [-lr]  [DIR/[FILES]] ...      list FILES from DIR in index
 +
    cd        DIR                    set current directory in index
 +
    find      PATTERN ...            find files following PATTERN in index
 +
    index      DIR|TARFILE            make index for folder/tarfile
 +
    exit                              exit the shell
 +
 
 +
More
 +
    avail      [-a]                    list (all) available index files
 +
    colour    1|0                    set colour usage
 +
    help      [COMMAND]              show help on (all) commands
 +
    pwd                                show current directory
 +
    register  [DIR]                  set new index file location
 +
    settings                          list settings
 +
    tar [-zcf] TARFILE DIR[/FILES] ... tar and make index
 +
    use        [INDEX]                use INDEX or list available ones
 +
    !COMMAND  [ARGS]                  local commands (ls, cd, pwd)
 +
 
 +
Only in archive queue (hpss)
 +
    hindex    DIR|TARFILE            make index for hpss folder/tarfile
 +
    htar [-cf] TARFILE DIR[/FILES] ... htar and make index
 +
 
 +
===Command line examples===
 +
 
 +
  Local tar and gzip directory 'code': ish tar -z code.tgz code
 +
  Long list of its content:            ish code.tgz.igz ls -l 'code/*'
 +
  Make index file for existing tar:    ish index another.tar
 +
  List its top level content:          ish another.tar.igz ls
 +
  List all of its content:            ish another.tar.igz ls -r
 +
  Find file 'hello.txt' in it:        ish another.tar.igz find hello.txt
 +
  Make index hspp.igz for hpss tree:  ish hindex
 +
  Create index file for an htar file:  ish hindex data.tar
 +
  Create htar and index file:          ish htar -cpf data.tar data/
 +
 
 +
 
 +
==Ish commands==
 +
 
 +
===help - ish command to show help on ish commands===
  
 
Usage:
 
Usage:
 +
    help [COMMAND]
  
−
ish -h|--help                        -  Show this help
+
If no COMMAND is given, a list of all ish commands, with a brief
−
ish --version                        -  Show version number
+
description, is given.
−
ish [INDEXFILE]                      -  Interactive shell for INDEXFILE
 
−
ish [INDEXFILE] COMMAND              -  Perform single COMMAND on INDEXFILE
 
  
−
Shell commands:
+
===ls - ish command to list directory contents===
  
−
  help                                  - Show available commands
+
Usage:
−
avail                                -  List available index files
+
    ls [OPTION] [ PATTERN [PATTERN ... ] ]
−
use          INDEXFILE                -  Use INDEXFILE as index
+
 
−
ls    [-lr]  [DIR/[FILES]] ...       -  List FILES from DIR in index
+
Lists files in the index according to one or more patterns. In
−
  cd           DIR                      -  Set current directory
+
interactive mode, the files are displayed in colour. The list is
−
pwd                                  - Show current directory
+
sorted alphabetically by name.
−
lls/lcd/lpwd                          -  Local ls, cd and pwd
+
 
−
find         PATTERN ...             -  Find files following PATTERN
+
Patterns are of the form [PATH[/]][FILES] and may contain wildcards *
−
colour       1|0                     -  Set colour usage
+
and ?.  Without FILES, the form PATH only lists the directory name,
−
  index       DIR|TARFILE              - Make index for local folder/tarfile
+
while PATH/ lists the files in the directory.
−
tar   [-zcf] TARFILE DIR[/FILES] ...  -  Tar and make index
+
 
−
  settings                              -  List settings (colour, register...)
+
When no patterns are given, ls lists files in the current directory in
−
  register     DIR                      - Set temporary index file location
+
the current index file (as given on the command line or set with use).
−
                                          ($ISHREGISTER specifies the default)
+
 
 +
Paths are relative to the current directory (as set with 'cd'). To
 +
specify a path from the root of the index, an initial colon (:)
 +
should be put in front of the path. If the index contains absolute
 +
paths, the root can be indicated by an initial slash (/) as well.
 +
 
 +
The optional argument can be:
 +
 
 +
  -l list in long format, displaying information on file sizes, modification times, and other metadata present in the index.
 +
  -r lists recursively into sub-directories.
 +
 
 +
To give both options, they have to be combined into one, e.g. -lr.
 +
 
 +
===cd - ish command to set current directory in index===
 +
 
 +
Usage:
 +
    cd [PATH]
 +
 
 +
Change the current directory within the index to the directory
 +
specified by PATH.
 +
 
 +
The new path is relative to the previous current directory, unless
 +
preceded by a colon (:), in which case it specifies a path from the
 +
root of the index. If the index contains absolute paths, the root can
 +
be indicated by an initial slash (/) as well.
 +
 
 +
If no PATH is given, the current directory is set to the root
 +
directory of the index file.
 +
 
 +
The current directory is always displayed in the ish prompt.
 +
 
 +
===find - ish command to recursively find files in index===
 +
 
 +
Usage:
 +
    find PATTERN ...
 +
 
 +
Recursively searches the directory tree in the index from the current
 +
directory (set by 'cd') for files following one or more patterns.
 +
PATTERN may be of the form PATH/FILE or just FILE, and can contain the
 +
wildcards * and ?.
 +
 
 +
Known bug: Currently, find returns more matches than is should, i.e.,
 +
'myish' is found when finding 'ish'.
 +
 
 +
===exit - ish command to exit the shell===
 +
 
 +
Usage:
 +
    exit
 +
 
 +
Ish will exit with the exit status of the last command that was run.
 +
 
 +
When input is redirected to be read from a file or here-document, the
 +
end-of-file also exits ish, so in that case exit is optional.
 +
 
 +
===index - ish command to make an index for a directory or tarfile===
 +
 
 +
Usage:
 +
    index PATH|TARFILE
 +
 
 +
Makes an index file of a directory (PATH) or tar file (TARFILE). The
 +
PATH, or the paths stored in the tar file, can be either absolute or
 +
relative (not both). The tar file can be in compressed format,
 +
provided the local tar installation supports it.
 +
 
 +
The index files are stored in the register directory (~/.ish_register
 +
by default). See 'register' for details.
 +
 
 +
For consistency, if the parent directory of any directory in PATH or
 +
the TARFILE is not in that PATH or in that TARFILE, it gets added to
 +
the index as a stub (i.e., not with its full content).  For example,
 +
'index /home/rzon' will include /home as a stub, which will seem to
 +
only contain the directory rzon. Similarly, if test.tar was created
 +
(not from within ish) with 'tar cf test.tar /home/rzon', then 'index
 +
test.tar' will add a stub for /home.
 +
 
 +
Ish automatically assigns a name to the index file, following this
 +
naming convention:
 +
 
 +
* A directory index stored in the current directory will be called PATH.igz, with any slashes replaced by underscores.
 +
* A directory index stored in the registry will be called ABSPATH.igz, where ABSPATH is the absolute path to the directory PATH, again, with slashes replaced by underscores.
 +
* A tar index will be called TARFILE.igz, with slashes replaced with underscores.
 +
* When running an index command again, or running an index command that would result in the name of an existing index that would be overwritten, the existing index file NAME.igz gets renamed to NAME_DATE_TIME.igz, where date is the date of the existing index file.<br>The exception to this rule is that if the new index file has identical content to the old one, the old one is removed.
 +
 
 +
Once the indexing is done, the index file becomes the current one in the ish shell.
 +
 
 +
The index file for a tar file can be created when the tar is made,
 +
using the ish-command tar (see 'tar').
 +
 
 +
Currently, the index of a local directory does not contain metadata
 +
such as file sizes and modification time.
 +
 
 +
===colour - ish command to set colour usage===
 +
 
 +
Usage:
 +
    colour ARG
 +
 
 +
Switches the usage of colour in listings on if ARG=1 and off if ARG=0.
 +
By default, colour is on in interactive sessions, and off in single
 +
command mode.
 +
 
 +
The current colour setting can be found with the 'settings' command.
 +
 
 +
===pwd - ish command to show current directory===
 +
 
 +
Usage:
 +
    pwd
 +
 
 +
Shows the current directory within the index (as set by 'cd'). 
 +
 
 +
This command should be rarely needed, as the current directory is
 +
always displayed in the ish prompt.
 +
 
 +
===register - ish command to set new index file location===
 +
 
 +
Usage:
 +
    register [DIR]
 +
 
 +
Sets the location where index files are stored.  This location is
 +
called the register. DIR must be an existing directory on the file
 +
system. If DIR is omitted, the register is set to the default
 +
location.
 +
 
 +
The default location of the register is the (hidden) directory
 +
~/.ish_register, or is set by the environment variable ISHREGISTER.
 +
 
 +
Index files in the current directory will be found by 'use' as well,
 +
regardless of the register setting. But to store an index file in the
 +
current directory (using 'index' or 'hindex'), the register has to be
 +
set to '.' first.
 +
 
 +
The current setting of the register can be found with 'settings'.
 +
 
 +
===settings - ish command to list settings===
 +
 
 +
Usage:
 +
    settings
 +
 
 +
Lists settings such as colour and register location.
 +
 
 +
===tar - ish command to tar and make index===
 +
 
 +
Usage:
 +
    tar [OPTION] TARFILE [DIR][FILES] ...
 +
 
 +
Tars a directory or files list as usual, AND creates the corresponding
 +
index file. It should be equivalent to running the same tar command
 +
from the shell (i.e. not within ish), followed by "ish index TARFILE".
 +
 
 +
Common tar options:
 +
 
 +
  -c create the file. This option is implied in the ish version, and
 +
      may be omitted.
 +
  -f should be the last option, and indicates that output should go to
 +
      the tarfile. Also implied and thus optional in the ish version.
 +
  -z  gzip the tarball.
 +
  -j  bzip2 the tarball.
 +
 
 +
To give multiple options, they have to be combined into one, e.g. -czf.
 +
If given, -f must be the last option. Other options can be found from
 +
the tar man page.
 +
 
 +
===avail - ish command to list available index files===
 +
 
 +
Usage:
 +
    avail [-a]
 +
 
 +
Displays a list of index files the register directory (see 'register')
 +
or the current directory. These index files can be accessed with the
 +
'use' command.
 +
 
 +
By default, older versions of the igz files (as automatically saved by
 +
index and hindex) are not shown.
 +
 
 +
The optional argument can be:
 +
 
 +
  -a When given, older saved version of the igz files are also listed.
 +
 
 +
See 'index' and 'hindex' on how to create index files, and on their
 +
naming scheme.
 +
 
 +
===use - ish command to set the index file===
 +
 
 +
Usage:
 +
    use INDEX
 +
 
 +
Sets the current index file to INDEX. This file should reside in the
 +
register directory (see 'register') or the current directory. INDEX
 +
should include the index file extension '.igz'.
 +
 
 +
Note that the index file can also be set when starting ish with the
 +
name of the index file as the first argument (e.g. ish data.tgz.igz).
 +
 
 +
See 'index' and 'hindex' on how to create index files, and on their
 +
naming scheme.
 +
 
 +
The current index file is always displayed in the ish prompt.
 +
 
 +
===!COMMAND - ish command to local commands (ls, cd, pwd)===
 +
 
 +
Usage:
 +
    !COMMAND [ARGS]
 +
 
 +
Runs a local command. This is intended to allow the local directory to
 +
be changed and listed. Any arguments will be passed to the command.
 +
 
 +
Only the commands ls, cd and pwd will be accepted to run locally.
 +
 
 +
===hindex - ish command to make index for hpss folder/tarfile (hpss only)===
 +
 
 +
Usage:
 +
    hindex [PATH|TARFILE]
 +
 
 +
Makes an index file of a directory (PATH) or tar file (TARFILE) on hpss.
 +
The distinction is made on whether the argument has the extension .tar.
 +
 
 +
The PATH, or the paths stored in the tar file, can be either absolute
 +
or relative (not both). The tar file cannot be in compressed format,
 +
and ideally should have been created with htar.
 +
 
 +
If neither a PATH nor a TARFILE is given, hindex will generate an
 +
index for your whole hpss directory, and store the index in hpss.igz.
 +
 
 +
The index files are stored in the register directory (~/.ish_register
 +
by default). See 'register' for details.
 +
 
 +
For consistency, if the parent directory of any directory in PATH or
 +
the TARFILE is not in that PATH or in that TARFILE, it gets added to
 +
the index as a stub (i.e., not with its full content).  For example,
 +
'hindex /archive/scinet/rzon' will include /archive and
 +
/archive/scinet as a stubs, which will seem to only contain one
 +
directory. Similarly, if test.tar was created (not from within ish)
 +
with 'htar cf test.tar /home/rzon', then 'hindex test.tar' will add a
 +
stub for /home.
 +
 
 +
Ish automatically assigns a name to the index file, following this
 +
naming convention:
 +
 
 +
* The directory index will be called PATH.igz, with any slashes replaced by underscores.
 +
* A tar index will be called TARFILE.igz, with slashes replaced with underscores.
 +
* The index will be called hpss.igz when hindex is called without argument.
 +
* When running an hindex command again, or running an hindex command that would result in an existing index getting overwritten, the existing index file NAME.igz gets renamed to NAME_DATE_TIME.igz, where DATE and TIME are of the existing index file. The exception to this rule is that if the new index file has identical content to the old one, the old one is removed.
 +
 
 +
Ish calls the hsi application under the hood to get the directory
 +
listings. This means that this command has to be run on a system that
 +
has hpss with hsi. This may mean you have to submit a job for building
 +
the index. With the index file, you can then locally traverse the
 +
listings in the index and see the modification times and file sizes.
 +
 
 +
If the tar file was created with htar (outside of ish), then the index
 +
is created from the remote index file generated by a call to htar. So
 +
this command has to be run on a system that has hpss with htar.
 +
 
 +
If the hpss-resident tar file was created with tar (outside of ish),
 +
then no remote index file exists and ish will first request htar to
 +
generate one. Because this may require the tar to be reclaimed from
 +
tape, this is an expensive operation. So it is recommended to 1) use
 +
htar, or, 2) at the point of tarring, use ish's tar command, or, 3) if
 +
the tar is already created and still available locally, to use the ish
 +
command index.
 +
 
 +
Once the indexing is done, the index file becomes the current one in
 +
the ish shell.
 +
 
 +
Note that the htarring and the creation of a local index file can be
 +
done with one command, using the 'htar' ish command.
 +
 
 +
===htar - ish command to htar and make index (hpss only)===
 +
 
 +
Usage:
 +
    htar [OPTIONS] TARFILE [DIR][FILES] ...
  
−
Archive-queue-only ish shell commands:
+
Htars a directory or set of files list as usual, AND creates the
 +
corresponding local index file. It should be equivalent to running the
 +
same htar command from the shell (i.e. not from within ish), followed
 +
by "ish hindex TARFILE".
  
−
hindex      DIR|TARFILE              -  Make index for hpss folder/tarfile
+
Common htar options:
−
htar [-cf]  TARFILE DIR[/FILES] ...  -  Htar and make index
 
  
−
Command line examples:
+
  -c  create the file. This option is implied in the ish version, and
 +
      may be omitted.
 +
  -f  should be the last option, and indicates that output should go to
 +
      the tarfile. Also implied and thus optional in the ish version.
 +
  -p  Preserve time stamps etc.
  
−
Local tar and gzip directory 'code':  ish tar -z code.tgz code
+
To give multiple options, they have to be combined into one, e.g. -cpf.
−
Long list of its content:              ish code.tgz.igz ls -l 'code/*'
+
If given, -f must be the last option. Other options can be found from
−
Make index file for existing tar:      ish index another.tar
+
the htar man page.
−
List its top level content:            ish another.tar.igz ls
 
−
List all of its content:              ish another.tar.igz ls -r
 
−
Find file 'hello.txt' in its content:  ish another.tar.igz find hello.txt
 
−
Make index file hsi.igz for hpss tree: ish hindex
 
−
Create index file for an htar file:    ish hindex data.tar
 

Revision as of 14:26, 30 June 2011

ish - inventory shell

Introduction

Browse directories or tarball content from local inventory files.

ish is located in /scinet/gpc/bin/ish.

Current version is 0.9 (a beta version).

Usage

From the command line

   ish -h|--help                      show this help
   ish --version                      show version number
   ish [INDEX]                        interactive shell for file INDEX
   ish [INDEX] COMMAND                perform single COMMAND on file INDEX

Shell Commands

Most common 

   ls [-lr]   [DIR/[FILES]] ...       list FILES from DIR in index
   cd         DIR                     set current directory in index
   find       PATTERN ...             find files following PATTERN in index
   index      DIR|TARFILE             make index for folder/tarfile
   exit                               exit the shell

More 

   avail      [-a]                    list (all) available index files
   colour     1|0                     set colour usage
   help       [COMMAND]               show help on (all) commands
   pwd                                show current directory
   register   [DIR]                   set new index file location 
   settings                           list settings 
   tar [-zcf] TARFILE DIR[/FILES] ... tar and make index
   use        [INDEX]                 use INDEX or list available ones
   !COMMAND   [ARGS]                  local commands (ls, cd, pwd)

Only in archive queue (hpss) 

   hindex     DIR|TARFILE             make index for hpss folder/tarfile
   htar [-cf] TARFILE DIR[/FILES] ... htar and make index

Command line examples

 Local tar and gzip directory 'code': ish tar -z code.tgz code
 Long list of its content:            ish code.tgz.igz ls -l 'code/*'
 Make index file for existing tar:    ish index another.tar
 List its top level content:          ish another.tar.igz ls
 List all of its content:             ish another.tar.igz ls -r
 Find file 'hello.txt' in it:         ish another.tar.igz find hello.txt
 Make index hspp.igz for hpss tree:   ish hindex
 Create index file for an htar file:  ish hindex data.tar
 Create htar and index file:          ish htar -cpf data.tar data/


Ish commands

help - ish command to show help on ish commands

Usage: 

   help [COMMAND]

If no COMMAND is given, a list of all ish commands, with a brief description, is given.

ls - ish command to list directory contents

Usage: 

   ls [OPTION] [ PATTERN [PATTERN ... ] ]

Lists files in the index according to one or more patterns. In interactive mode, the files are displayed in colour. The list is sorted alphabetically by name.

Patterns are of the form [PATH[/]][FILES] and may contain wildcards * and ?. Without FILES, the form PATH only lists the directory name, while PATH/ lists the files in the directory.

When no patterns are given, ls lists files in the current directory in the current index file (as given on the command line or set with use).

Paths are relative to the current directory (as set with 'cd'). To specify a path from the root of the index, an initial colon (:) should be put in front of the path. If the index contains absolute paths, the root can be indicated by an initial slash (/) as well.

The optional argument can be: 

 -l  list in long format, displaying information on file sizes, modification times, and other metadata present in the index.
 -r  lists recursively into sub-directories.

To give both options, they have to be combined into one, e.g. -lr.

cd - ish command to set current directory in index

Usage: 

   cd [PATH]

Change the current directory within the index to the directory specified by PATH.

The new path is relative to the previous current directory, unless preceded by a colon (:), in which case it specifies a path from the root of the index. If the index contains absolute paths, the root can be indicated by an initial slash (/) as well.

If no PATH is given, the current directory is set to the root directory of the index file.

The current directory is always displayed in the ish prompt.

find - ish command to recursively find files in index

Usage: 

   find PATTERN ...

Recursively searches the directory tree in the index from the current directory (set by 'cd') for files following one or more patterns. PATTERN may be of the form PATH/FILE or just FILE, and can contain the wildcards * and ?.

Known bug: Currently, find returns more matches than is should, i.e., 'myish' is found when finding 'ish'.

exit - ish command to exit the shell

Usage: 

   exit

Ish will exit with the exit status of the last command that was run.

When input is redirected to be read from a file or here-document, the end-of-file also exits ish, so in that case exit is optional.

index - ish command to make an index for a directory or tarfile

Usage: 

   index PATH|TARFILE

Makes an index file of a directory (PATH) or tar file (TARFILE). The PATH, or the paths stored in the tar file, can be either absolute or relative (not both). The tar file can be in compressed format, provided the local tar installation supports it.

The index files are stored in the register directory (~/.ish_register by default). See 'register' for details.

For consistency, if the parent directory of any directory in PATH or the TARFILE is not in that PATH or in that TARFILE, it gets added to the index as a stub (i.e., not with its full content). For example, 'index /home/rzon' will include /home as a stub, which will seem to only contain the directory rzon. Similarly, if test.tar was created (not from within ish) with 'tar cf test.tar /home/rzon', then 'index test.tar' will add a stub for /home.

Ish automatically assigns a name to the index file, following this naming convention:

  • A directory index stored in the current directory will be called PATH.igz, with any slashes replaced by underscores.
  • A directory index stored in the registry will be called ABSPATH.igz, where ABSPATH is the absolute path to the directory PATH, again, with slashes replaced by underscores.
  • A tar index will be called TARFILE.igz, with slashes replaced with underscores.
  • When running an index command again, or running an index command that would result in the name of an existing index that would be overwritten, the existing index file NAME.igz gets renamed to NAME_DATE_TIME.igz, where date is the date of the existing index file.
    The exception to this rule is that if the new index file has identical content to the old one, the old one is removed.

Once the indexing is done, the index file becomes the current one in the ish shell.

The index file for a tar file can be created when the tar is made, using the ish-command tar (see 'tar').

Currently, the index of a local directory does not contain metadata such as file sizes and modification time.

colour - ish command to set colour usage

Usage: 

   colour ARG

Switches the usage of colour in listings on if ARG=1 and off if ARG=0. By default, colour is on in interactive sessions, and off in single command mode.

The current colour setting can be found with the 'settings' command.

pwd - ish command to show current directory

Usage: 

   pwd

Shows the current directory within the index (as set by 'cd').

This command should be rarely needed, as the current directory is always displayed in the ish prompt.

register - ish command to set new index file location

Usage: 

   register [DIR]

Sets the location where index files are stored. This location is called the register. DIR must be an existing directory on the file system. If DIR is omitted, the register is set to the default location.

The default location of the register is the (hidden) directory ~/.ish_register, or is set by the environment variable ISHREGISTER.

Index files in the current directory will be found by 'use' as well, regardless of the register setting. But to store an index file in the current directory (using 'index' or 'hindex'), the register has to be set to '.' first.

The current setting of the register can be found with 'settings'.

settings - ish command to list settings

Usage: 

   settings

Lists settings such as colour and register location.

tar - ish command to tar and make index

Usage: 

   tar [OPTION] TARFILE [DIR][FILES] ...

Tars a directory or files list as usual, AND creates the corresponding index file. It should be equivalent to running the same tar command from the shell (i.e. not within ish), followed by "ish index TARFILE".

Common tar options: 

 -c  create the file. This option is implied in the ish version, and
     may be omitted.
 -f  should be the last option, and indicates that output should go to
     the tarfile. Also implied and thus optional in the ish version.
 -z  gzip the tarball. 
 -j  bzip2 the tarball.

To give multiple options, they have to be combined into one, e.g. -czf. If given, -f must be the last option. Other options can be found from the tar man page.

avail - ish command to list available index files

Usage: 

   avail [-a]

Displays a list of index files the register directory (see 'register') or the current directory. These index files can be accessed with the 'use' command.

By default, older versions of the igz files (as automatically saved by index and hindex) are not shown.

The optional argument can be: 

 -a  When given, older saved version of the igz files are also listed.

See 'index' and 'hindex' on how to create index files, and on their naming scheme.

use - ish command to set the index file

Usage: 

   use INDEX

Sets the current index file to INDEX. This file should reside in the register directory (see 'register') or the current directory. INDEX should include the index file extension '.igz'.

Note that the index file can also be set when starting ish with the name of the index file as the first argument (e.g. ish data.tgz.igz).

See 'index' and 'hindex' on how to create index files, and on their naming scheme.

The current index file is always displayed in the ish prompt.

!COMMAND - ish command to local commands (ls, cd, pwd)

Usage: 

   !COMMAND [ARGS]

Runs a local command. This is intended to allow the local directory to be changed and listed. Any arguments will be passed to the command.

Only the commands ls, cd and pwd will be accepted to run locally.

hindex - ish command to make index for hpss folder/tarfile (hpss only)

Usage: 

   hindex [PATH|TARFILE]

Makes an index file of a directory (PATH) or tar file (TARFILE) on hpss. The distinction is made on whether the argument has the extension .tar.

The PATH, or the paths stored in the tar file, can be either absolute or relative (not both). The tar file cannot be in compressed format, and ideally should have been created with htar.

If neither a PATH nor a TARFILE is given, hindex will generate an index for your whole hpss directory, and store the index in hpss.igz.

The index files are stored in the register directory (~/.ish_register by default). See 'register' for details.

For consistency, if the parent directory of any directory in PATH or the TARFILE is not in that PATH or in that TARFILE, it gets added to the index as a stub (i.e., not with its full content). For example, 'hindex /archive/scinet/rzon' will include /archive and /archive/scinet as a stubs, which will seem to only contain one directory. Similarly, if test.tar was created (not from within ish) with 'htar cf test.tar /home/rzon', then 'hindex test.tar' will add a stub for /home.

Ish automatically assigns a name to the index file, following this naming convention:

  • The directory index will be called PATH.igz, with any slashes replaced by underscores.
  • A tar index will be called TARFILE.igz, with slashes replaced with underscores.
  • The index will be called hpss.igz when hindex is called without argument.
  • When running an hindex command again, or running an hindex command that would result in an existing index getting overwritten, the existing index file NAME.igz gets renamed to NAME_DATE_TIME.igz, where DATE and TIME are of the existing index file. The exception to this rule is that if the new index file has identical content to the old one, the old one is removed.

Ish calls the hsi application under the hood to get the directory listings. This means that this command has to be run on a system that has hpss with hsi. This may mean you have to submit a job for building the index. With the index file, you can then locally traverse the listings in the index and see the modification times and file sizes.

If the tar file was created with htar (outside of ish), then the index is created from the remote index file generated by a call to htar. So this command has to be run on a system that has hpss with htar.

If the hpss-resident tar file was created with tar (outside of ish), then no remote index file exists and ish will first request htar to generate one. Because this may require the tar to be reclaimed from tape, this is an expensive operation. So it is recommended to 1) use htar, or, 2) at the point of tarring, use ish's tar command, or, 3) if the tar is already created and still available locally, to use the ish command index.

Once the indexing is done, the index file becomes the current one in the ish shell.

Note that the htarring and the creation of a local index file can be done with one command, using the 'htar' ish command.

htar - ish command to htar and make index (hpss only)

Usage: 

   htar [OPTIONS] TARFILE [DIR][FILES] ...

Htars a directory or set of files list as usual, AND creates the corresponding local index file. It should be equivalent to running the same htar command from the shell (i.e. not from within ish), followed by "ish hindex TARFILE".

Common htar options: 

 -c  create the file. This option is implied in the ish version, and
     may be omitted.
 -f  should be the last option, and indicates that output should go to
     the tarfile. Also implied and thus optional in the ish version.
 -p  Preserve time stamps etc.

To give multiple options, they have to be combined into one, e.g. -cpf. If given, -f must be the last option. Other options can be found from the htar man page.

Retrieved from ‘https://oldwiki.scinet.utoronto.ca/index.php?title=ISH&oldid=3550’