$Id: README,v 1.6 2003/12/15 21:36:27 dechavez Exp $

IDA System Interface (ISI) Toolkit, Version 1.0.2

This directory contains software for use by developers interested in
writing custom applications which work with IDA data systems that
support the ISI protocols.  The intent is to provide a simple API for a
reliable continuous or segemented data feed, as well as support for on
demand state of health and configuration queries.

This release has been tested under both sparc and intel Solaris with
GCC and Forte C.  It should build without the need for major
modification with any ANSI C compiler and under any modern Unix and
Linux.  A conscious effort to support Microsoft Windows has also been
made, however no Windows based testing has been done to date and it is
likely that some Unix specific features persist in the current
release.  New releases will be issued as additional platforms are
tested.

Note that this is BETA software.  While every effort has been made to
produce code which actually does what is described above, it is possible
or even likely that you will encounter some errors or problems.  Please
let me know of any know or merely suspected problems as well as any
suggestions you may have for improvements.

Overview
--------
In an effort to keep the API simple, most of the connection parameters
are hidden in an ISI_PARAM structure.  Specifying a NULL value for this
structure will invoke a default set of values which should be
appropriate for most applications.  However, should it be necessary to
override any of these defaults, that is possible via a collection of
functions.  Portable applications should leave the ISI_PARAM structure
opaque and operate on it via the provided functions.

Three basic functions are provided for determining remote status.  Each
of these returns a pointer to an array of appropriate type, or NULL if
there was an error.

ISI_SOH_REPORT *isiSoh(char *server, ISI_PARAM *par);

ISI_CNF_REPORT *isiCnf(char *server, ISI_PARAM *par);

ISI_WFDISC_REPORT *isiWfdisc(char *server, ISI_PARAM *par);

The 3 status report structures have two fields.  One is an array of
report specific type, and the other is the number of entries in that
array.  Specifically they are:

typedef struct {
    UINT32 nentry;
    ISI_STREAM_SOH *entry;
} ISI_SOH_REPORT;

typedef struct {
    UINT32 nentry;
    ISI_STREAM_CNF *entry;
} ISI_CNF_REPORT;

typedef struct {
    UINT32 nentry;
    char **entry;
} ISI_WFDISC_REPORT;

A description of each details of each of the report entry fields is
given in the Datatypes section later in this document.

A data feed is initiated with:

ISI *isiInitiateDataRequest(char *server, ISI_PARAM *par, ISI_DATA_REQUEST *dreq)

where "dreq" specifies return data type, transmission compression, and
which streams, time windows for each stream, for which waveform data are
desired The ISI_DATA_REQUEST structure is described in more detail in the
Data Request section later in this document.

isiInitiateDataRequest() returns a handle that is then used to read the
data frames in a loop of indefinite duration centered around a call to

ISI_GENERIC_TS *isiReadGenericTS(ISI *isi, int *status)

Timeouts and other I/O errors will be handled internally by
isiReadGenericTS() which also takes care of reestablishing the
connection to the data server and updating the data request as required.
This function will return a pointer to a structure that contains a
header describing the data, and pointer to the data Details on the
various fields are given in the Generic Data section later in this
document.

The returned data are guaranteed to be in decompressed host byte order,
unless the raw digitizer packets were requested.  Storage for this
structure resides in the handle, and the calling program must save,
copy or otherwise deal with the data before the next call to
isiReadGenericTS().

isiReadGenericTS() will return NULL when no more data are forthcoming,
and set the value of the status argument to one of the following:

ISI_DONE - when all requested data have been provided
something else - when it is impossible to sustain the connection

The underlying protocol takes care of maintaining the connection via
heartbeats while the server is waiting for new data to come into its
disk loop.

Datatypes
---------
The ISI library makes use of several portable datatypes, and uses these
to build the various data structures referred to above.  Full details
are in isi.h. The fundamental types are as follows:

UINTx - unsigned x-bit integer
INTx  - signed x-bit integer

where x = 8, 16, 32, 64

REAL32 - 32-bit floating point (typically a float)
REAL64 - 64-bit floating point (typically a double)

The library also uses the system "char" type for ASCII characters and
strings.

    ISI_STREAM_NAME
    ---------------
    Stream names are given as ISI_STREAM_NAME structures, containing
    3 fields as null-terminated strings:
    char *sta = station name
    char *chn = channel name
    char *loc = location code

    ISI_SRATE
    ---------
    Sample rate is given by the ISI_SRATE structure which uses the SEED
    convention for specifying sample interval as a combination of factor
    and multiplier parameters:
    INT16 factor = SEED sample rate factor
    INT16 multiplier = SEED sample rate multiplier
    These are used in combination as follows:

    If factor > 0 and multiplier > 0
        then sample rate = factor * multiplier
    If factor > 0 and multiplier < 0
        then sample rate = -1 * factor / multiplier
    If factor < 0 and multiplier > 0
        then sample rate = -1 * multiplier / factor
    If factor < 0 and multiplier < 0
        then sample rate =  multiplier / factor

    The ISI library includes functions to convert to and from ISI_SRATE:

    ISI_SRATE *isiSintToSrate(REAL64 sint, ISI_SRATE *srate);
    Takes the sample interval in seconds, sint, and fills the factor
    and multipler fields of srate.  Returns the srate pointer.

    REAL64 isiSrateToSint(ISI_SRATE *srate);
    Takes the factor and multiplier fields of the structure pointed to
    by srate and returns the equivalent sample interval in seconds.

    ISI_TSTAMP
    ----------
    Time values are given by ISI_TSTAMP structures, which contain two
    fields
    REAL64 value = seconds since 1 January 1970 (aka epoch time) or,
                   depending on context, arbitrary elapsed time interval
    INT16 status = clock status.  The interpretation of this is site
                   dependent and is not covered by this document

    ISI_COORDS
    ----------
    Coordinates are given by an ISI_COORDS structure:
      REAL32 lat = latitude, decimal degrees N+, S-
      REAL32 lat = longitude, decimal degrees E+, W-
     REAL32 elev = surface elevation, meters
    REAL32 depth = depth of burial, meters

    ISI_INST
    --------
    Gross instrument characteristics are given by an ISI_INST structure:
     REAL32 calib = CSS 3.0 calib
    REAL32 calper = CSS 3.0 calper
      REAL32 hang = CSS 3.0 hang
      REAL32 vang = CSS 3.0 vang
       char *type = CSS 3.0 instype

      ISI_RECONNECT_POLICY_MIN_DELAY.  On the other hand, if the
        application is concerned about data completeness then the
        policy should be set to ISI_RECONNECT_POLICY_MIN_GAP.

    ISI_STREAM_REQUEST *stream
        array of stream requests

    UINT32 nstream
        Number of ISI_STREAM_REQUEST structures in the stream array

Constructing the initial IDI_DATA_REQUEST is probably the most tedious
part of developing a custom application.  If the application has its
own disk loop underneath, then it will most likely want to include an
ISI_STREAM_REQUEST entry for each data stream in the disk loop, with
the beg field set according to the data currently available in the
local disk loop.  Given this heavy dependence on application specifics,
this version of the toolkit does not yet provide a simple way to build a
custom data request.  However, if specifying the same start/end times
for all streams is acceptable, then your application could make use of

    ISI_DATA_REQUEST *isiAllocSimpleDataRequest(REAL64 beg, REAL64 end, char *spec);

    beg = start epoch time (or wildcard)
    end = finish epoch time (or wildcard)
    spec = dot delimited sta.chn.loc stream specification, wild cards OK

    For example,

    dreq = isiAllocSimpleDataRequest(ISI_NEWEST, ISI_KEEPUP, "*.*.*")
    
    will generate (via malloc()) a request for a continuous data feed
    of all available streams, starting with the most recent data
    available.

    ISI_DATA_REQUEST *isiFreeDataRequest(ISI_DATA_REQUEST *dreq)

    is used to free a ISI_DATA_REQUEST structure previously allocated
    via isiAllocSimpleDataRequest().  It returns NULL, always.  The
    reason it returns NULL is so that one may have constructs of the
    form:

        dreq = isiFreeDataRequest(dreq);

Generic Data
------------
As described in the overview, a data request is followed by a loop
centered around calls to isiReadGenericTS(), which returns a pointer
to an ISI_GENERIC_TS structure, which consists of two basic parts:

    ISI_GENERIC_TSHDR hdr;
    VOID *data;

The "hdr" field describes the "data" which follow.  A ISI_GENERIC_TSHDR
has includes the following fields

    ISI_STREAM_NAME name - stream name
         ISI_SRATE srate - nominal sample rate
         ISI_TSTAMP tofs - time of first sample in the packet
         ISI_TSTAMP tols - time of last sample in the packet
            UINT32 nsamp - number of samples in the packet
           UINT32 nbytes - number of bytes pointed to by "data"
      ISI_DATA_DESC desc - description of the contents of "data"

The "data" field is a pointer to the payload, whose exact contents
depend upon the hdr.desc field.  As indicated above, the hdr.desc
field is a ISI_DATA_DESC structure which contains

    UINT8 comp - describes compression applied to waveform data, one of
        ISI_COMP_NONE - no compression
         ISI_COMP_IDA - IDA first difference
    Unless explicitly enabled via means not described here, the data
    returned by isiReadGenericTS() will always be uncompressed,
    regardless of whether or not they were compressed for transmission.

    UINT8 type - describes the uncompressed data type, one of
          ISI_TYPE_INT8 - 8-bit integers
         ISI_TYPE_INT16 - 16-bit integers
         ISI_TYPE_INT32 - 32-bit integers
         ISI_TYPE_INT64 - 64-bit integers
        ISI_TYPE_REAL32 - 32-bit floating point
        ISI_TYPE_REAL64 - 64-bit floating point
          ISI_TYPE_IDA8 - raw IDA rev 8 digitizer packet
         ISI_TYPE_IDA10 - raw IDA rev 10 digitzer packet

    UINT8 order - describes byte order of uncompressed data, one of
        ISI_ORDER_LTLENDIAN - little-endian data
        ISI_ORDER_BIGENDIAN - big-endian data

     UINT8 size - size of an uncompressed sample, in bytes

The order and size fields have no meaning when a native type
(ISI_TYPE_IDA8 or ISI_TYPE_IDA10) is specified.  Furthermore, unless
explicilty disabled via means not specified here, uncompressed generic
data will always be in host byte order, regardless of their native
order. For comparison convenience, all native types are guaranteed to
have a value larger than ISI_TYPE_NATIVE.

If one wishes to retain the ISI_GENERIC_TS returned by isiReadGenericTS
(say to compare between successive calls), then you can save a copy
using 

BOOL isiCopyGenericTS(ISI_GENERIC_TS *src, ISI_GENERIC_TS *dest)

which returns TRUE if the copy is successful.  The reason there
is a chance for failure is that the copy function will ensure that
there is sufficient space available in the destination structure to
hold the original data.

In order to ensure that the ISI_GENERIC_TS is properly initialized
you should explicitly create them as needed with

ISI_GENERIC_TS *isiCreateGenericTS(VOID);

which returns a pointer that can be freed with

VOID isiDestroyGenericTS(ISI_GENERIC_TS *ts);

Example
-------
Isi is an example program which exercises all the access features and
is included in this distribution.  To build the library and this
program, do the following:

% cd example
% make clean all

For Lunix you will need to edit the example/Makefile and lib/Makefile and
change OSNAME and possibly SOCKLIBS.  Eventually this distribution will
include Makefiles and project files for Windows builds, but for the moment
you are own your own.  If you can't wait and develop your own support files,
please send them to me and I'll incorporate them into the permanent
distribution tree.

The build process will produce a library, libisi.a, which you can use
to link with your custom application.

The example isi program is fairly useful as a standalone application. 
If you don't specify any arguments then it prints a simple (and
incomplete) help message and exits:

% isi
usage: isi [ -v server=string port=int log=string debug=int ] { data[=spec] cnf soh wd }

The arguments in [ square brackets ] are optional:

           -v - turns on verbose isi output (has no effect on library
                output, should logging be enabled)
server=string - sets the name of the server to the specified string. 
                Currently the only ISI server is idahub.ucsd.edu
     port=int - sets the port number.  This is only if the server is
                not running at the default port 39136. 
   log=string - turns on library logging using string as the "spec"
                option described earlier for isiSetLogging().
    debug=int - sets the debug level, described earlier for
                isiSetDebugFlag()

Specify must one of the arguments in { curly brackets }.  The 3 reports can be
obtained with

	cnf - configuration report
	soh - state of health report
	wd  - disk loop wfdisc dump

Data feeds are done using the "data" argument.  By itself, the data request
defaults to all streams, starting with the most current data and continuing
indefinitely.  The optional "=spec" suffix allows you to select specific 
streams.  The format of the "spec" string is dot delimited "sta.chn.loc".
If you use wild cards, be sure to quote the string so the shell won't get
confused.  You may string together multiple spec strings, delimited by a
plus (sta1.chn.loc+sta2.chn.loc, etc).

	data="*.*.*" is equivalent to simply data w/o any suffix
	data=pfo.bhz.00 requests one specific stream
	data="pfo.*.*+jts.*.*" requests all PFO and JTS streams

In addition, data requests can include beg=TimeString and end=TimeString
arguments which set the start/end time boundaries.  The TimeString has the
form YYYY:DDD-HH:MM:SS.MSC, and where you may truncate from the right
(eg, 2003:317-11 is equivalent to 2003:317-11:00:00.000)

Some specfic examples:

To generate a state of health report:

% isi -v server=idahub.ucsd.edu soh
  Sta Chn Loc  Nseg     Nrec  Oldest Sample Time   Youngest Sample Time     Last Write
 coco bhz  20     2   357608 2003:314-20:18:14.495 2003:321-20:18:17.150 000-00:00:03.000
 coco bhn  20     2   357608 2003:314-20:18:14.495 2003:321-20:18:17.150 000-00:00:03.000
 coco bhe  20     2   357608 2003:314-20:18:14.495 2003:321-20:18:17.150 000-00:00:03.000
...

The -v option turns on the column headers show above.
          Sta Chn Loc - ISI_STREAM_SOH name
                 Nseg - ISI_STREAM_SOH nseg
                 Nrec - ISI_STREAM_SOH nrec
  Oldest Sample Times - ISI_STREAM_SOH tofs.value
Youngest Sample Times - ISI_STREAM_SOH tols.value
           Last Write - ISI_STREAM_SOH tolw.value

To request remote configuration:

isi -v server=idahub.ucsd.edu cnf | head
isi Release 1.3.0
  Sta Chn Loc    Sint    Lat       Long        Elev     Depth     Calib      Calper    Hang    Vang   Inst
 coco bhz  20   0.025  -12.1901   96.8349      3.00     70.00  0.0000e+00 -1.0000e+00 -999.00 -999.00 undef
 coco bhn  20   0.025  -12.1901   96.8349      3.00     70.00  0.0000e+00 -1.0000e+00 -999.00 -999.00 undef
 coco bhe  20   0.025  -12.1901   96.8349      3.00     70.00  0.0000e+00 -1.0000e+00 -999.00 -999.00 undef
...

The -v option turns on the column headers show above.
                Sta Chn Loc - ISI_STREAM_CNF name
                       Sint - ISI_STREAM_CNF srate
         Lat Long Elev Dept - ISI_STREAM_CNF coords *
Calib Calper Hang Vang Inst - ISI_STREAM_CNF inst *

* denotes values which rely on a database at the server and may
or may not be correct.

To determine time span of all data segments, you can request wfdiscs. 
If you have direct of nfs access to the disk loop, then you can use
these wfdiscs as input to other applications (eg, dbpick).

isi server=idahub.ucsd.edu wd > snapshot.wfdisc

To request a continuous data feed, discarding the waveform data and
printing the packet headers:

% isi server=idahub.ucsd.edu data
coco.bhz.20 2003:323-21:11:11.670 0x0001 2003:323-21:11:17.645 0x0001  0.025 240  960 uncomp:int32:nbo
coco.bhn.20 2003:323-21:11:11.670 0x0001 2003:323-21:11:17.645 0x0001  0.025 240  960 uncomp:int32:nbo
coco.bhe.20 2003:323-21:11:11.670 0x0001 2003:323-21:11:17.645 0x0001  0.025 240  960 uncomp:int32:nbo
...

The summary lines describe each received packet and give the sta/chn/loc
stream name, followed by the time of first sample, clock status, time of
last sample, clock status, sample interval, number of samples, number of
bytes, and datum descriptor.  The datum descriptor has three fields
compression:datetype:byteorder.

To bring over native digitizer packets, use the format=native option:

% isi server=idahub.ucsd.edu data format=native
coco.bhz.20 2003:323-21:12:47.675 0x0001 2003:323-21:12:53.650 0x0001  0.025 240 1024 uncomp:ida8
coco.bhn.20 2003:323-21:12:47.675 0x0001 2003:323-21:12:53.650 0x0001  0.025 240 1024 uncomp:ida8
coco.bhe.20 2003:323-21:12:47.675 0x0001 2003:323-21:12:53.650 0x0001  0.025 240 1024 uncomp:ida8

To bring over a specifc set of streams for a specific time window:

% isi server=idahub.ucsd.edu data=coco.bhz.00+pfo.bhz.00 beg=2003:320-11:00 end=2003:320-11:01
isi server=idahub.ucsd.edu data=coco.bhz.00+pfo.bhz.00 beg=2003:320-11:00 end=2003:320-11:01
coco.bhz.00 2003:320-10:59:54.440 0x0001 2003:320-11:00:06.390 0x0001  0.050 240  960 uncomp:int32:nbo
 pfo.bhz.00 2003:320-10:59:53.500 0x060f 2003:320-11:00:05.450 0x060f  0.050 240  960 uncomp:int32:nbo
coco.bhz.00 2003:320-11:00:06.440 0x0001 2003:320-11:00:18.390 0x0001  0.050 240  960 uncomp:int32:nbo
 pfo.bhz.00 2003:320-11:00:05.500 0x060f 2003:320-11:00:17.450 0x060f  0.050 240  960 uncomp:int32:nbo
coco.bhz.00 2003:320-11:00:18.440 0x0001 2003:320-11:00:30.390 0x0001  0.050 240  960 uncomp:int32:nbo
 pfo.bhz.00 2003:320-11:00:17.500 0x060f 2003:320-11:00:29.450 0x060f  0.050 240  960 uncomp:int32:nbo
coco.bhz.00 2003:320-11:00:30.440 0x0001 2003:320-11:00:42.390 0x0001  0.050 240  960 uncomp:int32:nbo
 pfo.bhz.00 2003:320-11:00:29.500 0x060f 2003:320-11:00:41.450 0x060f  0.050 240  960 uncomp:int32:nbo
coco.bhz.00 2003:320-11:00:42.440 0x0001 2003:320-11:00:54.390 0x0001  0.050 240  960 uncomp:int32:nbo
 pfo.bhz.00 2003:320-11:00:41.500 0x060f 2003:320-11:00:53.450 0x060f  0.050 240  960 uncomp:int32:nbo
coco.bhz.00 2003:320-11:00:54.440 0x0001 2003:320-11:01:06.390 0x0001  0.050 240  960 uncomp:int32:nbo
 pfo.bhz.00 2003:320-11:00:53.500 0x060f 2003:320-11:01:05.450 0x060f  0.050 240  960 uncomp:int32:nbo

It is also possible to supress transmission data compression with the
comp=none option, however unless debugging is enabled or you are
snooping the network this will not result in any obvious changes 
since the data are always delivered uncompressed to the application.

Specifying the datdir=path argument will cause the data to be written
to the specified directory, in subdirectories named after each station,
and with the waveform data demultiplexed into files with names of the
form chnloc.  The present release does not include any tools to 
process the generic disk based data, primarily because little thought
has been given on how to store those data in a portable, self-describing
manner.  Such functionality may be added in a future release should
it be determined that there would be value in doing so.

ISI_PARAM set functions
-----------------------
Most application should be able to use the default connection parameters
by providing a NULL value for ISI_PARAM.  However, if default overrides
are desired, they may done using the functions listed below.  The internals
of the ISI_PARAM structure are still a bit fluid, so changes to this list
are likely in the future.

BOOL isiInitDefaultPar(ISI_PARAM *par);

Fills the structure pointed to by par with the default values. 
Returns TRUE if successful, FALSE if not.

ISI_PARAM *isiAllocDefaultParam(VOID);

Creates an ISI_PARAM structure (with malloc) and initializes it to the
default.  Returns a pointer to the newly allocated structure, or NULL
if unsuccessful.  You may use free() to release the memory allocated by
this function.

VOID isiSetServerPort(ISI_PARAM *par, int port);

Sets the server port number to the value given by port.

VOID isiSetTimeout(ISI_PARAM *par, int value);

Sets the I/O timeout interval to the supplied value, in milliseconds.

VOID isiSetTcpBuflen(ISI_PARAM *par, int sndbuf, int rcvbuf);

Sets the TCP/IP send buffer length to sndbuf bytes, and receive buffer
length to rcvbuf bytes.  Use 0 for either of sndbuf or rcvbuf to invoke
the default value determined by the underlying operating system.

VOID isiSetRetryInterval(ISI_PARAM *par, int value);

Set the inverval (in milliseconds) at which failed server connection
attempts will be retried.

VOID isiSetDebugFlag(ISI_PARAM *par, int value);

Set the debug verbosity level to the indicated value.  This flag will
only have effect if logging is enabled.  If so, the a level of 0 or
less will result in only warnings and error messages being generated,
level 1 and 2 will generate progressively more noise related to packets
being sent and received.  This feature still needs much tuning.

BOOL isiSetLogging(ISI_PARAM *par, char *spec, VOID(*func) (char *string), char *pname)

This will enable logging in the underlying library routines.  The
caller should specify _either_ spec or _func_, leaving the other NULL.
If you wish to log to a file, then spec should be a string which
contains the path of the log file. To log to standard output, use the
string "-" or "stdout", to log to standard error, use the string "stderr".
If you are running under Unix, then you can turn on logging via syslod
by giving spec as a string of the form "syslog:FACILITY" where FACILITY
gives the logging facility to use.  See syslog(3c) for a list of supported
facilities.  The facility string is case insensitive, and you can specify
it with or without the LOG_ prefix.  For example, "syslog:LOG_LOCAL0",
"syslog:LOCAL0", and "syslog:local0" are all equivalent.  Alternatively,
you can connect with your own logging system by providing instead of "spec"
a function "func" which takes as its input a null terminated string.  It
is up to you to have "func" do whatever is required with this string. 
The final argument, pname, is a prefix which is added to the message strings.
Typically this is the name of the calling program, but it can be
anything you like.  If you don't want to use this, set pname to be
NULL.

David Chavez
dchavez@ucsd.edu
