_das2, libdas2 Python bindings

Functions in this module provide bindings to the C libdas2 library. Most of the small server side programs used to reduce streams in time, compute power spectral densities, read cache blocks are based off of libdas2. Whenever possible das2py wraps libdas2 functionality instead of creating a parallel code base.

Reading Catalog Files

Das2 catalog data consists of nodes defined in JSON and XML. At present only JSON nodes are parsable by libdas2, though support for SPASE XML records may be added in funture versions. The _das2 module converts catalog data to python dictionaries. Higher level pure python code parses these dictionaries to create catalog objects such as das2.Collection and das2.Catalog.

_das2.get_node()

Get JSON data from a das2 Catalog Node by URL or URI.

Arguments
path_uri (string) - The URI of the node to load. By default the federated
catalog system is used to provide the URL to the catalog node file.
agent (string, optional) - If present (and not None) then the supplied text
will be the User Agent string supplied to any remote HTTP servers that are contacted.
url (string, optional) - If present (and not None) then the node is loaded
directly from the URL provided, potentially skipping the global catalog, this is useful for testing stand alone catalog files

Reading Das2 Streams

The C library handles reading parsing stream data into internal arrays that are passed to NumPy via pointer assignment (not a copy). At present only stream in das2/2.2 format are supported, but other formats such as the HAPI stream format will be supported as time permits.

Low-level Dataset Output

The main data reading functions:

all return the same, rather complex, output. Each function returns a python list containing correlated datasets. i.e.:

[ dataset_dict_1, dataset_dict_2, ... ]

The structure and meaning of each dataset is contained in a python dictionary with the following structure:

{
   'id':str,      # An identifier token usable as a C style variable name, no
                  # spaces or special charaters allowed.

   'group':str,   # Datasets occupying same kind and number of physical
                  # dimensions can be part of the same group and should
                  # usually be plotted together.  This is the group or
                  # 'join' id.

   'rank':int,    # Overall number of iteration dimensions for the dataset
                  # this is it's size in index space (not physical space)

   'shape':list,  # A list that is *rank* values long.  Provides the overall
                  # size of the dataset with one value for each iteration
                  # dimension.

   'props':dict,  # A dictionary of (type, value) 2-tuples providing any
                  # properties set on this dataset.

   'coords':dict, # A dictionary of physical coordinate dimension objects,
                  # (described below).

   'data':dict,   # A dictionary of physical data dimension objects,
                  # (described below).

   'arrays':dict  # A dictionary of all the backing ndarrays for this dataset.
}

Each element of the ‘coords’ and ‘data’ dicts are also dictionaries. Each one of these define a single physical dimension. The dimension dictionaries contains the following keys and items:

{
   'id':str,     # An identifier token usable as a C style variable name,
                 # no spaces or special charaters allowed.

   'type':str,   # The string 'COORD_DIM' or 'DATA_DIM'

   'props':str,  # A dictionary of (type, value) 2-tuples providing any
                 # properties set on this dimension.

   role:dict     # Here *role* is the name of a variable.  There are 1-N
                 # variables per dimension, each is named by it's role in
                 # describing values for a given dimension.  Example roles
                 # are "center", "offset", "reference", "minimum", etc.
}

Each variable in a dimension is also defined by a dictionary with the following keys and items:

{
   'role':str,   # A repeat of this variable's role (i.e. 'center', 'min' etc)

   'units':str,  # The units string.  Note that date-time values are ususally
                 # in non-physical units such as t1970, which is the number
                 # of seconds since midnight, Jan. 1st 1970 ignoring leap
                 # seconds.

   'expression':str,  # A summary of how to get values for this variable out
                      # of the ndarrays.  Used by higher level code to setup
                      # accessor functions and handle array broadcasts.
}

The upper level das2 module converts this low level output into das2.Dataset, das2.Dimension, and das2.Variable objects that are easier to work with since all the array indices have been broadcast to a uniform space.

_das2.read_cmd(sCmd)

Reads a Das2 stream from an external program and returns a list of dictionaries that describe dataset and hold the NumPy arrays containing the data.

Note

This function releases the global interpreter lock during data download

Parameters:sCmd (str) – The reader command line to run. Standard output from the command is expected to be a das2 stream.
Returns:A list of datasets, as defined in the section Low-level Dataset Output
_das2.read_file()

Reads a Das2 stream from a disk file and returns a list of DasDs (das dataset) objects containing the data in the stream

Thread Note: This function releases the global interpreter lock during stream
reading
Parameters:sFile (str) – The filename to read
Returns:
A list of correlated datasets. Each correlated dataset is a dictionary
with the following keys and items:
  • ‘rank’ - The number of array dimensions in each dataset
  • ‘id’ - A string containing an identifier token usable as a variable name
  • ‘groupId’ - A string containing the join group for this Correlated dataset
  • ‘shape’ - An array containing the maximum index value in each dimension
  • ‘coords’ - A list of coordinate dictionaries (defined below)
  • ‘datasets’ - A list of datasets correlated in the given coordinates (see below)
  • ‘arrays’ - A dictionary of all the backing ndarrays for the dataset (see below)
  • ‘properties’ - A list of 3-tuples providing string properties for the dataste

Each coordinate is defined by a dictionary with the following keys and items

  • ‘id’ - A string contanining an identifier token usable as a variable name
  • ‘units’ - A string containing the units of the coordinate
  • ‘array’ - A string containing the id if the backing array
  • ‘idxmap’ - A list of integers that is ‘rank’ long each entry given the index number in the backing array that corresponds to a dataset index (this can be abstract, see examples)
  • ‘properties’ - A list of 3-tuples providing string properiets for the coordinate

Each dataset is defined by a dictionary with the same keys as the coordinates

The array dictionary is a mapping of array ID names to backing ndarrays

_das2.read_server(sUrl, sAgent=None)

Reads a Das2 stream from a remote HTTP/HTTPS server.

Note

This function releases the global interpreter lock during data download

Parameters:
  • sUrl (str) – The URL to read, can be an extensive GET string
  • rConSec (float, optional) – How long to wait on the connection to the remote server in seconds. A value of <= 0.0 means wait as long as the operating system allows. sAgent (str,optional) : The user agent string you’d like to use
Returns:

A list of datasets. As defined in the section Low-level Dataset Output

Server Authentication

_das2.auth_set()

Set an authentication hash to be sent to remote servers when certain conditions are met.

The request must come from a particular server, for a particular realm, and for a particular dataset. The authentication hash is not transmitted unless the server asks for it and the request matches the given conditions.

Parameters:
  • base_url (str) – The full URL path to the das2 server, which is not typically the same as the host root, example: https://zeus.physics.uiowa.edu/das/server
  • realm (str) – The authentication realm. This is provided in the dsdf files under the securityRealm keyword.
  • hash (str) – The hash to send. Most servers, including das2 pyServer are looking for an HTTP Basic Authentication hash
  • dataset (str,optional) – The dataset, ex: ‘Juno/WAV/Survey’ Use None to match any dataset in this Realm. Some sites will not provide this information in which case None should be used to match request from those sites.
HTTP Basic Auth Hash Generation
Make the string ‘USERNAME:PASSWORD’ where ‘:’ is a literal colon and encode it using the base64 algorithm. The standard_b64encode() function from the python base64 module can be used to perform this task.

Power Spectral Density

class _das2.Dft

An amplitude preserving Discrete Fourier Transform converter __init__(nLen, sWindow)

Create a new DFT calculator

nLen The length of the data vectors that will be supplied
to the calculate function
sWindow A named window to apply to the data. If None then
no window will be used. Accepted values are [‘HANN’, None]
class _das2.Psd

Create a new Power Spectral Density calculator.

This estimator uses the equations given in Numerical Recipes in C, section 13.4, but not any of the actual Numerical Recipes source code.

__init__(nLen, bCenter, sWindow)

Create a new DFT calculator

nLen The length of the data vectors that will be supplied
to the calculate function
bCenter If true, input values will be centered on the Mean value.
This shifts-out the DC component from the input
sWindow A named window to apply to the data. If None then
no window will be used. Accepted values are [‘HANN’, None]

Time Handling

_das2.parsetime(sDateTime)

Converts most human-parseable time strings to numeric components.

This function has no concept of leap seconds. So the maximum number of seconds in a minute is 60, not 61 or 62.

Parameters:sDateTime (str) – A string to parse to a date
Returns:A 7-tuple containing the broken down time values
  • year (int): The (typically) 4-digit year value
  • month (int): The month of year starting with 1
  • mday (int): The day of the month from 1 to 31
  • yday (int): The day of the year from 1 to 366
  • hour (int): The hour of the day from 0 to 23
  • minute (int): The minute of the hour from 0 to 59
  • seconds (float): The seconds of the minute from 0.0 to < 60.0
Raises ValueError:
 If the time is not parsable
_das2.parse_epoch(rTime, sUnits)

Converts and floating point das2 epoch time into a numeric calendar components.

Parameters:
  • rTime (float) – A floating point time value whose units are and referenece point are determined by sUnits argument
  • sUnits (str) –

    One of the das2 timestamp units types as defined in units.c in the underlying libdas2 C library. Known unit types are:

    • ‘us2000’ : Microseconds since midnight, January 1st 2000
    • ‘mj1958’ : Days since midnight, January 1st 1958
    • ‘t2000’ : Seconds since midnight, January 1st 2000
    • ‘t1970’ : Seconds since midnight, January 1st 1970, commonly called the UNIX epoch.
Returns:

A 7-tuple containing the broken down time values

  • year (int): The (typically) 4-digit year value
  • month (int): The month of year starting with 1
  • mday (int): The day of the month from 1 to 31
  • yday (int): The day of the year from 1 to 366
  • hour (int): The hour of the day from 0 to 23
  • minute (int): The minute of the hour from 0 to 59
  • seconds (float): The seconds of the minute from 0.0 to < 60.0
Raises ValueError:
 

If sUnits is an unknown time value format
_das2.emitt()

Performs the inverse operation as ttime. Converts floating point seconds since the beginning of 1958 back into a broken down time tuple:

(year, month, mday, yday, hour, minute, float_seconds)
_das2.ttime()

Converts time components to a double precision floating point value (seconds since the beginning of 1958, ignoring leap seconds) and normalize inputs. Note that this floating point value should only be used for “internal” purposes. (There’s no need to propagate yet another time system, plus I want to be able to change/fix these values.)

There is no accomodation for calendar adjustments, for example the transition from Julian to Gregorian calendar, so I wouldn’t recommend using this routine for times prior to the 1800’s.

Arguments (will be normalized if necessary):

int year - year (1900 will be added to two-digit values) int month (optional) - month of year (1-12) int mday (optional) - day of month (1-31) int hour (optional) - hour of day (0-23) int minute (optional) - minute of hour (0-59) float second (optional) - second of minute (0.0 <= s < 60.0),

leapseconds ignored

Note: To use day of year as input, simple specify 1 for the month and the day of year in place of day of month. Beware of the normalization.

_das2.tnorm()

Normalizes date and time components Arguments (will be normalized if necessary):

int year - year (1900 will be added to two-digit values) int month (optional) - month of year (1-12) int mday (optional) - day of month (1-31) int hour (optional) - hour of day (0-23) int minute (optional) - minute of hour (0-59) float second (optional) - second of minute (0.0 <= s < 60.0),

leapseconds ignored

Note: To use day of year as input, simple specify 1 for the month and the day of year in place of day of month. Beware of the normalization. Returns a tuple of the form:

(year, month, mday, yday, hour, minute, float_seconds)