DasVar Struct Reference

Das2 fexible variables. More...

#include <das2/variable.h>

Collaboration diagram for DasVar:

Public Member Functions
DAS_API DasVar *	new_DasVarUnary (const char *sOp, const DasVar *pVar)
	Create a new variable from unary operation on an existing variable. More...
DAS_API DasVar *	new_DasVarBinary (const char *sId, DasVar *pLeft, const char *sOp, DasVar *pRight)
	Create a new variable from a binary operation on two other variables. More...
DAS_API DasVar *	new_DasConstant (const char *sId, const das_datum *pDm)
	Create a constant value on the heap. More...
DAS_API DasVar *	new_DasVarSeq (const char *sId, das_val_type vt, size_t vSz, const void *pMin, const void *pInterval, int nExtRank, int8_t *pMap, int nIntRank, das_units units)
	Create a simple linear sequence variable. More...
DAS_API DasVar *	new_DasVarArray (DasAry *pAry, int nExtRank, int8_t *pMap, int nIntRank)
	Create a variable backed by an Array. More...
DAS_API DasVar *	new_DasVarVecAry (DasAry *pAry, int nExtRank, int8_t *pMap, int nIntRank, const char *sFrame, ubyte nFrameId, ubyte frametype, ubyte nDirs, const ubyte *pDir)
	Create a vector backed by an array. More...
DAS_API int	inc_DasVar (DasVar *pThis)
	Increment the reference count on a variable. More...
DAS_API int	dec_DasVar (DasVar *pThis)
	Decrement the reference count on a variable. More...
int	ref_DasVar (const DasVar *pThis)
	Get number of references.
const char *	DasVar_id (const DasVar *pThis)
	Get id token for variable, may be NULL for anoymous vars.
enum var_type	DasVar_type (const DasVar *pThis)
	Get the type of variable.
das_val_type	DasVar_valType (const DasVar *pThis)
	Get the type of values held by the variable.
size_t	DasVar_valSize (const DasVar *pThis)
	Get the size in bytes of each value.
das_units	DasVar_units (const DasVar *pThis)
	Get the units for the values. More...
DAS_API DasAry *	DasVarAry_getArray (DasVar *pThis)
	Get the backing array if present. More...
DAS_API bool	DasVar_orthoginal (const DasVar *pThis, const DasVar *pOther)
	Getting data from a variable. More...
DAS_API int	DasVar_shape (const DasVar *pThis, ptrdiff_t *pShape)
	Return the current shape of this variable. More...
DAS_API int	DasVar_intrShape (const DasVar *pThis, ptrdiff_t *pShape)
	Return the internal composition of this variable. More...
DAS_API ptrdiff_t	DasVar_lengthIn (const DasVar *pThis, int nIdx, ptrdiff_t *pLoc)
	Return the current max value index value + 1 for any partial index. More...
DAS_API char *	DasVar_toStr (const DasVar *pThis, char *sBuf, int nLen)
	Get a string representation of this variable. More...
DAS_API bool	DasVar_isNumeric (const DasVar *pThis)
	Are the values in this variable convertable to doubles? More...
DAS_API bool	DasVar_get (const DasVar *pThis, ptrdiff_t *pIdx, das_datum *pDatum)
	Get a value given an index. More...
DAS_API bool	DasVar_isFill (const DasVar *pThis, const ubyte *pCheck, das_val_type vt)
	Check to see if a value is a fill value for this variable. More...
DAS_API bool	DasVar_isComposite (const DasVar *pVar)
	Is this a simple variable or more than one variable combinded via operators? More...
DAS_API DasAry *	DasVar_subset (const DasVar *pThis, int nRank, const ptrdiff_t *pMin, const ptrdiff_t *pMax)
	Copy a subset of a variable into a memory buffer. More...

Data Fields
bool(*	get )(const struct das_variable *pThis, ptrdiff_t *pIdx, das_datum *pDatum)
	Get a value at a specified index.
int(*	incRef )(struct das_variable *pThis)
	Increment the reference count for this variable and return the new count.
int(*	decRef )(struct das_variable *pThis)
	Returns the number of remaining references to this variable, if the reference count drops to 0, the structure is deleted and the reference count for any owned subvariables or arrays is decremented which may trigger further deletions.
bool(*	degenerate )(const struct das_variable *pThis, int iIndex)
	Returns true if a variable is a function of a given index.
void *	pUser
	User data pointer. More...

Detailed Description

Das2 fexible variables.

Like arrays, das2 variables are objects which produce values given a set of indicies. Unlike arrays the indicies provided need not correspond to the actual layout of the data on the disk.

To illustrate the difference between arrays and das2 variables consider the following arrays containing a typical set of values for time, frequency, amplitude spectrogram.

double time[1440];       // Time values at which frequency sweeps were triggered
double frequency[42];    // Center frequencies for each energy channel
double energy[1440][42]; // Energies measured for each frequency band in each sweep

So to get a correlated triplet of two coordinate values at index (14,34) would look like

x = time[14] 
y = frequency[34]
z = energy[14][34]

In contrast if the time, frequency and energy data were stored in a correlated set of das2 variables accessing the values would look like this:

DasVar* vTime = new_DasVarArry(time, MAP_2(0,DASIDX_UNUSED));
DasVar* vFrequency = new_DasVarArray(frequency, MAP_2(DASIDX_UNUSED, 0));
DasVar* vEnergy = new_DasVarArray(energy, MAP_2(0, 1));
 
// A correlated set is now:
 x = fTime([14,34]);
 y = fFreqency([14,34]);
 z = fEnergy([14,34]);

In addition to wrapping arrays, das2 variables may produce data via calculations involving other variables. For example:

// The premise here is that we are reading in an array of Mars Express 
// spacecraft altitudes.  We know the delay timing for the MARSIS instrument
// and need a way to get the altitude at which the radar return signal was
// generated.  We'll assume that there is nothing to bounce off of near
// the spacecraft except the planet below.  So the altitude at which the
// signal was generated is just the craft altitude minus a range which is
// calculated from the return time.
 
// Create a dynamic array of altitudes.  We don't know how big this will
// get or how large it is any any moment.
Array* pAltAry = new_DasAry("altitude", etDouble, 0, NULL, RANK_1(0));
 
// Create a fixed array of delay times.  There are 80 delay time measurements
// of the MARSIS radar for each altitude.
Array* pDelayAry = new_DasAry("delay", etDouble, 0, NULL, RANK_1(80));
// Fill in array values here
 
// We want an automatic variable that will give us the altitude of the 
// return signals no mater how many altitude values we get.  To do this
// we need to do the calculation:
//
//  Signal_Altitude = Craft_Altitude - (Speed_of_Light/2) * Delay_Time
//
// on any altitude we happen to have.  Since we have a variable number of
// altitudes but a fixed number of delay times it's natural to store these
// values in a 2-index array, namely: 
// 
// Signal_Altitude[i][j] where i marks the altitude and j marks the delay.
//
// So we are going to use the MAP_2 macro to map real array indices into
// virtual indexes.
 
// Map index 0 pAlt to index 0 of a Rank 2 index space
DasVar* pAlt = new_DasVar_array(pAltAry, MAP_2(0, DASIDX_UNUSED), Units_fromStr("km"));
 
// Map index 0 of pDelay to index 1 of a Rank 2 index space
DasVar* pDelay = new_DasVar_array(pDelayAry, MAP_2(DASIDX_UNUSED, 0), "μs");
 
// We need a constant. (Memory note, Variables copy the contents of the
// constant internally so it's okay to initialize constants with stack
// variables.)
int nConst = 299792 / 2;
DasVar* pConst = new_DasVar_const(etInt, &nConst, "km s**-1");
 
// Multiply the constant times the delay time to get the range.  Constants
// always return the same value for any index provided, thus they can be
// combined with anything.
DasVar* pRange = DasVar_binaryOp(&pConst, "*", pDelay);
 
// Subtract the range from the spacecraft altitude.  Since vAlt and vDelay
// have the same rank (due to index remapping) we can setup an element by
// element binary operation
DasVar* pSigAlt = DasVar_binaryOp( &pAlt, "-", &pRange);
 
// Here's how to get data out of the resulting variable
double sigHeight = DasVar_double(&vSigAlt, IDX2(1234, 77));
 
// The component variables are evaluated in the same index space
double craftAlt = DasVar_double(&vAlt, IDX2(1234,77));
double range = DasVar_double(&vRange, IDX2(1234,77));
 
// If the same set of data are going to be evaluated multiple times it
// might be faster to just re-write all the internal data at a certain
// point into a new array.
DasVar* pEvalAlt = DasVar_evaluate(pSigAlt);

In the example above the variable tree was created manually. To support Das2.3 general streams expressions like the following will be parsable by the dataset object:

DasVar* pSigAlt; 
pSigAlt = Dataset_eval(pDs, "$altitude[i] - (3.0e9 / 2)*delay[j]", "km" );

To keep variables light weight (especially constants) they use stack value semantics instead of heap value semantics. This is a pain because you can't just return variables from functions, as they may reference other variables and you end up with dangling pointers. On the other hand, using stack semantics results in less heap fragmentation and thus faster code on current processors. Also simultaneous data reads are thread safe. Writes can be too so long as different threads write to different index ranges.

See also

Dataset

Dimension

Array

Member Function Documentation

new_DasVarUnary()

DAS_API DasVar * new_DasVarUnary	(	const char *	sOp,
		const DasVar *	pVar
	)

Create a new variable from unary operation on an existing variable.

Create a virtual variable from Operation(SubVar) as needed on an element by element basis, for example "Var1**-2", or "- Var1". For efficency, simple powers are combined into the operator.

The new variable does not allocate any storage, Getting elements from this variable will result in a sub-variable lookup and a calculation based on the given operator.

If a variable is to be iterated over multiple times the function new_DasVarEval() can be used to run this calculation and any sub calculations over all internal arrays and output the result into a new storage array.

Parameters

sOp	a string of lowercase letters or numbers describing the operation to apply. The following strings are understood: "-", "**2", "**3" "**-2", "**-3", "ln", "log", "sqrt", "curt", "sin", "cos", "tan" For vectors the operation: "norm" is understood
pVar	The variable to modify

Returns

A new DasVar allocated on the heap with it's reference count set to one.

See also

new_DasVarArray new_DasVarVecAry new_DasVarUnary

new_DasVarBinary()

DAS_API DasVar * new_DasVarBinary	(	const char *	sId,
		DasVar *	pLeft,
		const char *	sOp,
		DasVar *	pRight
	)

Create a new variable from a binary operation on two other variables.

Create a virtual variable from Var1 Operator Var2, for example Var1 + Var2.

The new variable does not allocate any storage though it does pre-calculate any needed scaling factors. Getting elements from this variable will result in two sub-variable lookups and a calculation based on the given operator.

The two variables must produce the same number of values when given the same set of indices. Most variables do not have an internal index so this is usually satisfied.

The two variables must have units that can be combined using given operator. Typically this means they must have the same units, but epoch units can be combined with pure "length" (i.e. duration) units under the addition and subtraction operators but two epoch units cannot be combined.

The new variable created by this binary combination will the units of the right sub-variable. Output of the left sub-variable will be scaled if needed before being combined on an element by element basis with the right sub-variable.

If a variable is to be iterated over multiple times the function new_DasVarEval() can be used to run this calculation and any sub calculations over all internal arrays and output the result into a new storage array

Parameters

sId	A name for this new variable. Use NULL for an anonymous variable. Anoymous variables are ususally for sub-expressions that aren't intended to be a top level data access point.

Names never hurt, an clean up expression displays. When in doubt, give the result of the binary operation a name.

Parameters

pLeft	the left index variable for the binary operation
sOp	the operation to preform, The following strings are understood "+","-","/","*","pow","dot","cross" The last two operations only work if the two variables are a vector.
pRight	the indexed variable in the binary operation

Returns

the new variable or NULL if an error occurred such as an unknown operator string.

See also

new_DasVarArray new_DasVarVecAry new_DasVarUnary

new_DasConstant()

DAS_API DasVar * new_DasConstant	(	const char *	sId,
		const das_datum *	pDm
	)

Create a constant value on the heap.

Wrap a constant value as a variable allowing for binary operations.

When asked for it's length in an index dimension, constant variables report as "Function" of every index position if they are scalars. Otherwise the last index reports as the length of the vector or text string as appropriate.

Parameters

sId	A short name for this constant, ex: 'c' for the speed of light
pDatum	A pointer to a datum to wrap.

new_DasVarSeq()

DAS_API DasVar * new_DasVarSeq	(	const char *	sId,
		das_val_type	vt,
		size_t	vSz,
		const void *	pMin,
		const void *	pInterval,
		int	nExtRank,
		int8_t *	pMap,
		int	nIntRank,
		das_units	units
	)

Create a simple linear sequence variable.

A simple sequence variable is linear in a single index. Many measurements happen in a parameter that progresses as a simple linea function of a single index. For example time offest for a single A/D capture from the start of the capture sequence.

Parameters

sId	An identifier for this sequence, follows rules for array ids
vt	the value type must be one of the values in das_val_type
vSz	the size in bytes for the value type, only used for vtByteSeq types
pMin	The minimum value for the sequence
pInterval	The interval between values of the sequence.
nDsRank	The rank of the total index space, same as the length of pMap
pMap	A mapping from DasDs indices to this sequence's lone index. The mape can only have one value set to 0, the rest must be marked digenerate.
units	The units for values produced by this sequence except for sequences of type vtTime. For time sequences, this is the units of the interval only. Output datums from the sequence will have units of UNIT_UTC.

Returns

A DasVar structure allocated on the heap.

new_DasVarArray()

DAS_API DasVar * new_DasVarArray	(	DasAry *	pAry,
		int	nExtRank,
		int8_t *	pMap,
		int	nIntRank
	)

Create a variable backed by an Array.

This variable will be backed by an array though the array indicies do not have to match the variable indicies. For example an array of frequencies for a time, frequency spectrogram might only have a single index [i], but the variable could access these as index [i][j] where j for the function maps to i for the array and i for the variable is ignored.

Parameters

pAry	The array which contains coordinate values
nExtRank	The external rank of the variable. This should match it's enclosing dataset, though some indicies can be marked as degenerate and are thus not mapped to the backing array.

Don't set this directly if you can avoid it. Use the SCALAR_N and VECTOR_N macros instead.

Parameters

pMap	The mapping of external indexs to DasAry indexes. Not every external index needs to be mapped.

Don't set this directly if you can avoid it. Use the SCALAR_N and VECTOR_N macros instead.

Parameters

nIntRank

The number of additional array indexes used to make the internal structure of Rank 1 items such as strings or Geometric Vectors.

Don't set this directly if you can avoid it. Use the SCALAR_N and VECTOR_N macros instead.

Returns

A pointer to the new variable object allocated on the heap

new_DasVarVecAry()

DAS_API DasVar * new_DasVarVecAry	(	DasAry *	pAry,
		int	nExtRank,
		int8_t *	pMap,
		int	nIntRank,
		const char *	sFrame,
		ubyte	nFrameId,
		ubyte	frametype,
		ubyte	nDirs,
		const ubyte *	pDir
	)

Create a vector backed by an array.

This variable must have one and only one internal index, and that index must be fixed. These conditions allow the variable to be a vector.

Vectors can be operated on using the scalar binary operations:

"+","-","/","*","pow"

but be aware that these operate in a component wise manner. So:

vA = new_DasVecArray(...);
vB = new_DasVecArray(...);
vC = new_DasVarBinary(NULl, vA, "+", vB);

will produce a vector where each value is vC[index] = vA[index] * vB[index]

Vector binary operations include:

"cross", "dot"

which only work if two vectors have the same frame.

Parameters

pAry	The array which contains data values
nExtRank	The external rank of the variable. This should match it's enclosing dataset, though some indicies can be marked as degenerate and are thus not mapped to the backing array.

Don't set this directly if you can avoid it. Use the SCALAR_N and VECTOR_N macros instead.

Parameters

pMap	The mapping of external indexs to DasAry indexes. Not every external index needs to be mapped.

Don't set this directly if you can avoid it. Use the SCALAR_N and VECTOR_N macros instead.

Parameters

nIntRank

The number of additional array indexes used to make the internal structure of Rank 1 items such as strings or Geometric Vectors.

Don't set this directly if you can avoid it. Use the SCALAR_N and VECTOR_N macros instead.

Parameters

sFrame	A named coordinate frame for the vector. If two vectors have different coordinate frames, they cannot be the subject of binary operations. Cannot be NULL.
nFrameId	The integer id of this frame. GeoVec datums only store the frame ID, not the name for faster comparisons
frametype	The lower byte of DasFrame.flags
pDir	A mapping between coordinate directions and the components of each vector, may be NULL.
nDirs	The number of directions in the vector direction map, can be 0

inc_DasVar()

DAS_API int inc_DasVar

(

DasVar *

pThis

)

Increment the reference count on a variable.

Returns

the new number of references to this variable

dec_DasVar()

DAS_API int dec_DasVar

(

DasVar *

pThis

)

Decrement the reference count on a variable.

If the reference count of a variable drops to zero then the variable decrements the reference count on all other variables and arrays that it may be using and then free's it's own memory.

You should set any local pointers refering to this variable to NULL after calling dec_DasVar as it may no longer exist.

Returns

the number of remaining references

DasVar_units()

das_units DasVar_units

(

const DasVar *

pThis

)

Get the units for the values.

Warning

For some vectors, only the magnitude has these units. To determine if this is the case use

DasVarAry_getArray()

DAS_API DasAry * DasVarAry_getArray

(

DasVar *

pThis

)

Get the backing array if present.

Returns

NULL if the variable is not backed directly by an array

DasVar_orthoginal()

DAS_API bool DasVar_orthoginal	(	const DasVar *	pThis,
		const DasVar *	pOther
	)

Getting data from a variable.

Answer the question: is one variable orthogonal in index space to another.

Parameters

pThis	pointer to the first variable object
pOther	pointer to the second variable object

Returns

True if the indicies that trigger a change in the first variable's output are completly separate from the indices that change the second variable's output

DasVar_shape()

DAS_API int DasVar_shape	(	const DasVar *	pThis,
		ptrdiff_t *	pShape
	)

Return the current shape of this variable.

Cause this variable to inspect it's managed array or sub-variables and determine the current extents in index space.

Parameters

pThis	The variable for which the shape is desired
pShape	a pointer to an array of size DASIDX_MAX. Each element of the array will be filled in with either one of the following:

• An integer from 0 to LONG_MAX to indicate the valid index range.
• The value D2IDX_UNUSED to indicate the given index position is ignored by this variable
• The value D2IDX_RAGGED to indicate that the valid index is variable and depends on the values of other indices.
• The value D2IDX_FUNC to indicate that the values are not stored but rather calculated from the given index itself. This is true for variables backed by un-bounded sequences instead of arrays.

Returns

The rank of the underlying storage or generation mechanism for the variable. This is typically not a useful item but could be if attempting to minimize DasVar_copy memory usage.

DasVar_intrShape()

DAS_API int DasVar_intrShape	(	const DasVar *	pThis,
		ptrdiff_t *	pShape
	)

Return the internal composition of this variable.

Variables may contain scalars, or they may contain items with internal structure. For example an array strings has an internal index on the byte number, geometric vectors have an internal index that represents the component in each direction.

Parameters

pThis	The variable for which the shape is desired
pShape	a pointer to an array of size DASIDX_MAX - 1. Each element of the array will be filled in with either one of the following

• An integer from 0 to LONG_MAX to indicate the valid index range. for a typicall geometric vector, this will be the number 3
• The value D2IDX_RAGGED to indicate that the valid index is variable and depends on the values of the external indicies. This is common for string data.

Returns

The rank of the interal components. In the case of scalars this will be 0, and pShape is not touched.

DasVar_lengthIn()

DAS_API ptrdiff_t DasVar_lengthIn	(	const DasVar *	pThis,
		int	nIdx,
		ptrdiff_t *	pLoc
	)

Return the current max value index value + 1 for any partial index.

This is a more general version of DasVar_shape that works for both cubic arrays and with ragged dimensions, or sequence values.

Parameters

pThis	A pointer to a DasVar structure
nIdx	The number of location indices which may be less than the number needed to specify an exact value.
pLoc	A list of values for the previous indexes, must be a value greater than or equal to 0

Returns

The number of sub-elements at this index location or D2IDX_UNUSED if this variable doesn't depend on a given location, or D2IDX_FUNC if this variable returns computed results for this location.

See also

DasAry_lengthIn

DasVar_toStr()

DAS_API char * DasVar_toStr	(	const DasVar *	pThis,
		char *	sBuf,
		int	nLen
	)

Get a string representation of this variable.

Parameters

pThis	a pointer to variable in question
sBuf	a buffer to hold the output, 128 bytes should be more than enough unless describing a deeply nested set of binary operation variables are preset.
nLen	the length of the string buffer. This function will not write more than nLen - 1 bytes to the buffer and will insure NULL termination

DasVar_isNumeric()

DAS_API bool DasVar_isNumeric

(

const DasVar *

pThis

)

Are the values in this variable convertable to doubles?

DasVar can hold any enum das_val_type. Many times applications just are expecting numeric values that can be converted to doubles. This function returns true if the underlying array elements are of type:

vtUShort, vtShort, vtInt, vtLong, vtFloat, vtDouble, vtTime

if the array value type is:

vtUByte

then values from this variable are considered to be convertable to doubles if the underlying DasAry doesn't indicate that the values are actually strings by way of the DasAry_getUsage() function.

For variables that are backed combinations of other variables instead of an array, sub-variables in the expression tree are consulted to get the answer. The first false return sticks.

See also

DasVar for a general description of variable expressions

Returns

True if the output a DasVar_getDatum() call (or the use of an iterator) will produce datums whose values are convertable to doubles. False othewise.

DasVar_get()

DAS_API bool DasVar_get	(	const DasVar *	pThis,
		ptrdiff_t *	pIdx,
		das_datum *	pDatum
	)

Get a value given an index.

This is the "slow boat from China" way to retrieve elements but it always works, even for non-orthogonal data sets, ragged arrays and variables built on expressions involving other variables. This is useful when re-gridding a data set onto a rectangular array such as a pixel or voxel raster.

Parameters

pThis	the variable in question
pIdx	The location to retrieve. Unmapped indices are ignored.
pDatum	pointer to a datum structure to fill in with the value

Returns

false if the indices represented by pIdx are invalid, a pointer to the data value otherwise. It is up to the caller to cast the pointer to the appropriate type and difference it to get the value.

DasVar_isFill()

DAS_API bool DasVar_isFill	(	const DasVar *	pThis,
		const ubyte *	pCheck,
		das_val_type	vt
	)

Check to see if a value is a fill value for this variable.

The two ways to check fill would be to get the data value and store it in some external variable, or to call this function and ask if a value is a fill value. Since das variables can contain many different fundamental types (int, double, das_time, const char* etc.) it's easier for applications build to this library to use this function, as all casting is handled by the variable itself.

Parameters

pThis	the variable in question.
pCheck	a pointer to the value to check for equivalence to fill.
vt	The type of the value to check.

Returns

true if this is a fill value, false otherwise.

DasVar_isComposite()

DAS_API bool DasVar_isComposite

(

const DasVar *

pVar

)

Is this a simple variable or more than one variable combinded via operators?

If variable actually represents an expression tree of variables combined via operations

DasVar_subset()

DAS_API DasAry * DasVar_subset	(	const DasVar *	pThis,
		int	nRank,
		const ptrdiff_t *	pMin,
		const ptrdiff_t *	pMax
	)

Copy a subset of a variable into a memory buffer.

Any evaluations needed to convert sequences, constants, binary operations etc. are handled and all values are output in a single continuous array.

Indexes that are sliced to a single value are removed from the returned shape function. Selection is by inclusive lower bound and an exclusive upper bound.

Giving away data memory buffers: The output DasAry may or may not own it's own memory. If it does you can take the memory and delete the array using DasAry_disownElements(). Otherwise, get the size and pointer to the memory using DasAry_getIn() and make a copy.

Parameters

pThis	the variable in question.
nRank	The length of the subset range arrays, which should be the same as the rank of the dataset. This argument is set when using the range macros (i.e. RNG_1, RNG_2, ...)
pMin	The inclusive lower bound for each index. Must be nRank elements long. This argument is set when using range macros.
pMax	The exclusive upper bound for each index. Must be nRank elements long. This argument is set when using range macros.

Returns

A pointer to a DasAry containing the selected range, or NULL if there is a problem. The output DasAry may or may not own it's own memory. The output DasAry will always be rectangular, regardless of any underlying raggedness. Calling DasAry_shape() on the return value will give non-zero, non-negative values.

Field Documentation

pUser

void* pUser

User data pointer.

The stream -> dataset -> dimension -> variable hierarchy provides a goood organizational structure for application data, especially applications that filter streams. It is initialized to NULL when a variable is created but otherwise the library dosen't deal with it.

---

The documentation for this struct was generated from the following file:

• das2/variable.h