1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 <2005> 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 | Index | 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 <2005> 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 |
<== Date ==> | <== Thread ==> |
---|
Subject: | Data Access Class Library Tutorial |
From: | Jeff Hill <[email protected]> |
To: | [email protected] |
Date: | Tue, 22 Feb 2005 15:22:20 -0700 |
Hello all, Attached is the latest Data Access Class Library Tutorial for your review. As described in previous talks given at user's group meetings and at ICALEPCS by Ralph Lange and myself, the Data Access interface and its associated support libraries were designed and written for incorporation into future Channel Access client and server interfaces. The work on a Data Access implementation is in its 3rd generation. The next stage will be to begin using it. The original CA client interface will of course be preserved through use of Data Access interfaces for all of the preexisting dbr_xxx types. Thanks for your patience and consideration for our ideas in the past, and thanks in advance for any time you can spare for further comments. Best regards, Jeff Hill and Ralph LangeTitle: Abstract Data Class Library Tutorial
Data Access Class Library TutorialCopyright © 2002 The University of Chicago, as Operator of Argonne National Laboratory. Copyright © 2002 The Regents of the University of California, as Operator of Los Alamos National Laboratory. Copyright © 2002 Berliner Elektronenspeicherringgesellschaft für Synchrotronstrahlung. EPICS BASE Versions 3.13.7 and higher are distributed subject to a Software License Agreement found in the file LICENSE that is included with this distribution. Modified on $Date: 2005/02/22 22:19:56 $ Table of Contents
ScopeThis document is a tutorial introduction to a generic C++ programming interface for introspecting proprietary data. Who to BlameJeff Hill (LANL SNS Division) and Ralph Lange (BESSY) are responsible for the design of this interface and the contents of this document. IntroductionData Access is a generic interface for introspection of proprietary data with complex structure. A program may choose to export its proprietary data using this interface. Once this is accomplished then any programs that know the interface may examine and or potentially modify the data. The user is not required to store his data in a particular format, but nevertheless knowledge of the structure of the data may be determined at compile time, and therefore efficient access to the data can be made. A support library is supplied for copying and comparing between properly interfaced property sets. Why We Need ThisCurrently EPICS has a fixed set of meta-data, but this needs to expand because EPICS base developers can't anticipate all possible meta-data, and all possible meta-data permutations. Expansion of the toolset will hopefully accelerate if application developers can define new meta-data. Pivotal to a tool based approach is proper decoupling of tools from each other so that changes in one tool do not cause another tool to break. Data Access is about expanding the meta-data set while keeping the tools properly decoupled. If the meta-data set is expanded in a data source we must not require that all of the clients of that source be rewritten. In multi-agent systems synchronization is a reoccurring theme. Currently, EPICS synchronizes a single parameter with a fixed set of meta data. Data Access facilitates synchronization of an arbitrary application defined set of meta-data with a time stamp, an arbitrary (application defined) event, a client’s read or write request, or a synchronized multi-channel read / write. EPICS needs an application extensible event set. For example, a server might post an “arcDown” event with an application specific data capsule. A client might subscribe for event “arcDown” and specify a subset of meta-data to be acquired from the capsule posted with the event. It is essential in this scenario for the client and server data spaces to be decoupled. The Data Access support libraries efficiently copy between dissimilar, decoupled types. Intelligent instruments are becoming the norm, and they require message passing. Devices communicate using arbitrary request / response capsules, and Data Access interfaces arbitrary data capsules. PropertiesThe data access interface requires that all data be assigned a property name. A property name might be weight, units, maximum, or potentially any meta-data name and purpose that a group of programs mutually agree upon. A set of data with unique property names may be stored in a container, and this container is also assigned a property name. Properties are organized in hierarchies. For example, the weight property might have subordinate high display limit and low display limit properties that belong to it. Likewise, the height property might need to have the same properties assigned to it. If the weight property and the height property both exist in the same container then the appropriate subordinate properties may need to have independent values depending on whether they are used with the weight property or the height property. Therefore, the weight and height properties both have optional subordinate properties. This results in a tree structured property hierarchy. It is anticipated that the facilities in this library will not be generally useful unless users develop standards early on for the names and purposes of the properties shared by cooperating programs. InterfacesPrograms using the data access interface can be roughly categorized into three different roles working as a property catalog, a data viewer, or a data manipulator.
C++ Name SpaceAll of the interfaces described in this document are in C++ name space
using namespace da; Primitive Data TypesInterfaced data may be stored in any of the C++ primitive types, or in specialized types for strings and time stamps. Additional specialized types may need to be supported in the future as convenience or efficiency dictate. Property IdentifiersAll interfaced data must be assigned a property name. A property name can be converted to a property identifier as follows. #include "daPropertyId.h" static const propertyId propertyIdWeight ( "weight" ); static const propertyId propertyIdHeight ( "height" ); static const propertyId propertyIdValue ( "value" ); static const propertyId propertyIdHighLimit ( "high limit" ); static const propertyId propertyIdLowLimit ( "low limit ); Built-in precompiled property identifiers will be supplied for some of the commonly used property names. A partial list follows (the standard property set needs to be defined first before this can be documented). Property CatalogA property catalog derives from the interface class
#include "dataAccess.h" class myGirth : public propertyCatalog { private: double height; double weight; }; Property Catalog TraversalFrequently it is necessary to traverse through all of the published
properties. For example, a utility program could be used to archive many
different types of data to and from disk storage — as long as each
type of data provides an implementation of a generic property traversal
interface. To this end the data interfacing class provides a
void myGirth::traverse ( dataViewer & viewer ) const { viewer.reveal ( propertyIdHeight, this->height ); viewer.reveal ( propertyIdWeight, this->weight ); } A similarly structured non- traverseModifyStatus myGirth::traverse ( dataManipulator & manipulator ) { viewer.reveal ( propertyIdWeight, this->weight ); return tmsSuccess; } When a data interfacing class must verify that a value modified by the
traverseModifyStatus myGirth::traverse ( dataManipulator & manipulator ) { double tmpWeight = this->weight; manipulator.reveal ( propertyIdWeight, tmpWeight ); if ( tmpWeight < 0.0 ) return tmsOutOfRangeLow; if ( tmpWeight > 10.0 ) return tmsOutOfRangeHigh; this->weight = tmpWeight; return tmsSuccess; } When a robustly interfaced property set calls void myGirth::traverse ( dataManipulator & manipulator ) { double tmpHeight = this->height; manipulator.reveal ( propertyIdHeight, tmpHeight ); if ( tmpHeight < 0.0 ) return tmsOutOfRangeLow; if ( tmpHeight > 10.0 ) return tmsOutOfRangeHigh; double tmpWeight = this->weight; manipulator.reveal ( propertyIdWeight, tmpWeight ); if ( tmpWeight < 0.0 ) return tmsOutOfRangeLow; if ( tmpWeight > 10.0 ) return tmsOutOfRangeHigh; this->height = tmpHeight; this->weight = tmpWeight; return tmsSuccess; } Subordinate Container TraversalSuppose that we have limit properties and would like to publish them using the data access interfaces. As expected, we will need to create an interfacing class for the limits with traverse functions for the limit properties. class myLimits : public propertyCatalog { private: double highLimit; double lowLimit; }; void myLimits::traverse ( dataViewer & viewer ) const { viewer.reveal ( propertyIdHighLimit, this->highLimit ); viewer.reveal ( propertyIdLowLimit, this->lowLimit ); } Next, we might add limits to the class myGirth : public propertyCatalog { private: double height; double weight; myLimits limits; }; Next, we need to bind the limits subordinate properties to the height and
weight properties. This is accomplished in the traverse function for the
void myGirth::traverse ( dataViewer & viewer ) const { viewer.reveal ( propertyIdHeight, this->height, this->limits ); viewer.reveal ( propertyIdWeight, this->weight, this->limits ); } Property Catalog Indexed by Property IdentifierA program might choose to extract out of a data container only the specific properties that it needs, and would therefore need to locate a particular property in an unknown container indexed only by its property identifier. For example, we might choose to move data between dissimilar container types. Compared to the traversal mechanism above, we expect to introduce an additional degree of flexibility required by certain applications at the expense of some loss of runtime efficiency. Data that is interfaced for this type of access must provide the following function. void myGirth::find ( const propertyId & type, dataViewer & adt ) const; The Recognition that indexing mechanisms can be greatly simplified when containers have a limited number of properties may prove to be an impetus to design property hierarchies with a limited number of properties on each level. To use this #include "daLocator.h" class myGirth : public propertyCatalog { private: static bool init; static locator; }; Here is an example. void myGirth::find ( const propertyId & id, propertyViewer & viewer ) const { myData::locator.callExportFunction ( *this, id, viewer ); } Next, "binding" member functions are added for each property with indexed access. void myGirth::heightReadBinder ( propertyViewer & viewer ) const { viewer.reveal ( propertyIdHeight, this->height ); } Finally, these "binding" member functions must be installed into the
myGirth::myGirth () { if ( myGirth::init == false ) { myGirth::locator.installExportFunction ( propertyIdHeight, heightReadBinder ); myGirth::locator.installExportFunction ( propertyIdWeight, weightReadBinder ); myGirth::init = true; } } Note that the central aspect of the Operators for Properly Interfaced DataThe data access library provides support for assignment and equivalence
operations. Users may perform assignment and equivalence comparison between
two enum assignStatus { asSuccess = 0, asOutOfRangeLow = 1, asOutOfRangeHigh = 2, asInvalidState = 3, asIncompatibleTypes = 4, asElementIndexOverflow = 5, asUndefinedElements = 6, asUndefinedProperty = 7, asUnableToExtend = 8, asUnexpected = 9 }; epicsShareFunc assignStatus assign ( propertyCatalog & lhs, const propertyCatalog & rhs ); epicsShareFunc assignStatus assign ( propertyCatalog & lhs, const stringSegment & rhs, const propertyCatalog & rhsMeta ); epicsShareFunc assignStatus assign ( stringSegment & lhs, const propertyCatalog & lhsMeta, const propertyCatalog & rhs ) ; epicsShareFunc assignStatus assign ( arraySegment & lhs, const propertyCatalog & lhsMeta, const arraySegment & rhs, const propertyCatalog & rhsMeta ); epicsShareFunc void throwExceptionIfUnsuccessful ( assignStatus ); enum equivStatus { esEqual = 0, esNotEqual = 1, esIncompatible = 2, esElementIndexOverflow = 3, esUndefinedElements = 4, esUndefinedProperty = 5 }; epicsShareFunc equivStatus equiv ( const propertyCatalog & lhs, const propertyCatalog & rhsMeta ); epicsShareFunc equivStatus equiv ( const propertyCatalog & lhs, const stringSegment & rhs, const propertyCatalog & rhsMeta ); epicsShareFunc equivStatus equiv ( const arraySegment & lhs, const propertyCatalog & lhsMeta, const arraySegment & rhs, const propertyCatalog & rhsMeta ); bool equivStatusToBool ( equivStatus stat ); Specialized Data TypesArraysAs with scalar properties, arrays are interfaced by calling a
#include "daArray.h" class myArray : public arraySegment { private: float chunkOne[1024]; float chunkTwo[1024]; }; class myArrayData : public propertyCatalog { private: myArray array; }; void myArrayData::traverse ( propertyViewer & viewer ) { viewer.reveal ( propertyIdValue, this->array ); } Array BoundsInformation about the bounds of an array is provided using the
unsigned myArray::numberOfDimensions () const { return 1u; } arrayBounds::bound myArray :: getBound ( unsigned dimension ) const { arrayBounds::bound bd; bd.first = 0; if ( dimension == 0u ) { bd.count = sizeof ( chunkOne ) + sizeof ( chunkTwo ); } else { bd.count = 1; } return bd; } Array TraversalArrays may be stored in non-contiguous blocks of memory. This allows for
improved memory management (less fragmentation). The
void myArray::traverse ( arrayViewer & adt ) const { adt.reveal ( this->chunkOne, sizeof ( this->chunkOne ) ); adt.reveal ( this->chunkTwo, sizeof ( this->chunkTwo ) ); } When there are multiple calls to If the array is multi-dimensional then array elements are revealed in the
natural order for multi-dimensional arrays in the C language. That is,
elements are revealed in row-major order where the right most subscript in
the C language declaration Array Slice TraversalA multi-dimensional slice is specified by a user defined class deriving
from interface class mySice : public arrayBounds { public: unsigned numberOfDimensions () const; arrayBounds::bound getBound ( unsigned dimension ) const; }; A slice sequence index argument is also passed to the array slice
interfacing Similar to ordinary array Here is an example implementation of an array slice sliceTraverseStatus myArray::traverse ( arrayBounds & slice, const arrayBounds::bound & sliceSeqIndex, arrayViewer & viewer ) const { // verify that the slice is in bounds static const unsigned arrayLength = sizeof ( this->chunkOne ) + sizeof ( this->chunkTwo ); unsigned nDim = slice.numberOfDimensions (); if ( nDim < 1u ) { return stsSliceOutOfBounds; } daArrayDescriptor::bound bd = slice.getBound ( 0u ); if ( bd.count > arrayLength ) { return stsSliceOutOfBounds; } if ( bd.first > arrayLength - bd.count ) { return stsSliceOutOfBounds; } // this is one dimensional array - verify that // any multi-dimensional bounds are scalar for ( unsigned dim = 1u; dim < nDim; dim++ ) { arrayBounds::bound multiBd = slice.getBound ( dim ); if ( multiBd.first != 0u || multiBd.count != 1u ) { return stsSliceOutOfBounds; } } // verify that the slice sequence index is within the specified slice if ( sliceSeqIndex.count >= bd.count ) { return stsSliceIndexOutOfBounds; } if ( sliceSeqIndex.first > bd.count - sliceSeqIndex.count ) { return stsSliceIndexOutOfBounds; } bd.first = bd.first + sliceSeqIndex.first; bd.count = sliceSeqIndex.count // reveal array slice const unsigned last = bd.first + bd.count - 1u; if ( bd.first < sizeof ( this->chunkOne ) ) { if ( last > sizeof ( this->chunkOne ) - 1u ) { viewer.reveal ( &this->chunkOne[bd.first], sizeof ( this->chunkOne ); viewer.reveal ( this->chunkTwo, bd.count - sizeof ( this->chunkOne ) ); } else { viewer.reveal ( &this->chunkOne[bd.first], bd.count ); } } else { viewer.reveal ( &this->chunkTwo[ bd.first-sizeof ( this->chunkOne ) ], bd.count ); } return stsSuccess; } When Native Array Storage Isn't Matching C's Native Row-Major OrderIn this situation the user will need to less efficiently call the
StringsAll strings are interfaced through the At the highest level a string is considered to be a linear sequence of
tokens convertable to Implementations of the enum stringDiff { sdBelow, sdEqual, sdAbove }; class stringSegment : public streamPosition, public primTypeConversion { public: virtual ~stringSegment () = 0; virtual bool getChar ( unsigned & ) const throw () = 0; // returns 0 when end of string is reached virtual bool putChar ( unsigned ) = 0; virtual stringDiff compare ( const stringSegment & ) const = 0; }; The stream position interface allows the total length of the string, its current position, and its number of tokens available in memory at the current position to be queried. Likewise, the current position can be set, all tokens from the current position to the end of the stream can be removed (pruned), and any tokens in a cache can be flushed. The first token in the string has position zero, the second token has position one, and all subsequent tokens are sequentially numbered. Strings are initialized with the first element in the string being the current token. class streamPosition { public: virtual size_t length () const throw () = 0; virtual size_t position () const throw () = 0; virtual bool movePosition ( size_t newPosition ) throw () = 0; virtual size_t viewable () = 0; virtual bool prune () throw () = 0; virtual void flush () throw () = 0; }; The The enum streamWriteStatus { swsSuccess = 0, swsUnableToExtend = 1 }; class streamWrite { public: virtual streamWriteStatus write ( const double &, const propertyCatalog & = voidCatalog ) = 0; virtual streamWriteStatus write ( const int &, const propertyCatalog & = voidCatalog ) = 0; virtual streamWriteStatus write ( const long &, const propertyCatalog & = voidCatalog ) = 0; virtual streamWriteStatus write ( const unsigned &, const propertyCatalog & = voidCatalog ) = 0; virtual streamWriteStatus write ( const unsigned long &, const propertyCatalog & = voidCatalog ) = 0; virtual streamWriteStatus write ( const epicsTime &, const propertyCatalog & = voidCatalog ) = 0; virtual streamWriteStatus write ( const class stringSegment &, const propertyCatalog & = voidCatalog ) = 0; }; The enum streamReadStatus { srsSuccess = 0, srsOutOfRangeLow = 1, srsOutOfRangeHigh = 2, srsIncompatible = 3, srsIncomplete = 4 }; class streamRead { public: virtual streamReadStatus read ( double &, const propertyCatalog & = voidCatalog ) const = 0; virtual streamReadStatus read ( int &, const propertyCatalog & = voidCatalog ) const = 0; virtual streamReadStatus read ( long &, const propertyCatalog & = voidCatalog ) const = 0; virtual streamReadStatus read ( unsigned &, const propertyCatalog & = voidCatalog ) const = 0; virtual streamReadStatus read ( unsigned long &, const propertyCatalog & = voidCatalog ) const = 0; virtual streamReadStatus read ( epicsTime &, const propertyCatalog & = voidCatalog ) const = 0; virtual streamReadStatus read ( class stringSegment &, const propertyCatalog & = voidCatalog ) const = 0; }; Enumerated (Limited Set of Labeled States) DataThe data access interface allows enumerated (Limited Set of Labeled
States) data to be stored in any of the primitive types as long as all of the
state set values are convertable to traverseModifyStatus myData::traverse ( propertyManipulator & manipulator ) { unsigned tmp = this->enumValue; manipulator.reveal ( propertyIdValue, tmp, this->stateSet ); // range check if ( tmp != 0 && tmp != 1 ) { return tmsInvalidState; } this->enumValue = tmp; return tmsSuccess; } There is also an interface allowing applications to view and manipulate
the supported states. This is accomplished by providing a subordinate
property of primitive type class enumViewer { public: virtual void reveal ( const int state, const stringSegment & ) = 0; }; class enumStateSet { public: virtual unsigned numberOfStates () const = 0; virtual void traverse ( enumViewer & ea ) const = 0; virtual bool matchingState ( const stringSegment & label, int & state ) const = 0; virtual bool matchingState ( const int state, stringSegment & label ) const = 0; // return false only if unsuccessful virtual bool removeAllStates () = 0; virtual bool removeState ( int state ) = 0; virtual bool setLabel ( int state, const stringSegment & ) = 0; }; Design GoalsThe overall design goal was to present concise interface to users and to, whenever possible, shift programming labor from users to the library implementors where there is maximum benefit and minimized code duplication. Data Access isn't being designed for office computing. For a control system we need to adhere to some basic principals.
The user should not be required to store his data in a particular format. Nevertheless, knowledge of the structure of the data must be permitted to be determined at compile time so that access to the data can be efficient. The interface must not preclude user data stored in multiple non-contiguous blocks. Memory management based on fixed sized non-contiguous blocks allows for predictable free lists based memory allocation which implies low latency, no memory fragmentation, and predictable latency. The interface must not require C-RTL general purpose memory management -
AKA Object Code SizeThere has been a significant preoccupation surrounding the object code size of data access and so its necessary to clarify this issue. Data Access is in essence only an interface. When looking at code size we
are comparing the sizes of the accompanying support library components.
Currently the support library provides equivalence and assignment between
properly interfaces data containers. In an IOC it is unlikely that the
equivalence functionality will be needed and so we should consider the size
only of the assignment component which on When comparing the sizes of object code between C++ and C one must use the
UNIX Frequently Asked QuestionsWhoa, this thing is called Data Access!Don’t O.O. systems use messages and remote procedure calls? That’s the new technology! Data Access was invented for the purpose of passing messages - to specify the parameters of the messages. Why not use a data description compiler like XDR, CORBA IDL, or EPICS DBD.This is certainly worthy of consideration, but proper decoupling of sender and receiver data spaces appears to be important for a tool based approach. Conventional data description compiler based systems require interfaces of the sender and receiver to be utterly identical parameter-for-parameter, field-for-field, and bit-for-bit. The sender and receiver must have the same unique data structure identifier. If not, no communication is possible. However, consider for example the requirements for future implementations of EPICS. In these systems events posted to the server may have many associated subsystem unique properties. Clients will rarely need all of them, and there will be many permutated subsets requested by a range of different clients. Likewise, clients written in the past should continue to function if new properties are added to an event. Furthermore, schemes like XDR require that the data be stored in, or converted to, a particular format as produced by the XDR compiler. This works fine for simple scalar datums, but becomes cumbersome for variable length datums such as strings and arrays. When complex data is stored in a proprietary format significant overhead can arrise. For excample, in limited memory systems it is important to allow scattered, non-contiguous storage of datums. It must not be required that random sized dynamically allocated blocks of memory exist only for the short duration that a complex dataum is passed between two different layers in the system. The XDR approach also tends to be inflexible when it comes to interfacing with multi-dimensional arrays. In contrast, Data Access does not enforce a native storage format and therefore does not suffer from the above limitations, and our perception is that this is a more flexible, less intrusive, and better performing approach. Where are the size locked types?Past versions of CA were based on size locked types - i.e. "typedef
epicsInt16 dbr_int_t;". However, in my experience users seldom bothered to
use dbr_xxx_t, and therefore my perspective has evolved. Size locked types
should not be in interfaces used by users. Becauase Data Access uses
overloaded MisconceptionsThis is a C++ template based interface.In fact, this is a pure virtual base class based interfaced. Templates are used only in the implementation of the support library. Templates need not be seen by users. This is a data object.In fact, this is a universal interface to non-uniform data. Proprietary data storage formats need not change. We are not creating an object that defines a storage format, allocates space, and stores data.This isn't another GDD or cdevData. Its best to code everything in C because C can be called from C++, Java, Python etc...It is of course possible to call back and forth between any of these
languages and, IMHO, none of them would be successful if that were not the
case. The C++ interfaces described herein do not have templates in them and
so wrappers can be easily be provided so that they can be called from from C,
Java, Python, etc. A C++ plug in for the Compared to C, the code and interfaces described herein can be more efficiently maintained and more efficiently executed at runtime when written in C++. This maintenance efficiency stems from C++'s template capabilities. In C we must either write and maintain a program to create the conversion matrix or else write and maintain many functions which could be supplied in C++ with one template. The runtime efficiency derives from C++'s overloaded function and template capabilities. In C, the user would see externally an interface with a void pointer and an additional parameter specifying a data type code. Internally a C implementation would call the conversion matrix using a data type code indexed jump table. With C++ this additional step can be eliminated through use of overloaded functions. The C++ approach with overloaded functions is also less error prone for the user. Contrast C++ overloaded functions with the typical C interface to this type of functionality requiring a type code and a void pointer. The C interface is without doubt more error prone leaving problems related to improperly specified type codes to be discovered (and debugged) at run time. With C++ overloaded functions the compiler enforces the type system at compile time. This interface isn't compatible with pure Java.A pure Java implementation could also be written. Java does not have templates so when implementing a conversion matrix for the copy and assignment operators a program that creates a program would probably need to be written as would also be the case if this functionality were developed in pure C. IMHO, this is mostly not a technical issue, but instead a resource limitation and or administrative issue. With a small amount of up front planning we could probably write and maintain every component of EPICS in both C++ and Java as our needs and budgets allow. Oustanding IssuesVirtual Base Class for Time Stamps?Currently, time stamps are interfaced through class epicsTime. Should time stamps be interfaced using a pure virtual base class as has been the standard approach for all other complex data types such as strings and enumerated state set descriptions. $Id: dataAccessTutorial.htm,v 1.7 2005/01/06 00:21:01 jhill Exp $ |