Re: Question on Accessing Data in Custom PV Structures

[Maren Purves via Tech-talk <tech-talk at aps.anl.gov>]

Coming from the astronomy side, I agree with both Timo and Michael - but we write those out as data files in a more or less fixed format by processes that do not run in the IOC. There is a file header that contains the metadata, and for things that change, with start values and end values (like observation start and end times, weather data, etc., given that observations, especially in case of large maps, can take an hour or so) and a data array that contains the actual data, where the X and Y that Michael mentions can be time and signal, but for most instruments we don't collect the data with EPICS only. At the JCMT we do not have any instruments in use that collect the actual data using EPICS but most of our metadata are, and e.g. weather data (in a broader sense) with the timestamp of the record. At UKIRT (with which I am still somewhat involved) one of the instruments collects all the data using EPICS.

To get back to the original question: how do you store the data? Maybe each data (and metadata) point should be stored with its timestamp?

Maren Purves

Head of Instrument and Telescope Software

East Asian Observatory / JCMT

On Tue, Jul 11, 2023 at 12:25 AM Timo Korhonen via Tech-talk <tech-talk at aps.anl.gov> wrote:

I have a slightly different angle on this, even if I mostly agree with Michael.

In my opinion it makes often sense to group together data items even if they do not need to be strictly atomic, and are only loosely (to be defined) related.
With PVA groups one can create consistent data items that contain not only the payload data but descriptive metadata as well. For an acquired waveform for instance, it is very useful to know the sampling frequency for instance, instead of having to separately fish it from different PVs, or hard-code the number in client applications.
Also, some other descriptive metadata can be very useful to have transported in one unit together with the payload.

This will make many applications simpler and reduce the uncertainty when one tries to backtrack what was going on with a particular measurement.

For sure, atomicity is the killer feature but I think there are several other reasons why one would like to use PVA groups.

Just my 2 cents.

Timo


On 2023-07-10, 18:00, "Tech-talk on behalf of Michael Davidsaver via Tech-talk" <tech-talk-bounces at aps.anl.gov <mailto:tech-talk-bounces at aps.anl.gov> on behalf of tech-talk at aps.anl.gov <mailto:tech-talk at aps.anl.gov>> wrote:


On 7/10/23 08:18, Srinivas, Dhruv wrote:
> Right now, I'm attempting to group multiple records into a single structure for improved organization of data. Essentially letting us only need to import this PV structure rather than importing several records manually.


The motivating use case for the PVA group feature is data atomicity. In the
context of your example if "rec:X" and "rec:Y" really are two parts of a tuple,
and must be consumed together, then a group make sense to avoid the possibility
of off-by-one errors of X wrt. Y which are probable when subscribing to each
individually.


On the other hand, if "X" and "Y" are only loosely related. Just happening to
be in in the same IOC. Then I am unsure including them in the same PVA group
will be of any benefit. It may even be a drawback as it means coming up with
names for the fields of the group. As we all know, naming is the hardest
problem in controls!




> As for what client(s) will consume this data structure. That would primarily be other IOCs themselves that are on the same subnet, but in different machines.


In that case, I would suggest looking at


https://epics-base.github.io/pva2pva/qsrv_page.html#qsrv_link <https://epics-base.github.io/pva2pva/qsrv_page.html#qsrv_link>


In particular the "field" key, which can index into an arbtrary
PVD structure. For the example below, something like


> record(ai, "client:X") {
> field(INP, {pva:{
> pv:"CustomDataStructure",
> field:"X_Scalar",
> proc:"CP"
> }})
> }








> Sincerely,
>
> Dhruv Srinivas
>
> -----Original Message-----
> From: Michael Davidsaver <mdavidsaver at gmail.com <mailto:mdavidsaver at gmail.com>>
> Sent: Saturday, July 8, 2023 6:05 PM
> To: Srinivas, Dhruv <srinivas1 at llnl.gov <mailto:srinivas1 at llnl.gov>>
> Cc: tech-talk at aps.anl.gov <mailto:tech-talk at aps.anl.gov>
> Subject: Re: Question on Accessing Data in Custom PV Structures
>
> On 7/7/23 14:03, Srinivas, Dhruv via Tech-talk wrote:
>> To whom it may concern,
>>
>> I’m attempting to create custom PVStructures which contain differing normative types. At the moment, I’ve created the below code in an IOC which creates a PV Structure with 2 NTScalars and 1 NTScalarArray.
>
> Can you say something about what you are trying to accomplish, and what kind of client(s) will consume this data structure?
>
>
>> record(ai, "rec:X") {
>>
>> field(VAL, "20")
>>
>> field(HIGH, "30")
>>
>> field(HSV, "MINOR")
>>
>> info(Q:group, {
>>
>> "CustomDataStructure": {
>>
>> "X_Scalar": {
>>
>> +channel:"VAL"
>>
>> }
>>
>> }
>>
>> })
>>
>> }
>>
>> record(ai, "rec:Y") {
>>
>> field(VAL, "40")
>>
>> info(Q:group, {
>>
>> "CustomDataStructure": {
>>
>> "Y_Scalar": {
>>
>>
>> +type:"scalar",
>>
>>
>> +channel:"VAL",
>>
>>
>> +id:"MyTestingPVA2PVA",
>>
>> }
>>
>> }
>>
>> })
>>
>> }
>>
>> record(aai, "input_this") {
>>
>> field(NELM, "4")
>>
>> field(FTVL, "INT64")
>>
>> field(INP, "[1,2,3,4]")
>>
>> info(Q:group, {
>>
>> "CustomDataStructure": {
>>
>> "Z_Aray": {
>>
>> +channel:"VAL"
>>
>> }
>>
>> }
>>
>> })
>>
>> }
>>
>> At the moment, this does create the PV Structure, and I can access it through “pvget”. However I’m unsure if this is the right way to do this. Can someone with more experience with Epics v7 correct me if so? Also, I’m not sure how to go about accessing parts of “CustomDataStructure”. Namely how would I go about accessing the value for Y_Scalar through CustomDataStructure?
>>
>> I’ve also copied the output of pvget below. I apologize for the lengthy post, but I thought it was best to provide as much context as possible.
>>
>> * pvget CustomDataStructure
>>
>> CustomDataStructure MyTestingPVA2PVA
>>
>> structure record
>>
>> structure _options
>>
>> uint queueSize 0
>>
>> boolean atomic true
>>
>> epics:nt/NTScalar:1.0 X_Scalar
>>
>> double value 20
>>
>> alarm_t alarm
>>
>> int severity 0
>>
>> int status 2
>>
>> string message UDF
>>
>> structure timeStamp
>>
>> long secondsPastEpoch 631152000
>>
>> int nanoseconds 0
>>
>> int userTag 0
>>
>> structure display
>>
>> double limitLow 0
>>
>> double limitHigh 0
>>
>> string description
>>
>> string units
>>
>> int precision 0
>>
>> enum_t form (0) Default
>>
>> int index 0
>>
>> string[] choices [Default, String, Binary, Decimal,
>> Hex, Exponential, Engineering]
>>
>> control_t control
>>
>> double limitLow 0
>>
>> double limitHigh 0
>>
>> double minStep 0
>>
>> valueAlarm_t valueAlarm
>>
>> boolean active false
>>
>> double lowAlarmLimit nan
>>
>> double lowWarningLimit nan
>>
>> double highWarningLimit 30
>>
>> double highAlarmLimit nan
>>
>> int lowAlarmSeverity 0
>>
>> int lowWarningSeverity 0
>>
>> int highWarningSeverity 0
>>
>> int highAlarmSeverity 0
>>
>> byte hysteresis 0
>>
>> epics:nt/NTScalar:1.0 Y_Scalar
>>
>> double value 40
>>
>> alarm_t alarm
>>
>> int severity 0
>>
>> int status 2
>>
>> string message UDF
>>
>> structure timeStamp
>>
>> long secondsPastEpoch 631152000
>>
>> int nanoseconds 0
>>
>> int userTag 0
>>
>> structure display
>>
>> double limitLow 0
>>
>> double limitHigh 0
>>
>> string description
>>
>> string units
>>
>> int precision 0
>>
>> enum_t form (0) Default
>>
>> int index 0
>>
>> string[] choices [Default, String, Binary, Decimal,
>> Hex, Exponential, Engineering]
>>
>> control_t control
>>
>> double limitLow 0
>>
>> double limitHigh 0
>>
>> double minStep 0
>>
>> valueAlarm_t valueAlarm
>>
>> boolean active false
>>
>> double lowAlarmLimit nan
>>
>> double lowWarningLimit nan
>>
>> double highWarningLimit nan
>>
>> double highAlarmLimit nan
>>
>> int lowAlarmSeverity 0
>>
>> int lowWarningSeverity 0
>>
>> int highWarningSeverity 0
>>
>> int highAlarmSeverity 0
>>
>> byte hysteresis 0
>>
>> epics:nt/NTScalarArray:1.0 Z_Scalar
>>
>> long[] value [1,2,3,4]
>>
>> alarm_t alarm INVALID DRIVER UDF
>>
>> int severity 3
>>
>> int status 2
>>
>> string message UDF
>>
>> structure timeStamp
>>
>> long secondsPastEpoch 631152000
>>
>> int nanoseconds 0
>>
>> int userTag 0
>>
>> structure display
>>
>> double limitLow 0
>>
>> double limitHigh 0
>>
>> string description
>>
>> string units
>>
>> int precision 0
>>
>> enum_t form (0) Default
>>
>> int index 0
>>
>> string[] choices [Default, String, Binary, Decimal,
>> Hex, Exponential, Engineering]
>>
>> control_t control
>>
>> double limitLow 0
>>
>> double limitHigh 0
>>
>> double minStep 0
>>
>> valueAlarm_t valueAlarm
>>
>> boolean active false
>>
>> double lowAlarmLimit nan
>>
>> double lowWarningLimit nan
>>
>> double highWarningLimit nan
>>
>> double highAlarmLimit nan
>>
>> int lowAlarmSeverity 0
>>
>> int lowWarningSeverity 0
>>
>> int highWarningSeverity 0
>>
>> int highAlarmSeverity 0
>>
>> byte hysteresis 0
>>
>> Sincerely,
>>
>> Dhruv Srinivas
>>
>> srinivas1 at llnl.gov <mailto:srinivas1 at llnl.gov> <mailto:srinivas1 at llnl.gov <mailto:srinivas1 at llnl.gov>>
>>
>