Re: Structured PVs archiving and archived data retrieval problem

[Aleš Kete via Tech-talk <tech-talk at aps.anl.gov>]

On Mon, 2025-03-17 at 18:39 -0700, Michael Davidsaver wrote:

On 3/17/25 04:46, Aleš Kete via Tech-talk wrote:

Hello everyone,

(I am pretty new to EPICS software ecosystem and am trying to learn about typical workflows for acquiring and archiving data through EPICS.)

To start, I have two questions:

* How complete is support for structured PVs in EPICS archiver appliance and aaclient tools (more details below)?

Unless specifically handled in Archiver Appliance code, arbitrary PVA Structures can only be stored and retrieved verbatim.  So most (all?) of the binning and statistical operations can not be used.

To my knowledge, all normative types have timeStamp attribute, so at least this can be used for binning. As EPICS Archiver novice, I assume by "binning" you mean assigning/moving to different storages and somehow indexing for efficient retrieval.

* Is there (also some other) standard tool for archiving EPICS data?

There are standard tools for the standard data types.  Primarily the scalar and simple array types (NTScalar and NTScalarArray).

With NTTable, there is also some ambiguity on how to interpret.  eg. I wrote one tool which subscribes to an NTTable PV, then treats each update as additional rows to be appended to one very long table.  In this case, that does not seem like what you are looking for.  So, I don't have a recommendation for an out-of-the-box solution.

IMO data interpretation can and (most times) should be left to the client . Doesn't selection of (generic) data "chunks" based on timestamp range cover most users use cases?

Some context from which the questions arose:

We are  working on custom record type support for PVs providing multi-channel signal acquisitions, we added support (via qsrv / qsrv2) for obtaining data through PV structured according to epics:nt/NTTable:1.0 specification.

If you are working in the context of QSRV1/2, I would suggest configuring Archiver Appliance to record the individual waveform/aai records.  If you can arrange that each one updates with exactly matching timestamps (eg. copied with the TSEL link), that would provide sufficient information to reconstruct on retrieval.  Still extra work though.

In relation to how our record and device type is implemented, and how signal data is usually interpreted by clients, this would lead to significant deoptimisation and make database (unnecessarily?) complex.

Same behavior was observer, however, using "Tbl" PV from pva2pva's iocimagedemo.

We have trouble using EPICS Archiver (https://github.com/archiver-appliance/epicsarchiverap, v2.0.7, deployed using Docker singletomcat image): despite data being archived,

it cannot be  viewed using built-in viewer,

To my knowledge, none of the Archiver Appliance viewers/clients know how to handle arbitrary Structures.

Actually, this is probably not an issue. I assume builtin viewer is meant as a convenience, not primary, analysis tool. 

Thanks for all suggestions.

nor can we retrieve data using aaclient tools (v0.0.3, https://github.com/mdavidsaver/aaclient) - segfault occurs.

oops.  https ://github.com/mdavidsaver/aaclient/issues/1

That said, I'm not sure how this case should be handled.  So in the short term, I will just make this fail more obviously.

$ aaget ioc7:signals:dod:pva

Segmentation fault (core dumped)

$ aah5 test.h5 ioc7:signals:dod:pva

INFO:aaclient.cmd.h5:Writing /home/ales/test.h5

free(): corrupted unsorted chunks

Aborted (core dumped)

From PV details page in EPICS Archiver appliance:

Archiver DBR type (from typeinfo): DBR_V4_GENERIC_BYTES

Archiver DBR type (initial) DBR_V4_GENERIC_BYTES

Is this a scalar: Yes

Number of elements: 1

Data can be retrieved using PVA CLI tools, pvaPy and p4p.

$ pvinfo ioc7:signals:dod:pva

ioc7:signals:dod:pva

Server: 10.0.6.13:5075

Type:

    epics:nt/NTTable:1.0

        structure record

            structure _options

                int queueSize

                boolean atomic

        alarm_t alarm

            int severity

            int status

            string message

        time_t timeStamp

            long secondsPastEpoch

            int nanoseconds

            int userTag

        string[] labels

        structure value

            short[] A

            short[] B

            short[] C

            short[] D

$ pvget ioc7:signals:dod:pva

ioc7:signals:dod:pva 2025-03-14 10:17:41.584    

  A  B   C   D

108  9 118 102

108 10 119 102

111 11 121 105

108  9 119 102

104  6 115  98

106  7 116  99

108  9 119 102

...

>>> from p4p.client.thread import Context
>>> ctxt = Context('pva')

>>> sub = ctxt.monitor('ioc7:signals:dod:pva', lambda pv: print(pv.value['A'][1:5]))
>>> [109 110 109 106]
[109 108 105 108]
[105 108 111 106]
[110 108 104 105]
[109 107 107 112]
[106 102 110 112]
[105 109 110 105]
...
>>> sub = c.close()
...

Any information or directions for continuing will be appreciated,

BR, Ales