Re: Structured PVs archiving and archived data retrieval problem

[Sky Brewer via Tech-talk <tech-talk at aps.anl.gov>]

Hi Ales,

 

  How complete is support for structured PVs in EPICS archiver appliance and aaclient tools (more details below)?

 

Archiver Appliance 1.1.0 had support for storing all pvAccess structures and some minimal retrieval in json. Archiver Appliance 2.0.5+ has support for storing all pvAccess structures and I believe complete retrieval with json.

 

To fetch the data from the archiver using json you go to something like:

 

http://hostname:17668/retrieval/data/getData.json?pv=pvName&from=2025-03-18T08:57:20.000Z&to=2025-03-18T09:57:22.734Z&fetchLatestMetadata=true

 

I think the postprocessors that should work are:

• lastSample
• firstSample
• count
• ncount

 

I agree with Michael that I don’t think any client has support for these structures, but it should be possible to add support. The data is stored as the pvAccess structure then the data itself, so to unpack the data is quite easy. The Archiver uses https://github.com/archiver-appliance/epicsarchiverap/blob/master/src/main/org/epics/archiverappliance/data/ByteBufSampleValue.java to convert the structure to json.

 

Maybe I’ll add it to py-epicsarchiver….

 

But I agree with Michael for NTTable its best to split up and make a pv for each array to be archived, as generally it works better with all the tools.

 

In terms of alternatives the Archiver Appliance, I would like to ask if you want a new feature or improvement to the Archiver Appliance to please open tickets! https://github.com/archiver-appliance/epicsarchiverap/issues And or support more development! I think adding support for more Normative Types makes sense, but its good to get a ticket to follow up on testing etc.

 

But to be fair when I did a review of alternatives I found:

• https://link.springer.com/article/10.1007/s41605-021-00277-2  - China ADS front-end demo linac (CAFe) 
• https://indico.fnal.gov/event/58280/contributions/264774/ - Osprey
• https://github.com/paulscherrerinstitute/daqingest/tree/dev - PSI (Channel access only?)
• https://gitlab.esss.lu.se/ics-software/sds - SDS focused on waveforms and data triggered by a PV
• https://indico.fnal.gov/event/58280/contributions/264782/attachments/166397/221584/EPICS2023_MLIntro_v3.pdf - Retreiving data for Machine learning (similar to SDS project)
• https://github.com/ControlSystemStudio/phoebus/tree/master/services/archive-engine Archive Engine

Hope that helps,

 

Sky