RE: EPICS module/service that can be used for saving waveform data/metadata to HDF5

["Pearson, Matthew via Tech-talk" <tech-talk at aps.anl.gov>]

Hi Jakub,

 

There is an areaDetector driver that might help:

https://github.com/areaDetector/NDDriverStdArrays

 

That allows you to read any waveform and effectively turn it into a NDArray object, then use areaDetector file saving (with metadata handled in the usual way).

 

I also think bluesky really needs a client/server model, with the experiment logic run remotely on a server. Last time I checked, it didn’t have this feature, but I now see there is something called ‘Bluesky Queue Server’. So would that be a solution, with the server process run under a privileged account with write-access to the data storage?

 

Cheers,

Matt

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Wlodek, Jakub via Tech-talk
Sent: Tuesday, June 11, 2024 9:22 AM
To: tech-talk at aps.anl.gov
Subject: [EXTERNAL] EPICS module/service that can be used for saving waveform data/metadata to HDF5

 

Hi all,

 

During the EPICS codeathon this week at NSLS-II I mentioned a need for us here at NSLS-II w.r.t. data saving. Essentially, we are beginning to work on hardening the data access security for scientific data, one consequence of this being that operator accounts that users run their experiments under will no longer have r/w access to data directories, and all data needs to be written by either a softioc service account, or by a workflow service account.

 

The problem with this, is that for a fair number of detectors/devices that did not have any in-built file saving capabilities (basically anything that isn't an areaDetector driver), the previous approach was to basically caget​ the waveform data and any metadata signals, and in the bluesky experiment orchestration layer use the h5py library to save the data, all as this operator account. Since this approach will no longer work due to the new permissions, to work around this, we've been moving this code into python IOCs that expose some PVs that are then used to more or less call the same functions from bluesky - just now with the file saving done as a softioc account. However, I don't think this is an ideal solution - it is leading to a lot of one-off IOCs for each of these devices.

 

I've been thinking of ways to build something more general that will allow for specifying a waveform record and some additional metadata and saving it when asked, and it was suggested to me at the codeathon to reach out here to the community to see if others have a similar need or ideas on how to handle this.

 

Thanks!

Jakub