Hi Valentyn,
Ø
I can imagine two different peripheral data collection strategies.
- First, my image collecting software subscribes to the PVs, and whenever I get the new image I save it and all last known values of PVs with
it.
- Second, I sent a request to an archiver(a place where all PV values are saved with according timestamps) that returns values at the requested
time. The second one requires additional infrastructure, some kind of server(a middle man) that will collect all PV values and process request from clients.
Many beamlines use strategy #1 because it is very well supported in the EPICS areaDetector software. It works like this:
- The areaDetector driver (marCCD, Pilatus, Point Grey, Eiger, etc.) collects data into NDArray class objects.
- NDArrays objects have a list of NDAttributes attached to them that contain the metadata. NDAttributes can get their values from any EPICS PV, in addition to other sources.
- Most areaDetector file saving plugins can save both the NDArray data as well as the NDAttribute metadata. This includes the HDF5, Nexus, netCDF, and TIFF plugins.
The list of NDAttributes to be attached to each NDArray is specified in an XML file.
This is an example of part of such an XML file:
(base) corvette:CARS/iocBoot/ioc13bmd_pg1>more tomoDetectorAttributes.xml
<?xml version="1.0" standalone="no" ?>
<!-- Attributes -->
<Attributes>
<Attribute name="DetectorManufacturer" type="EPICS_PV" source="$(DET)cam1:Manufacturer_RBV" dbrtype="DBR_STRING" description="Detector manufacturer"/>
<Attribute name="DetectorModel" type="EPICS_PV" source="$(DET)cam1:Model_RBV" dbrtype="DBR_STRING" description="Detector model"/>
<Attribute name="SerialNumber" type="EPICS_PV" source="$(DET)cam1:SerialNumber_RBV" dbrtype="DBR_STRING" description="Detector serial number"/>
<Attribute name="FirmwareVersion" type="EPICS_PV" source="$(DET)cam1:FirmwareVersion_RBV" dbrtype="DBR_STRING" description="Detector firmware version"/>
<Attribute name="SDKVersion" type="EPICS_PV" source="$(DET)cam1:SDKVersion_RBV" dbrtype="DBR_STRING" description="Detector vendor SDK
version"/>
<Attribute name="DriverVersion" type="EPICS_PV" source="$(DET)cam1:DriverVersion_RBV" dbrtype="DBR_STRING" description="areaDetector driver
version"/>
<Attribute name="ADCoreVersion" type="EPICS_PV" source="$(DET)cam1:ADCoreVersion_RBV" dbrtype="DBR_STRING" description="areaDetector ADCore
version"/>
…
<Attribute name="AcquireTime" type="EPICS_PV" source="$(DET)cam1:AcquireTime_RBV" dbrtype="DBR_NATIVE" description="Exposure time"/>
<Attribute name="AcquirePeriod" type="EPICS_PV" source="$(DET)cam1:AcquirePeriod_RBV" dbrtype="DBR_NATIVE" description="Exposure period"/>
<Attribute name="ExposurePeriod" type="EPICS_PV" source="13BMD:SIS1:Dwell" dbrtype="DBR_NATIVE" description="SIS Dwell=exposure period"/>
<Attribute name="FrameRate" type="EPICS_PV" source="$(DET)cam1:FrameRate_RBV" dbrtype="DBR_NATIVE" description="Frame rate"/>
<Attribute name="FrameRateEnable" type="EPICS_PV" source="$(DET)cam1:FrameRateEnable_RBV" dbrtype="DBR_STRING" description="Frame rate enable"/>
<Attribute name="Gain" type="EPICS_PV" source="$(DET)cam1:Gain_RBV" dbrtype="DBR_NATIVE" description="Gain"/>
<Attribute name="GainAuto" type="EPICS_PV" source="$(DET)cam1:GainAuto_RBV" dbrtype="DBR_NATIVE" description="Gain auto"/>
<Attribute name="PixelFormat" type="EPICS_PV" source="$(DET)cam1:PixelFormat_RBV" dbrtype="DBR_STRING" description="Readout pixel format"/>
<Attribute name="ConvertPixelFormat" type="EPICS_PV" source="$(DET)cam1:ConvertPixelFormat_RBV" dbrtype="DBR_STRING" description="Converted pixel format"/>
<Attribute name="ArrayCounter" type="EPICS_PV" source="$(DET)cam1:ArrayCounter_RBV" dbrtype="DBR_NATIVE" description="Frames collected to
date"/>
<Attribute name="DetectorTemperature" type="EPICS_PV" source="$(DET)cam1:TemperatureActual" dbrtype="DBR_NATIVE" description="Detector temperature"/>
…
<Attribute name="DateTimeStart" type="EPICS_PV" source="S:IOC:timeOfDayForm1SI" dbrtype="DBR_STRING" description="Date and time at start"/>
<Attribute name="DateTimeEnd" type="EPICS_PV" source="S:IOC:timeOfDayForm1SI" dbrtype="DBR_STRING" description="Date and time at end"/>
<Attribute name="RingCurrent" type="EPICS_PV" source="S:SRcurrentAI" dbrtype="DBR_NATIVE" description="Ring current"/>
<Attribute name="TopUpStatus" type="EPICS_PV" source="S:TopUpStatus" dbrtype="DBR_STRING" description="Top-up status"/>
<Attribute name="BeamMode" type="EPICS_PV" source="OPS:message3" dbrtype="DBR_STRING" description="Beam mode"/>
<Attribute name="SampleX" type="EPICS_PV" source="13BMD:m85.RBV" dbrtype="DBR_NATIVE" description="Bottom X stage translation"/>
<Attribute name="SampleOmega" type="EPICS_PV" source="13BMD:m38.RBV" dbrtype="DBR_NATIVE" description="Sample rotation"/>
<Attribute name="SampleY" type="EPICS_PV" source="13BMD:m90.VAL" dbrtype="DBR_NATIVE" description="Sample vertical height"/>
<Attribute name="SampleXCent" type="EPICS_PV" source="13BMD:m92.VAL" dbrtype="DBR_NATIVE" description="Sample X centering"/>
<Attribute name="SampleZCent" type="EPICS_PV" source="13BMD:m93.VAL" dbrtype="DBR_NATIVE" description="Sample Z centering"/>
…
<Attribute name="HDF5FrameLocation" type="EPICS_PV" source="$(TS)HDF5Location" dbrtype="DBR_STRING" description="Where to store frame
in HDF5 file"/>
</Attributes>
This is part of the output of the ncdump command on a netCDF file saved by areaDetector, looking at the value of the ring current attribute:
(base) corvette:~/tomo_data/NaCl_tomoscan>ncdump -v Attr_RingCurrent NaCl_C1.nc
netcdf NaCl_C1 {
dimensions:
numArrays = UNLIMITED ; // (10 currently)
dim0 = 1200 ;
dim1 = 1920 ;
attrStringSize = 256 ;
variables:
int uniqueId(numArrays) ;
double timeStamp(numArrays) ;
int epicsTSSec(numArrays) ;
int epicsTSNsec(numArrays) ;
short array_data(numArrays, dim0, dim1) ;
char Attr_DetectorManufacturer(numArrays, attrStringSize) ;
…
data:
Attr_RingCurrent = 101.876796241609, 101.876058881609, 101.876288921609,
101.876337641609, 101.876575521609, 101.875805921609, 101.874051801609,
101.875392481609, 101.874212281609, 101.874040321609 ;
In this case there are 10 values of the ring current because this file contains 10 images, and the ring current attribute is captured and stored for each one.
This is part of the output of the h5dump command on an HDF5 file saved by areaDetector, looking at the value of the ring current attribute:
(base) corvette:~/scratch>h5dump --dataset /RingCurrent hdf5_test_097.h5
HDF5 "hdf5_test_097.h5" {
DATASET "/RingCurrent" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 10 ) / ( H5S_UNLIMITED ) }
DATA {
(0): 102.062, 102.062, 102.062, 102.062, 102.062, 102.063, 102.063,
(7): 102.063, 102.063, 102.063
}
ATTRIBUTE "NDAttrDescription" {
DATATYPE H5T_STRING {
STRSIZE 20;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
(0): "Storage ring current"
}
}
ATTRIBUTE "NDAttrName" {
DATATYPE H5T_STRING {
STRSIZE 11;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
(0): "RingCurrent"
}
}
ATTRIBUTE "NDAttrSource" {
DATATYPE H5T_STRING {
STRSIZE 13;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
(0): "S:SRcurrentAI"
}
}
ATTRIBUTE "NDAttrSourceType" {
DATATYPE H5T_STRING {
STRSIZE 19;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
(0): "NDAttrSourceEPICSPV"
}
}
}
}
In this case there are also 10 values of the ring current because this file contains 10 images, and the ring current attribute is captured and stored for each one.
Mark
-----Original Message-----
From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Valentyn Stadnytskyi via Tech-talk
Sent: Friday, July 17, 2020 10:13 AM
To: tech-talk at aps.anl.gov
Subject: data acquisition strategies
Hello All,
I hope you all are doing well.
I am a user at BioCars. I have a few questions about data acquisition strategies. But before I ask them let me briefly describe a situation which might be very familiar to all of you.
I am collecting X-ray images (of whatever sample/ technique). I want to know and save other data associated with the image. For example, I want to save temperature, humidity, motor1 position, motor2 position, laser intensity and the
pressure(and list can go on and on) at which the image was collected. Let us assume that all of these devices and DAQs are on the network and have PVs associated with the values I am seeking. I can imagine two different peripheral data collection strategies.
- First, my image collecting software subscribes to the PVs, and whenever I get the new image I save it and all last known values of PVs with it.
- Second, I sent a request to an archiver(a place where all PV values are saved with according timestamps) that returns values at the requested time. The second one requires additional infrastructure, some kind of server(a middle man)
that will collect all PV values and process request from clients.
How do you collect data from all additional devices\motors etc. Do you use strategy 1 or 2 or else? If you use strategy 1, how do you ensure that you don’t get very old PV value or you don’t get NAN. If subscription to PVs works well
my main program should always have the most recent value. Is it correct?
Can someone recommend something to read on how a beamline handles data collection?
Valentyn