Experimental Physics and
Industrial Control System

On Jul 17, 2020, at 11:45 AM, Mark Rivers <rivers at cars.uchicago.edu> wrote:

Hi Valentyn,

 

Ø  I can imagine two different peripheral data collection strategies.

 

- First, my image collecting software subscribes to the PVs, and whenever I get the new image I save it and all last known values of PVs with it. 

- Second, I sent a request to an archiver(a place where all PV values are saved with according timestamps) that returns values at the requested time. The second one requires additional infrastructure, some kind of server(a middle man) that will collect all PV values and process request from clients.

 

Many beamlines use strategy #1 because it is very well supported in the EPICS areaDetector software.  It works like this:

 

- The areaDetector driver (marCCD, Pilatus, Point Grey, Eiger, etc.) collects data into NDArray class objects.

- NDArrays objects have a list of NDAttributes attached to them that contain the metadata. NDAttributes can get their values from any EPICS PV, in addition to other sources.

- Most areaDetector file saving plugins can save both the NDArray data as well as the NDAttribute metadata.  This includes the HDF5, Nexus, netCDF, and TIFF plugins.

 

The list of NDAttributes to be attached to each NDArray is specified in an XML file.

 

This is an example of part of such an XML file:

 

(base) corvette:CARS/iocBoot/ioc13bmd_pg1>more tomoDetectorAttributes.xml

<?xml version="1.0" standalone="no" ?>

<!-- Attributes -->

<Attributes>

    <Attribute name="DetectorManufacturer"          type="EPICS_PV"     source="$(DET)cam1:Manufacturer_RBV"             dbrtype="DBR_STRING"  description="Detector manufacturer"/>

    <Attribute name="DetectorModel"                 type="EPICS_PV"     source="$(DET)cam1:Model_RBV"                    dbrtype="DBR_STRING"  description="Detector model"/>

    <Attribute name="SerialNumber"                  type="EPICS_PV"     source="$(DET)cam1:SerialNumber_RBV"             dbrtype="DBR_STRING"  description="Detector serial number"/>

    <Attribute name="FirmwareVersion"               type="EPICS_PV"     source="$(DET)cam1:FirmwareVersion_RBV"          dbrtype="DBR_STRING"  description="Detector firmware version"/>

    <Attribute name="SDKVersion"                    type="EPICS_PV"     source="$(DET)cam1:SDKVersion_RBV"               dbrtype="DBR_STRING"  description="Detector vendor SDK version"/>

    <Attribute name="DriverVersion"                 type="EPICS_PV"     source="$(DET)cam1:DriverVersion_RBV"            dbrtype="DBR_STRING"  description="areaDetector driver version"/>

    <Attribute name="ADCoreVersion"                 type="EPICS_PV"     source="$(DET)cam1:ADCoreVersion_RBV"            dbrtype="DBR_STRING"  description="areaDetector ADCore version"/>

 

…

    <Attribute name="AcquireTime"                   type="EPICS_PV"     source="$(DET)cam1:AcquireTime_RBV"              dbrtype="DBR_NATIVE"  description="Exposure time"/>

    <Attribute name="AcquirePeriod"                 type="EPICS_PV"     source="$(DET)cam1:AcquirePeriod_RBV"            dbrtype="DBR_NATIVE"  description="Exposure period"/>

    <Attribute name="ExposurePeriod"                type="EPICS_PV"     source="13BMD:SIS1:Dwell"                        dbrtype="DBR_NATIVE"  description="SIS Dwell=exposure period"/>

    <Attribute name="FrameRate"                     type="EPICS_PV"     source="$(DET)cam1:FrameRate_RBV"                dbrtype="DBR_NATIVE"  description="Frame rate"/>

    <Attribute name="FrameRateEnable"               type="EPICS_PV"     source="$(DET)cam1:FrameRateEnable_RBV"          dbrtype="DBR_STRING"  description="Frame rate enable"/>

    <Attribute name="Gain"                          type="EPICS_PV"     source="$(DET)cam1:Gain_RBV"                     dbrtype="DBR_NATIVE"  description="Gain"/>

    <Attribute name="GainAuto"                      type="EPICS_PV"     source="$(DET)cam1:GainAuto_RBV"                 dbrtype="DBR_NATIVE"  description="Gain auto"/>

    <Attribute name="PixelFormat"                   type="EPICS_PV"     source="$(DET)cam1:PixelFormat_RBV"              dbrtype="DBR_STRING"  description="Readout pixel format"/>

    <Attribute name="ConvertPixelFormat"            type="EPICS_PV"     source="$(DET)cam1:ConvertPixelFormat_RBV"       dbrtype="DBR_STRING"  description="Converted pixel format"/>

    <Attribute name="ArrayCounter"                  type="EPICS_PV"     source="$(DET)cam1:ArrayCounter_RBV"             dbrtype="DBR_NATIVE"  description="Frames collected to date"/>

    <Attribute name="DetectorTemperature"           type="EPICS_PV"     source="$(DET)cam1:TemperatureActual"            dbrtype="DBR_NATIVE"  description="Detector temperature"/>

 

…

 

    <Attribute name="DateTimeStart"                 type="EPICS_PV"     source="S:IOC:timeOfDayForm1SI"                  dbrtype="DBR_STRING"  description="Date and time at start"/>

    <Attribute name="DateTimeEnd"                   type="EPICS_PV"     source="S:IOC:timeOfDayForm1SI"                  dbrtype="DBR_STRING"  description="Date and time at end"/>

    <Attribute name="RingCurrent"                   type="EPICS_PV"     source="S:SRcurrentAI"                           dbrtype="DBR_NATIVE"  description="Ring current"/>

    <Attribute name="TopUpStatus"                   type="EPICS_PV"     source="S:TopUpStatus"                           dbrtype="DBR_STRING"  description="Top-up status"/>

    <Attribute name="BeamMode"                      type="EPICS_PV"     source="OPS:message3"                            dbrtype="DBR_STRING"  description="Beam mode"/>

 

    <Attribute name="SampleX"                       type="EPICS_PV"     source="13BMD:m85.RBV"                           dbrtype="DBR_NATIVE"  description="Bottom X stage translation"/>

    <Attribute name="SampleOmega"                   type="EPICS_PV"     source="13BMD:m38.RBV"                           dbrtype="DBR_NATIVE"  description="Sample rotation"/>

    <Attribute name="SampleY"                       type="EPICS_PV"     source="13BMD:m90.VAL"                           dbrtype="DBR_NATIVE"  description="Sample vertical height"/>

    <Attribute name="SampleXCent"                   type="EPICS_PV"     source="13BMD:m92.VAL"                           dbrtype="DBR_NATIVE"  description="Sample X centering"/>

    <Attribute name="SampleZCent"                   type="EPICS_PV"     source="13BMD:m93.VAL"                           dbrtype="DBR_NATIVE"  description="Sample Z centering"/>

 

…

    <Attribute name="HDF5FrameLocation"             type="EPICS_PV"     source="$(TS)HDF5Location"                       dbrtype="DBR_STRING"  description="Where to store frame in HDF5 file"/>

 

</Attributes>

 

This is part of the output of the ncdump command on a netCDF file saved by areaDetector, looking at the value of the ring current attribute:

 

(base) corvette:~/tomo_data/NaCl_tomoscan>ncdump -v Attr_RingCurrent NaCl_C1.nc

netcdf NaCl_C1 {

dimensions:

        numArrays = UNLIMITED ; // (10 currently)

        dim0 = 1200 ;

        dim1 = 1920 ;

        attrStringSize = 256 ;

variables:

        int uniqueId(numArrays) ;

        double timeStamp(numArrays) ;

        int epicsTSSec(numArrays) ;

        int epicsTSNsec(numArrays) ;

        short array_data(numArrays, dim0, dim1) ;

        char Attr_DetectorManufacturer(numArrays, attrStringSize) ;

…

data:

 

Attr_RingCurrent = 101.876796241609, 101.876058881609, 101.876288921609,

    101.876337641609, 101.876575521609, 101.875805921609, 101.874051801609,

    101.875392481609, 101.874212281609, 101.874040321609 ;

 

In this case there are 10 values of the ring current because this file contains 10 images, and the ring current attribute is captured and stored for each one.

 

This is part of the output of the h5dump command on an HDF5 file saved by areaDetector, looking at the value of the ring current attribute:

 

(base) corvette:~/scratch>h5dump --dataset /RingCurrent hdf5_test_097.h5

HDF5 "hdf5_test_097.h5" {

DATASET "/RingCurrent" {

   DATATYPE  H5T_IEEE_F64LE

   DATASPACE  SIMPLE { ( 10 ) / ( H5S_UNLIMITED ) }

   DATA {

   (0): 102.062, 102.062, 102.062, 102.062, 102.062, 102.063, 102.063,

   (7): 102.063, 102.063, 102.063

   }

   ATTRIBUTE "NDAttrDescription" {

      DATATYPE  H5T_STRING {

         STRSIZE 20;

         STRPAD H5T_STR_NULLTERM;

         CSET H5T_CSET_ASCII;

         CTYPE H5T_C_S1;

      }

      DATASPACE  SCALAR

      DATA {

      (0): "Storage ring current"

      }

   }

   ATTRIBUTE "NDAttrName" {

      DATATYPE  H5T_STRING {

         STRSIZE 11;

         STRPAD H5T_STR_NULLTERM;

         CSET H5T_CSET_ASCII;

         CTYPE H5T_C_S1;

      }

      DATASPACE  SCALAR

      DATA {

      (0): "RingCurrent"

      }

  }

   ATTRIBUTE "NDAttrSource" {

      DATATYPE  H5T_STRING {

         STRSIZE 13;

         STRPAD H5T_STR_NULLTERM;

         CSET H5T_CSET_ASCII;

         CTYPE H5T_C_S1;

      }

      DATASPACE  SCALAR

      DATA {

      (0): "S:SRcurrentAI"

      }

   }

   ATTRIBUTE "NDAttrSourceType" {

      DATATYPE  H5T_STRING {

         STRSIZE 19;

         STRPAD H5T_STR_NULLTERM;

         CSET H5T_CSET_ASCII;

         CTYPE H5T_C_S1;

      }

      DATASPACE  SCALAR

      DATA {

      (0): "NDAttrSourceEPICSPV"

      }

   }

}

}

 

In this case there are also 10 values of the ring current because this file contains 10 images, and the ring current attribute is captured and stored for each one.

 

Mark

 

-----Original Message-----
From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Valentyn Stadnytskyi via Tech-talk
Sent: Friday, July 17, 2020 10:13 AM
To: tech-talk at aps.anl.gov
Subject: data acquisition strategies

 

Hello All,

I hope you all are doing well.

I am a user at BioCars. I have a few questions about data acquisition strategies. But before I ask them let me briefly describe a situation which might be very familiar to all of you.

I am collecting X-ray images (of whatever sample/ technique). I want to know and save other data associated with the image. For example, I want to save temperature, humidity, motor1 position, motor2 position, laser intensity and the pressure(and list can go on and on) at which the image was collected. Let us assume that all of these devices and DAQs are on the network and have PVs associated with the values I am seeking. I can imagine two different peripheral data collection strategies.

 

- First, my image collecting software subscribes to the PVs, and whenever I get the new image I save it and all last known values of PVs with it.

- Second, I sent a request to an archiver(a place where all PV values are saved with according timestamps) that returns values at the requested time. The second one requires additional infrastructure, some kind of server(a middle man) that will collect all PV values and process request from clients. 

 

How do you collect data from all additional devices\motors etc. Do you use strategy 1 or 2 or else? If you use strategy 1, how do you ensure that you don’t get very old PV value or you don’t get NAN. If subscription to PVs works well my main program should always have the most recent value. Is it correct? 

 

Can someone recommend something to read on how a beamline handles data collection?

Valentyn