EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  <20202021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  <20202021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: RE: data acquisition strategies
From: Mark Rivers via Tech-talk <tech-talk at aps.anl.gov>
To: 'Valentyn Stadnytskyi' <v.stadnytskyi at gmail.com>
Cc: "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov>
Date: Fri, 17 Jul 2020 15:45:00 +0000

Hi Valentyn,

 

Ø  I can imagine two different peripheral data collection strategies.

 

- First, my image collecting software subscribes to the PVs, and whenever I get the new image I save it and all last known values of PVs with it.

- Second, I sent a request to an archiver(a place where all PV values are saved with according timestamps) that returns values at the requested time. The second one requires additional infrastructure, some kind of server(a middle man) that will collect all PV values and process request from clients.

 

Many beamlines use strategy #1 because it is very well supported in the EPICS areaDetector software.  It works like this:

 

- The areaDetector driver (marCCD, Pilatus, Point Grey, Eiger, etc.) collects data into NDArray class objects.

- NDArrays objects have a list of NDAttributes attached to them that contain the metadata. NDAttributes can get their values from any EPICS PV, in addition to other sources.

- Most areaDetector file saving plugins can save both the NDArray data as well as the NDAttribute metadata.  This includes the HDF5, Nexus, netCDF, and TIFF plugins.

 

The list of NDAttributes to be attached to each NDArray is specified in an XML file.

 

This is an example of part of such an XML file:

 

(base) corvette:CARS/iocBoot/ioc13bmd_pg1>more tomoDetectorAttributes.xml

<?xml version="1.0" standalone="no" ?>

<!-- Attributes -->

<Attributes>

    <Attribute name="DetectorManufacturer"          type="EPICS_PV"     source="$(DET)cam1:Manufacturer_RBV"             dbrtype="DBR_STRING"  description="Detector manufacturer"/>

    <Attribute name="DetectorModel"                 type="EPICS_PV"     source="$(DET)cam1:Model_RBV"                    dbrtype="DBR_STRING"  description="Detector model"/>

    <Attribute name="SerialNumber"                  type="EPICS_PV"     source="$(DET)cam1:SerialNumber_RBV"             dbrtype="DBR_STRING"  description="Detector serial number"/>

    <Attribute name="FirmwareVersion"               type="EPICS_PV"     source="$(DET)cam1:FirmwareVersion_RBV"          dbrtype="DBR_STRING"  description="Detector firmware version"/>

    <Attribute name="SDKVersion"                    type="EPICS_PV"     source="$(DET)cam1:SDKVersion_RBV"               dbrtype="DBR_STRING"  description="Detector vendor SDK version"/>

    <Attribute name="DriverVersion"                 type="EPICS_PV"     source="$(DET)cam1:DriverVersion_RBV"            dbrtype="DBR_STRING"  description="areaDetector driver version"/>

    <Attribute name="ADCoreVersion"                 type="EPICS_PV"     source="$(DET)cam1:ADCoreVersion_RBV"            dbrtype="DBR_STRING"  description="areaDetector ADCore version"/>

 

    <Attribute name="AcquireTime"                   type="EPICS_PV"     source="$(DET)cam1:AcquireTime_RBV"              dbrtype="DBR_NATIVE"  description="Exposure time"/>

    <Attribute name="AcquirePeriod"                 type="EPICS_PV"     source="$(DET)cam1:AcquirePeriod_RBV"            dbrtype="DBR_NATIVE"  description="Exposure period"/>

    <Attribute name="ExposurePeriod"                type="EPICS_PV"     source="13BMD:SIS1:Dwell"                        dbrtype="DBR_NATIVE"  description="SIS Dwell=exposure period"/>

    <Attribute name="FrameRate"                     type="EPICS_PV"     source="$(DET)cam1:FrameRate_RBV"                dbrtype="DBR_NATIVE"  description="Frame rate"/>

    <Attribute name="FrameRateEnable"               type="EPICS_PV"     source="$(DET)cam1:FrameRateEnable_RBV"          dbrtype="DBR_STRING"  description="Frame rate enable"/>

    <Attribute name="Gain"                          type="EPICS_PV"     source="$(DET)cam1:Gain_RBV"                     dbrtype="DBR_NATIVE"  description="Gain"/>

    <Attribute name="GainAuto"                      type="EPICS_PV"     source="$(DET)cam1:GainAuto_RBV"                 dbrtype="DBR_NATIVE"  description="Gain auto"/>

    <Attribute name="PixelFormat"                   type="EPICS_PV"     source="$(DET)cam1:PixelFormat_RBV"              dbrtype="DBR_STRING"  description="Readout pixel format"/>

    <Attribute name="ConvertPixelFormat"            type="EPICS_PV"     source="$(DET)cam1:ConvertPixelFormat_RBV"       dbrtype="DBR_STRING"  description="Converted pixel format"/>

    <Attribute name="ArrayCounter"                  type="EPICS_PV"     source="$(DET)cam1:ArrayCounter_RBV"             dbrtype="DBR_NATIVE"  description="Frames collected to date"/>

    <Attribute name="DetectorTemperature"           type="EPICS_PV"     source="$(DET)cam1:TemperatureActual"            dbrtype="DBR_NATIVE"  description="Detector temperature"/>

 

 

    <Attribute name="DateTimeStart"                 type="EPICS_PV"     source="S:IOC:timeOfDayForm1SI"                  dbrtype="DBR_STRING"  description="Date and time at start"/>

    <Attribute name="DateTimeEnd"                   type="EPICS_PV"     source="S:IOC:timeOfDayForm1SI"                  dbrtype="DBR_STRING"  description="Date and time at end"/>

    <Attribute name="RingCurrent"                   type="EPICS_PV"     source="S:SRcurrentAI"                           dbrtype="DBR_NATIVE"  description="Ring current"/>

    <Attribute name="TopUpStatus"                   type="EPICS_PV"     source="S:TopUpStatus"                           dbrtype="DBR_STRING"  description="Top-up status"/>

    <Attribute name="BeamMode"                      type="EPICS_PV"     source="OPS:message3"                            dbrtype="DBR_STRING"  description="Beam mode"/>

 

    <Attribute name="SampleX"                       type="EPICS_PV"     source="13BMD:m85.RBV"                           dbrtype="DBR_NATIVE"  description="Bottom X stage translation"/>

    <Attribute name="SampleOmega"                   type="EPICS_PV"     source="13BMD:m38.RBV"                           dbrtype="DBR_NATIVE"  description="Sample rotation"/>

    <Attribute name="SampleY"                       type="EPICS_PV"     source="13BMD:m90.VAL"                           dbrtype="DBR_NATIVE"  description="Sample vertical height"/>

    <Attribute name="SampleXCent"                   type="EPICS_PV"     source="13BMD:m92.VAL"                           dbrtype="DBR_NATIVE"  description="Sample X centering"/>

    <Attribute name="SampleZCent"                   type="EPICS_PV"     source="13BMD:m93.VAL"                           dbrtype="DBR_NATIVE"  description="Sample Z centering"/>

 

    <Attribute name="HDF5FrameLocation"             type="EPICS_PV"     source="$(TS)HDF5Location"                       dbrtype="DBR_STRING"  description="Where to store frame in HDF5 file"/>

 

</Attributes>

 

This is part of the output of the ncdump command on a netCDF file saved by areaDetector, looking at the value of the ring current attribute:

 

(base) corvette:~/tomo_data/NaCl_tomoscan>ncdump -v Attr_RingCurrent NaCl_C1.nc

netcdf NaCl_C1 {

dimensions:

        numArrays = UNLIMITED ; // (10 currently)

        dim0 = 1200 ;

        dim1 = 1920 ;

        attrStringSize = 256 ;

variables:

        int uniqueId(numArrays) ;

        double timeStamp(numArrays) ;

        int epicsTSSec(numArrays) ;

        int epicsTSNsec(numArrays) ;

        short array_data(numArrays, dim0, dim1) ;

        char Attr_DetectorManufacturer(numArrays, attrStringSize) ;

data:

 

Attr_RingCurrent = 101.876796241609, 101.876058881609, 101.876288921609,

    101.876337641609, 101.876575521609, 101.875805921609, 101.874051801609,

    101.875392481609, 101.874212281609, 101.874040321609 ;

 

In this case there are 10 values of the ring current because this file contains 10 images, and the ring current attribute is captured and stored for each one.

 

This is part of the output of the h5dump command on an HDF5 file saved by areaDetector, looking at the value of the ring current attribute:

 

(base) corvette:~/scratch>h5dump --dataset /RingCurrent hdf5_test_097.h5

HDF5 "hdf5_test_097.h5" {

DATASET "/RingCurrent" {

   DATATYPE  H5T_IEEE_F64LE

   DATASPACE  SIMPLE { ( 10 ) / ( H5S_UNLIMITED ) }

   DATA {

   (0): 102.062, 102.062, 102.062, 102.062, 102.062, 102.063, 102.063,

   (7): 102.063, 102.063, 102.063

   }

   ATTRIBUTE "NDAttrDescription" {

      DATATYPE  H5T_STRING {

         STRSIZE 20;

         STRPAD H5T_STR_NULLTERM;

         CSET H5T_CSET_ASCII;

         CTYPE H5T_C_S1;

      }

      DATASPACE  SCALAR

      DATA {

      (0): "Storage ring current"

      }

   }

   ATTRIBUTE "NDAttrName" {

      DATATYPE  H5T_STRING {

         STRSIZE 11;

         STRPAD H5T_STR_NULLTERM;

         CSET H5T_CSET_ASCII;

         CTYPE H5T_C_S1;

      }

      DATASPACE  SCALAR

      DATA {

      (0): "RingCurrent"

      }

  }

   ATTRIBUTE "NDAttrSource" {

      DATATYPE  H5T_STRING {

         STRSIZE 13;

         STRPAD H5T_STR_NULLTERM;

         CSET H5T_CSET_ASCII;

         CTYPE H5T_C_S1;

      }

      DATASPACE  SCALAR

      DATA {

      (0): "S:SRcurrentAI"

      }

   }

   ATTRIBUTE "NDAttrSourceType" {

      DATATYPE  H5T_STRING {

         STRSIZE 19;

         STRPAD H5T_STR_NULLTERM;

         CSET H5T_CSET_ASCII;

         CTYPE H5T_C_S1;

      }

      DATASPACE  SCALAR

      DATA {

      (0): "NDAttrSourceEPICSPV"

      }

   }

}

}

 

In this case there are also 10 values of the ring current because this file contains 10 images, and the ring current attribute is captured and stored for each one.

 

Mark

 

-----Original Message-----
From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Valentyn Stadnytskyi via Tech-talk
Sent: Friday, July 17, 2020 10:13 AM
To: tech-talk at aps.anl.gov
Subject: data acquisition strategies

 

Hello All,

I hope you all are doing well.

I am a user at BioCars. I have a few questions about data acquisition strategies. But before I ask them let me briefly describe a situation which might be very familiar to all of you.

I am collecting X-ray images (of whatever sample/ technique). I want to know and save other data associated with the image. For example, I want to save temperature, humidity, motor1 position, motor2 position, laser intensity and the pressure(and list can go on and on) at which the image was collected. Let us assume that all of these devices and DAQs are on the network and have PVs associated with the values I am seeking. I can imagine two different peripheral data collection strategies.

 

- First, my image collecting software subscribes to the PVs, and whenever I get the new image I save it and all last known values of PVs with it.

- Second, I sent a request to an archiver(a place where all PV values are saved with according timestamps) that returns values at the requested time. The second one requires additional infrastructure, some kind of server(a middle man) that will collect all PV values and process request from clients.

 

How do you collect data from all additional devices\motors etc. Do you use strategy 1 or 2 or else? If you use strategy 1, how do you ensure that you don’t get very old PV value or you don’t get NAN. If subscription to PVs works well my main program should always have the most recent value. Is it correct?

 

Can someone recommend something to read on how a beamline handles data collection?

Valentyn


Replies:
Re: data acquisition strategies Valentyn Stadnytskyi via Tech-talk
References:
data acquisition strategies Valentyn Stadnytskyi via Tech-talk

Navigate by Date:
Prev: data acquisition strategies Valentyn Stadnytskyi via Tech-talk
Next: Re: Pure Python IOC (CAProcess issue) Ralph Lange via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  <20202021  2022  2023  2024 
Navigate by Thread:
Prev: data acquisition strategies Valentyn Stadnytskyi via Tech-talk
Next: Re: data acquisition strategies Valentyn Stadnytskyi via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  <20202021  2022  2023  2024 
ANJ, 19 Jul 2020 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·