RE: Using NDPluginGather to collect NDArrays from multiple detectors?

[Mark Rivers via Tech-talk <tech-talk at aps.anl.gov>]

Hi Jörn,

 

I just tested what you are trying to do with 3 ADSimDetector drivers running in the same IOC.  It worked fine.

 

I set each simDetector with AcquirePeriod=1 second, Continuous acquisition.  See first attachment.

 

I set the NDGather plugin to get its data from the 3 simDetectors.  See second attachment.  Note that ArrayRate=3 frames/s as expected.

 

I set the HDF plugin to get its data from GATHER1.  I set the WriteMode to Stream, NumCapture=100, and pressed Start.  Note that ArrayRate is also 3 for this plugin, which is expected.  It took 33 seconds to write the HDF5 file.  See third attachment, which was captured when it had saved 70/100 arrays.

 

This is the output of h5dump –contents on the resulting HDF5 file.

 

(base) [epics@corvette scratch]$ h5dump --contents Gather_test_024.h5

HDF5 "Gather_test_024.h5" {

FILE_CONTENTS {

group      /

group      /entry

group      /entry/data

dataset    /entry/data/data

group      /entry/instrument

group      /entry/instrument/NDAttributes

dataset    /entry/instrument/NDAttributes/NDArrayEpicsTSSec

dataset    /entry/instrument/NDAttributes/NDArrayEpicsTSnSec

dataset    /entry/instrument/NDAttributes/NDArrayTimeStamp

dataset    /entry/instrument/NDAttributes/NDArrayUniqueId

group      /entry/instrument/detector

group      /entry/instrument/detector/NDAttributes

dataset    /entry/instrument/detector/NDAttributes/ColorMode

dataset    /entry/instrument/detector/data -> /entry/data/data

group      /entry/instrument/performance

dataset    /entry/instrument/performance/timestamp

}

}

 

This is the output of h5dump –header for the /entry/data/ group.  Note that the dimensions are (100, 1024, 1024) as expected.

 

      GROUP "data" {

         ATTRIBUTE "NX_class" {

            DATATYPE  H5T_STRING {

               STRSIZE 7;

               STRPAD H5T_STR_NULLTERM;

               CSET H5T_CSET_ASCII;

               CTYPE H5T_C_S1;

            }

            DATASPACE  SCALAR

         }

         DATASET "data" {

            DATATYPE  H5T_STD_U8LE

            DATASPACE  SIMPLE { ( 100, 1024, 1024 ) / ( 100, 1024, 1024 ) }

            ATTRIBUTE "NDArrayDimBinning" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SIMPLE { ( 2 ) / ( 2 ) }

            }

            ATTRIBUTE "NDArrayDimOffset" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SIMPLE { ( 2 ) / ( 2 ) }

            }

            ATTRIBUTE "NDArrayDimReverse" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SIMPLE { ( 2 ) / ( 2 ) }

            }

            ATTRIBUTE "NDArrayNumDims" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SCALAR

            }

            ATTRIBUTE "signal" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SCALAR

            }

         }

      }

 

Mark

 

From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Tuesday, February 25, 2025 8:25 AM
To: Jörn Dreyer <j.dreyer at hzdr.de>; tech-talk at aps.anl.gov
Subject: Re: Using NDPluginGather to collect NDArrays from multiple detectors?

 

Hi Jörn,

 

Please set each camera so it is acquiring continuously at 1 Hz.  Then send a screen shot of the Gather plugin, which should have an Array Rate = 3.

 

Set the HDF5 plugin to Stream mode with NumCapture=100.  Then press the Capture Start button to start streaming.  It should save 3 arrays per second for 33 seconds and then close the file.  Send a screenshot of the HDF5 screen after you press the Capture Start button.

 

Mark

 

 

---

From: Jörn Dreyer <j.dreyer at hzdr.de>
Sent: Tuesday, February 25, 2025 2:25 AM
To: tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>; Mark Rivers <rivers at cars.uchicago.edu>
Subject: Re: Using NDPluginGather to collect NDArrays from multiple detectors?

 

Hi Mark,

I checked the requirements and all are fullfilled:

1) all detectors are created in the same IOC:

># Adding Basler cameras. Modify the second parameter acording to the camera name.
>aravisConfig("$(PORT1)", "Basler-acA640-100gm-xxxxxxxx")
>aravisConfig("$(PORT2)", "Basler-acA640-100gm-yyyyyyyy")
>aravisConfig("$(PORT3)", "Basler-acA640-120gm-zzzzzzzz")

2) All cameras are of the same type, so the image size is the same (659 * 494) and of the same datatype (UINT8).
The attributes are the standard set. I did not add any.


I attached the screenshots of my NDPluginGather and NDPluginFileHDF5 settings.
There are no errors reported from the HDF5 plugin, but still only one image shows up in the output.
If I do a dump of the file structure, I get:

h5dump -n test_007.h5
HDF5 "test_007.h5" {
FILE_CONTENTS {
 group      /
 group      /entry
 group      /entry/data
 dataset    /entry/data/data
 group      /entry/instrument
 group      /entry/instrument/NDAttributes
 dataset    /entry/instrument/NDAttributes/BayerPattern
 dataset    /entry/instrument/NDAttributes/NDArrayEpicsTSSec
 dataset    /entry/instrument/NDAttributes/NDArrayEpicsTSnSec
 dataset    /entry/instrument/NDAttributes/NDArrayTimeStamp
 dataset    /entry/instrument/NDAttributes/NDArrayUniqueId
 group      /entry/instrument/detector
 group      /entry/instrument/detector/NDAttributes
 dataset    /entry/instrument/detector/NDAttributes/ColorMode
 dataset    /entry/instrument/detector/data -> /entry/data/data
 group      /entry/instrument/performance
 dataset    /entry/instrument/performance/timestamp
 }
}

I tried the capture and the single mode. Always there is only one image in the file.
I used the commonPluginsExample.cmd with some modifications to be able to use the same file for all three cameras.
The Gather Plugin is from the first camera.

Regards,

Jörn
Am Mittwoch, 19. Februar 2025, 17:35:20 Mitteleuropäische Normalzeit schrieb Mark Rivers:
> Hi Jörn,
>
> In principle NDPluginGather can do what you are asking.  There are 2 restrictions:
>
>
>   1.
> All of the detectors must be running in the same IOC, i.e. the same process.
>   2.
> All of the detectors must be producing NDArrays with the same dimensions, data type, and attributes.
>
> The second restriction is due to the way the HDF5 plugin works.  In Stream mode it saves a single dataset of dimensions [Nx, Ny, Nimages].  The datatype of that dataset and also the attribute datasets are determined by the first array it receives.  All subsequent arrays must be the same.  If not, you should see an error at the IOC shell and on the OPI screen.
>
> If your setup meets these restrictions and you are still having problems, then please send screenshots of the Gather and HDF5 plugin screens.
>
> Mark
>
> ________________________________
> From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Jörn Dreyer via Tech-talk <tech-talk at aps.anl.gov>
> Sent: Wednesday, February 19, 2025 2:21 AM
> To: tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
> Subject: Using NDPluginGather to collect NDArrays from multiple detectors?
>
> Hi,
>
> I would like to write the images from multiple detectors into a single file.
> Therefor I tried to setup an NDPluginGather and connected the Array ports to the asynArray's of the detectors.
> The output of this was connected to an NDPluginFileHDF5. But in the resulting HDF5 file was only one image.
>
> I gues I misunderstood the functionality of NDPluginGather.
> Looking at the documentation it seems as if it seems that it can only collect data from Plugins (Attributes).
>
> Is there a way to achieve that, or do I have to live with one file per detector?
>
> Regards
>
> Jörn
>
>
>


--
Dr. Jörn Dreyer
HIBEF DAQ + Controls
Institut für Strahlenpysik
Abt. FWKX
Tel: +49 351 260 3263
Fax: +49 351 260 123263

Helmholtz-Zentrum Dresden - Rossendorf e.V. (HZDR)
Bautzner Landstr. 400 | 01328 Dresden | Germany
Tel: +49 (0) 351 260 1234
http://www.hzdr.de
Vorstand: Prof. Dr. Sebastian M. Schmidt, Dr. Diana Stiller
Vereinsregister: VR 1693 beim Amtsgericht Dresden

Attachment: simDetector_cam1_Gather.png
Description: simDetector_cam1_Gather.png

Attachment: Gather_Gather.png
Description: Gather_Gather.png

Attachment: HDF5_Gather.png
Description: HDF5_Gather.png