Re: Using NDPluginGather to collect NDArrays from multiple detectors?

[Jörn Dreyer via Tech-talk <tech-talk at aps.anl.gov>]

Hi Mark,

the usage of h5dump fooled me. After installing hdf5view and looking to the slices of the file, I found all my three images as expected.
Maybe the documentation of the HDF5 plugin should note that the image is stored in a slice and if multiple arrays get feed into the plugin
they show up in different slices of the data array. In my case 0,2,3... Somehow slice 1,5,... alway are empty in my case.

Regards and sorry for the confusion this might have caused,

Jörn

Am Dienstag, 25. Februar 2025, 18:27:50 Mitteleuropäische Normalzeit schrieben Sie:
> Hi Jörn,
> 
> I just tested what you are trying to do with 3 ADSimDetector drivers running in the same IOC.  It worked fine.
> 
> I set each simDetector with AcquirePeriod=1 second, Continuous acquisition.  See first attachment.
> 
> I set the NDGather plugin to get its data from the 3 simDetectors.  See second attachment.  Note that ArrayRate=3 frames/s as expected.
> 
> I set the HDF plugin to get its data from GATHER1.  I set the WriteMode to Stream, NumCapture=100, and pressed Start.  Note that ArrayRate is also 3 for this plugin, which is expected.  It took 33 seconds to write the HDF5 file.  See third attachment, which was captured when it had saved 70/100 arrays.
> 
> This is the output of h5dump -contents on the resulting HDF5 file.
> 
> (base) [epics@corvette scratch]$ h5dump --contents Gather_test_024.h5
> HDF5 "Gather_test_024.h5" {
> FILE_CONTENTS {
> group      /
> group      /entry
> group      /entry/data
> dataset    /entry/data/data
> group      /entry/instrument
> group      /entry/instrument/NDAttributes
> dataset    /entry/instrument/NDAttributes/NDArrayEpicsTSSec
> dataset    /entry/instrument/NDAttributes/NDArrayEpicsTSnSec
> dataset    /entry/instrument/NDAttributes/NDArrayTimeStamp
> dataset    /entry/instrument/NDAttributes/NDArrayUniqueId
> group      /entry/instrument/detector
> group      /entry/instrument/detector/NDAttributes
> dataset    /entry/instrument/detector/NDAttributes/ColorMode
> dataset    /entry/instrument/detector/data -> /entry/data/data
> group      /entry/instrument/performance
> dataset    /entry/instrument/performance/timestamp
> }
> }
> 
> This is the output of h5dump -header for the /entry/data/ group.  Note that the dimensions are (100, 1024, 1024) as expected.
> 
>       GROUP "data" {
>          ATTRIBUTE "NX_class" {
>             DATATYPE  H5T_STRING {
>                STRSIZE 7;
>                STRPAD H5T_STR_NULLTERM;
>                CSET H5T_CSET_ASCII;
>                CTYPE H5T_C_S1;
>             }
>             DATASPACE  SCALAR
>          }
>          DATASET "data" {
>             DATATYPE  H5T_STD_U8LE
>             DATASPACE  SIMPLE { ( 100, 1024, 1024 ) / ( 100, 1024, 1024 ) }
>             ATTRIBUTE "NDArrayDimBinning" {
>                DATATYPE  H5T_STD_I32LE
>                DATASPACE  SIMPLE { ( 2 ) / ( 2 ) }
>             }
>             ATTRIBUTE "NDArrayDimOffset" {
>                DATATYPE  H5T_STD_I32LE
>                DATASPACE  SIMPLE { ( 2 ) / ( 2 ) }
>             }
>             ATTRIBUTE "NDArrayDimReverse" {
>                DATATYPE  H5T_STD_I32LE
>                DATASPACE  SIMPLE { ( 2 ) / ( 2 ) }
>             }
>             ATTRIBUTE "NDArrayNumDims" {
>                DATATYPE  H5T_STD_I32LE
>                DATASPACE  SCALAR
>             }
>             ATTRIBUTE "signal" {
>                DATATYPE  H5T_STD_I32LE
>                DATASPACE  SCALAR
>             }
>          }
>       }
> 
> Mark
> 
> From: Mark Rivers <rivers at cars.uchicago.edu>
> Sent: Tuesday, February 25, 2025 8:25 AM
> To: Jörn Dreyer <j.dreyer at hzdr.de>; tech-talk at aps.anl.gov
> Subject: Re: Using NDPluginGather to collect NDArrays from multiple detectors?
> 
> Hi Jörn,
> 
> Please set each camera so it is acquiring continuously at 1 Hz.  Then send a screen shot of the Gather plugin, which should have an Array Rate = 3.
> 
> Set the HDF5 plugin to Stream mode with NumCapture=100.  Then press the Capture Start button to start streaming.  It should save 3 arrays per second for 33 seconds and then close the file.  Send a screenshot of the HDF5 screen after you press the Capture Start button.
> 
> Mark
> 
> 
> ________________________________
> From: Jörn Dreyer <j.dreyer at hzdr.de<mailto:j.dreyer at hzdr.de>>
> Sent: Tuesday, February 25, 2025 2:25 AM
> To: tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov> <tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>>; Mark Rivers <rivers at cars.uchicago.edu<mailto:rivers at cars.uchicago.edu>>
> Subject: Re: Using NDPluginGather to collect NDArrays from multiple detectors?
> 
> Hi Mark,
> 
> I checked the requirements and all are fullfilled:
> 
> 1) all detectors are created in the same IOC:
> 
> ># Adding Basler cameras. Modify the second parameter acording to the camera name.
> >aravisConfig("$(PORT1)", "Basler-acA640-100gm-xxxxxxxx")
> >aravisConfig("$(PORT2)", "Basler-acA640-100gm-yyyyyyyy")
> >aravisConfig("$(PORT3)", "Basler-acA640-120gm-zzzzzzzz")
> 
> 2) All cameras are of the same type, so the image size is the same (659 * 494) and of the same datatype (UINT8).
> The attributes are the standard set. I did not add any.
> 
> 
> I attached the screenshots of my NDPluginGather and NDPluginFileHDF5 settings.
> There are no errors reported from the HDF5 plugin, but still only one image shows up in the output.
> If I do a dump of the file structure, I get:
> 
> h5dump -n test_007.h5
> HDF5 "test_007.h5" {
> FILE_CONTENTS {
>  group      /
>  group      /entry
>  group      /entry/data
>  dataset    /entry/data/data
>  group      /entry/instrument
>  group      /entry/instrument/NDAttributes
>  dataset    /entry/instrument/NDAttributes/BayerPattern
>  dataset    /entry/instrument/NDAttributes/NDArrayEpicsTSSec
>  dataset    /entry/instrument/NDAttributes/NDArrayEpicsTSnSec
>  dataset    /entry/instrument/NDAttributes/NDArrayTimeStamp
>  dataset    /entry/instrument/NDAttributes/NDArrayUniqueId
>  group      /entry/instrument/detector
>  group      /entry/instrument/detector/NDAttributes
>  dataset    /entry/instrument/detector/NDAttributes/ColorMode
>  dataset    /entry/instrument/detector/data -> /entry/data/data
>  group      /entry/instrument/performance
>  dataset    /entry/instrument/performance/timestamp
>  }
> }
> 
> I tried the capture and the single mode. Always there is only one image in the file.
> I used the commonPluginsExample.cmd with some modifications to be able to use the same file for all three cameras.
> The Gather Plugin is from the first camera.
> 
> Regards,
> 
> Jörn
> Am Mittwoch, 19. Februar 2025, 17:35:20 Mitteleuropäische Normalzeit schrieb Mark Rivers:
> > Hi Jörn,
> >
> > In principle NDPluginGather can do what you are asking.  There are 2 restrictions:
> >
> >
> >   1.
> > All of the detectors must be running in the same IOC, i.e. the same process.
> >   2.
> > All of the detectors must be producing NDArrays with the same dimensions, data type, and attributes.
> >
> > The second restriction is due to the way the HDF5 plugin works.  In Stream mode it saves a single dataset of dimensions [Nx, Ny, Nimages].  The datatype of that dataset and also the attribute datasets are determined by the first array it receives.  All subsequent arrays must be the same.  If not, you should see an error at the IOC shell and on the OPI screen.
> >
> > If your setup meets these restrictions and you are still having problems, then please send screenshots of the Gather and HDF5 plugin screens.
> >
> > Mark
> >
> > ________________________________
> > From: Tech-talk <tech-talk-bounces at aps.anl.gov<mailto:tech-talk-bounces at aps.anl.gov>> on behalf of Jörn Dreyer via Tech-talk <tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>>
> > Sent: Wednesday, February 19, 2025 2:21 AM
> > To: tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov> <tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>>
> > Subject: Using NDPluginGather to collect NDArrays from multiple detectors?
> >
> > Hi,
> >
> > I would like to write the images from multiple detectors into a single file.
> > Therefor I tried to setup an NDPluginGather and connected the Array ports to the asynArray's of the detectors.
> > The output of this was connected to an NDPluginFileHDF5. But in the resulting HDF5 file was only one image.
> >
> > I gues I misunderstood the functionality of NDPluginGather.
> > Looking at the documentation it seems as if it seems that it can only collect data from Plugins (Attributes).
> >
> > Is there a way to achieve that, or do I have to live with one file per detector?
> >
> > Regards
> >
> > Jörn
> >
> >
> >
> 
>