Experimental Physics and
Industrial Control System

<[email protected]> · Wed, 14 Feb 2018 18:07:27 +0000

Hi Mark/Ulrik

Thanks for the feedback and advice. In the end I decided to avoid post processing of the data files. To keep the code development as simple as possible I decided to develop a bespoke plugin to create the required layout. Happy to share if anyone wants similar but presume too specific to put back in area detector.

Thanks again

Steve

________________________________
From: Pedersen, Ulrik (DLSLtd,RAL,TEC)
Sent: 08 February 2018 14:51:29
To: Mark Rivers; Kinder, Steve (STFC,DL,TECH); [email protected]
Subject: RE: Data format using NDFileHDF5

Hi Steve,

As Mark mention; no we can't do that with the NDFileHDF5 plugin - and it differs from fundamental principles in how we currently write NDArrays and NDAttributes to HDF5 files.

Here at Diamond we write additional HDF5/NeXus files using a supervisory process application where HD5 External Links [1], or in more recent cases VDS [2], is used to link and re-shape the data without copying it for application-specific purposes.

It is a pretty complex topic, come and see me or Alan Greer about this when you're next at Diamond :-)

Cheers,
Ulrik

[1]: https://support.hdfgroup.org/HDF5/doc/RM/RM_H5L.html
[2]: https://support.hdfgroup.org/HDF5/docNewFeatures/NewFeaturesVirtualDatasetDocs.html

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Mark Rivers
Sent: 08 February 2018 12:02
To: Kinder, Steve (STFC,DL,TECH) <[email protected]>; [email protected]
Subject: Re: Data format using NDFileHDF5

Hi Steve,

> They would however like what I think is a list of separate 2D arrays with the attributes directly connected to each array they are relevant to.

I don't think you can do that via the XML layout file with the existing NDFileHDF5 plugin.  When collecting NumArrays arrays into a single HDF5 file the plugin creates an image array [NX, NY, [NZ, etc.], NumArrays], and for each attribute a 1-D array [NumArrays].

I believe the NDPosPlugin allows you to fill in those arrays non-sequentially, for example to handle a bi-directional "snake scan", but the array structure is fixed.

The simplest solution would be a post-processing program in Python, etc. that re-structures the array into a new HDF5 file.  The plugin is already fairly complex, so it may not be trivial to add that feature.

Mark

________________________________
From: [email protected] <[email protected]> on behalf of [email protected] <[email protected]>
Sent: Thursday, February 8, 2018 3:15 AM
To: [email protected]
Subject: Data format using NDFileHDF5

I am using the NDFileHDF5 areaDetector plugin to write data and attributes collected from PCO cameras. I've been able to store data and attributes using XML files to define the attributes of interest and the following for the HDF5 layout:

<entry>
  <group name="Data" ndattr_default="true">
    <dataset name="Images" source="detector" det_default="true"></dataset>
  </group>
</entry>

My end user sees this as a 3D data array with separate attributes which seemed fine. They would however like what I think is a list of separate 2D arrays with the attributes directly connected to each array they are relevant to. I've spent some time trying to figure out how to do that via the XML layout file but not got there yet. I will continue but I am beginning to wonder whether I will have to modifiy the plugin source.

In the meantime I wondered whether anyone had an example XML layout file that might help me get the desired structure? Or advice on whether the plugin will need modifying?

Thanks

Steve

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System