Experimental Physics and
Industrial Control System

<[email protected]> · Thu, 8 Feb 2018 14:51:29 +0000

Hi Steve,

As Mark mention; no we can't do that with the NDFileHDF5 plugin - and it differs from fundamental principles in how we currently write NDArrays and NDAttributes to HDF5 files.

Here at Diamond we write additional HDF5/NeXus files using a supervisory process application where HD5 External Links [1], or in more recent cases VDS [2], is used to link and re-shape the data without copying it for application-specific purposes.

It is a pretty complex topic, come and see me or Alan Greer about this when you're next at Diamond :-)

Cheers,
Ulrik

[1]: https://support.hdfgroup.org/HDF5/doc/RM/RM_H5L.html
[2]: https://support.hdfgroup.org/HDF5/docNewFeatures/NewFeaturesVirtualDatasetDocs.html 

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Mark Rivers
Sent: 08 February 2018 12:02
To: Kinder, Steve (STFC,DL,TECH) <[email protected]>; [email protected]
Subject: Re: Data format using NDFileHDF5

Hi Steve,

> They would however like what I think is a list of separate 2D arrays with the attributes directly connected to each array they are relevant to.

I don't think you can do that via the XML layout file with the existing NDFileHDF5 plugin.  When collecting NumArrays arrays into a single HDF5 file the plugin creates an image array [NX, NY, [NZ, etc.], NumArrays], and for each attribute a 1-D array [NumArrays].

I believe the NDPosPlugin allows you to fill in those arrays non-sequentially, for example to handle a bi-directional "snake scan", but the array structure is fixed.

The simplest solution would be a post-processing program in Python, etc. that re-structures the array into a new HDF5 file.  The plugin is already fairly complex, so it may not be trivial to add that feature.

Mark

________________________________
From: [email protected] <[email protected]> on behalf of [email protected] <[email protected]>
Sent: Thursday, February 8, 2018 3:15 AM
To: [email protected]
Subject: Data format using NDFileHDF5

I am using the NDFileHDF5 areaDetector plugin to write data and attributes collected from PCO cameras. I've been able to store data and attributes using XML files to define the attributes of interest and the following for the HDF5 layout:

<entry>
  <group name="Data" ndattr_default="true">
    <dataset name="Images" source="detector" det_default="true"></dataset>
  </group>
</entry>

My end user sees this as a 3D data array with separate attributes which seemed fine. They would however like what I think is a list of separate 2D arrays with the attributes directly connected to each array they are relevant to. I've spent some time trying to figure out how to do that via the XML layout file but not got there yet. I will continue but I am beginning to wonder whether I will have to modifiy the plugin source.

In the meantime I wondered whether anyone had an example XML layout file that might help me get the desired structure? Or advice on whether the plugin will need modifying?

Thanks

Steve

-- 
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System