Hi Steve,
As Mark mention; no we can't do that with the NDFileHDF5 plugin - and it differs from fundamental principles in how we currently write NDArrays and NDAttributes to HDF5 files.
Here at Diamond we write additional HDF5/NeXus files using a supervisory process application where HD5 External Links [1], or in more recent cases VDS [2], is used to link and re-shape the data without copying it for application-specific purposes.
It is a pretty complex topic, come and see me or Alan Greer about this when you're next at Diamond :-)
Cheers,
Ulrik
[1]: https://support.hdfgroup.org/HDF5/doc/RM/RM_H5L.html
[2]: https://support.hdfgroup.org/HDF5/docNewFeatures/NewFeaturesVirtualDatasetDocs.html
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Mark Rivers
Sent: 08 February 2018 12:02
To: Kinder, Steve (STFC,DL,TECH) <[email protected]>; [email protected]
Subject: Re: Data format using NDFileHDF5
Hi Steve,
> They would however like what I think is a list of separate 2D arrays with the attributes directly connected to each array they are relevant to.
I don't think you can do that via the XML layout file with the existing NDFileHDF5 plugin. When collecting NumArrays arrays into a single HDF5 file the plugin creates an image array [NX, NY, [NZ, etc.], NumArrays], and for each attribute a 1-D array [NumArrays].
I believe the NDPosPlugin allows you to fill in those arrays non-sequentially, for example to handle a bi-directional "snake scan", but the array structure is fixed.
The simplest solution would be a post-processing program in Python, etc. that re-structures the array into a new HDF5 file. The plugin is already fairly complex, so it may not be trivial to add that feature.
Mark
________________________________
From: [email protected] <[email protected]> on behalf of [email protected] <[email protected]>
Sent: Thursday, February 8, 2018 3:15 AM
To: [email protected]
Subject: Data format using NDFileHDF5
I am using the NDFileHDF5 areaDetector plugin to write data and attributes collected from PCO cameras. I've been able to store data and attributes using XML files to define the attributes of interest and the following for the HDF5 layout:
<entry>
<group name="Data" ndattr_default="true">
<dataset name="Images" source="detector" det_default="true"></dataset>
</group>
</entry>
My end user sees this as a 3D data array with separate attributes which seemed fine. They would however like what I think is a list of separate 2D arrays with the attributes directly connected to each array they are relevant to. I've spent some time trying to figure out how to do that via the XML layout file but not got there yet. I will continue but I am beginning to wonder whether I will have to modifiy the plugin source.
In the meantime I wondered whether anyone had an example XML layout file that might help me get the desired structure? Or advice on whether the plugin will need modifying?
Thanks
Steve
--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
- Replies:
- Re: Data format using NDFileHDF5 steve.kinder
- References:
- Data format using NDFileHDF5 steve.kinder
- Re: Data format using NDFileHDF5 Mark Rivers
- Navigate by Date:
- Prev:
Re: Sequencer seem to skip states with EPICS 3.16.1 Benjamin Franksen
- Next:
Re: Why is post_event deprecated? Till Straumann
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
<2018>
2019
2020
2021
2022
2023
2024
- Navigate by Thread:
- Prev:
Re: Data format using NDFileHDF5 Mark Rivers
- Next:
Re: Data format using NDFileHDF5 steve.kinder
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
<2018>
2019
2020
2021
2022
2023
2024
|