Hi Mark/Ulrik
Thanks for the feedback and advice. In the end I decided to avoid post processing of the data files. To keep the code development as simple as possible I decided to develop a bespoke plugin to create the required layout. Happy to share if anyone wants similar but presume too specific to put back in area detector.
Thanks again
Steve
________________________________
From: Pedersen, Ulrik (DLSLtd,RAL,TEC)
Sent: 08 February 2018 14:51:29
To: Mark Rivers; Kinder, Steve (STFC,DL,TECH); [email protected]
Subject: RE: Data format using NDFileHDF5
Hi Steve,
As Mark mention; no we can't do that with the NDFileHDF5 plugin - and it differs from fundamental principles in how we currently write NDArrays and NDAttributes to HDF5 files.
Here at Diamond we write additional HDF5/NeXus files using a supervisory process application where HD5 External Links [1], or in more recent cases VDS [2], is used to link and re-shape the data without copying it for application-specific purposes.
It is a pretty complex topic, come and see me or Alan Greer about this when you're next at Diamond :-)
Cheers,
Ulrik
[1]: https://support.hdfgroup.org/HDF5/doc/RM/RM_H5L.html
[2]: https://support.hdfgroup.org/HDF5/docNewFeatures/NewFeaturesVirtualDatasetDocs.html
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Mark Rivers
Sent: 08 February 2018 12:02
To: Kinder, Steve (STFC,DL,TECH) <[email protected]>; [email protected]
Subject: Re: Data format using NDFileHDF5
Hi Steve,
> They would however like what I think is a list of separate 2D arrays with the attributes directly connected to each array they are relevant to.
I don't think you can do that via the XML layout file with the existing NDFileHDF5 plugin. When collecting NumArrays arrays into a single HDF5 file the plugin creates an image array [NX, NY, [NZ, etc.], NumArrays], and for each attribute a 1-D array [NumArrays].
I believe the NDPosPlugin allows you to fill in those arrays non-sequentially, for example to handle a bi-directional "snake scan", but the array structure is fixed.
The simplest solution would be a post-processing program in Python, etc. that re-structures the array into a new HDF5 file. The plugin is already fairly complex, so it may not be trivial to add that feature.
Mark
________________________________
From: [email protected] <[email protected]> on behalf of [email protected] <[email protected]>
Sent: Thursday, February 8, 2018 3:15 AM
To: [email protected]
Subject: Data format using NDFileHDF5
I am using the NDFileHDF5 areaDetector plugin to write data and attributes collected from PCO cameras. I've been able to store data and attributes using XML files to define the attributes of interest and the following for the HDF5 layout:
<entry>
<group name="Data" ndattr_default="true">
<dataset name="Images" source="detector" det_default="true"></dataset>
</group>
</entry>
My end user sees this as a 3D data array with separate attributes which seemed fine. They would however like what I think is a list of separate 2D arrays with the attributes directly connected to each array they are relevant to. I've spent some time trying to figure out how to do that via the XML layout file but not got there yet. I will continue but I am beginning to wonder whether I will have to modifiy the plugin source.
In the meantime I wondered whether anyone had an example XML layout file that might help me get the desired structure? Or advice on whether the plugin will need modifying?
Thanks
Steve
- References:
- Data format using NDFileHDF5 steve.kinder
- Re: Data format using NDFileHDF5 Mark Rivers
- RE: Data format using NDFileHDF5 ulrik.pedersen
- Navigate by Date:
- Prev:
RE: Possible issue with epics-base 7 compilation Gofron, Kazimierz
- Next:
RE: Possible issue with epics-base 7 compilation Mark Rivers
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
<2018>
2019
2020
2021
2022
2023
2024
- Navigate by Thread:
- Prev:
RE: Data format using NDFileHDF5 ulrik.pedersen
- Next:
Waveform Runtime Database Access Error Tobin R Weber
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
<2018>
2019
2020
2021
2022
2023
2024
|