Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  <20122013  2014  2015  2016  2017  2018  2019  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  <20122013  2014  2015  2016  2017  2018  2019 
<== Date ==> <== Thread ==>

Subject: Re: hdf5 (h5py) anyone?
From: Matt Newville <newville@cars.uchicago.edu>
To: "Malitsky, Nikolay D" <malitsky@bnl.gov>
Cc: epics <tech-talk@aps.anl.gov>
Date: Thu, 22 Mar 2012 13:44:01 -0500
Nikolay, Emmanuel,


On Thu, Mar 22, 2012 at 12:42 PM, Malitsky, Nikolay D <malitsky@bnl.gov> wrote:
>>what does an 'extra service' mean?
>>Are you talking about a client/server architecture?
>
> yes. In short, you can consider it as the Channel Archiver extension/upgrade
> with
> (one of) a new driver to the hdf5 backend.  As a result, conceptually
> everything
> (including file structure) would be the same except several extensions like
> support
> of the EPICS 4 structured data types. The type registry and associated
> topics will be
> presented at the next EPICS meeting.

I would be very cautious about archiving data directly into HDF5.  See:
     http://www.hdfgroup.org/hdf5-quest.html#gconc1
     http://www.hdfgroup.org/hdf5-quest.html#grdwt

Saving to some more robust format (say, ASCII files such as SDDS or a
relational database), and then converting to HDF5 for fast and
hierarchical read access is probably a much safer approach.

Whether hierarchical vs relational data storage is the better model
for an archiving system where the principle axis is time is an
interesting theoretical topic, but I think that archiving directly to
HDF5 is very likely to end up with lost data.

HDF5 is great at what it does, but it is not really a database
management system.

Cheers,

--Matt Newville


References:
hdf5 (h5py) anyone? Emmanuel Mayssat
RE: hdf5 (h5py) anyone? Malitsky, Nikolay D
Re: hdf5 (h5py) anyone? Emmanuel Mayssat
RE: hdf5 (h5py) anyone? Malitsky, Nikolay D
Re: hdf5 (h5py) anyone? Emmanuel Mayssat
RE: hdf5 (h5py) anyone? Malitsky, Nikolay D

Navigate by Date:
Prev: RE: hdf5 (h5py) anyone? Mark Rivers
Next: RE: menuFtype for 64 bit integers aka. long long and unsigned long long Mark Rivers
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  <20122013  2014  2015  2016  2017  2018  2019 
Navigate by Thread:
Prev: RE: hdf5 (h5py) anyone? Malitsky, Nikolay D
Next: RE: hdf5 (h5py) anyone? Mark Rivers
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  <20122013  2014  2015  2016  2017  2018  2019 
ANJ, 18 Nov 2013 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·