| 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 <2012> 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 | Index | 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 <2012> 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 |
| <== Date ==> | <== Thread ==> |
|---|
| Subject: | RE: hdf5 (h5py) anyone? |
| From: | "Malitsky, Nikolay D" <[email protected]> |
| To: | Emmanuel Mayssat <[email protected]>, epics <[email protected]> |
| Date: | Thu, 22 Mar 2012 10:33:46 +0000 |
|
hdf5 offers three important conceptual components (multi-dimensional datasets, named data types, and groups) and a few nice technical features like chunks. As a result, it looks friendly for accommodating historical data (one-dimensional time series of user-defined data types) and multi-dimensional images.
As I said, it is under consideration ….
-Nikolay From: Emmanuel Mayssat [[email protected]]
Sent: Thursday, March 22, 2012 3:22 AM To: Malitsky, Nikolay D; epics Subject: Re: hdf5 (h5py) anyone? Interesting... that's exactly why I want to use hdf5 for: archiving!
That is I would like a one-virtual-file database.
Can hdf5 do that? Can you give me pointers to h5 multi-file data processing?
I cannot find anything about it.
I heard that there can be read/write issues that could lead to data corruption.
(even with one single writer and several readers)
Can you confirm that? is there a way to minimize risk?
(I am thinking a 1 file per day archive, writer is on today's file, all the readers have access to history minus today)
So far I use sddslogger, but am looking at other alternatives like this one.
--
Emmanuel
From: "Malitsky, Nikolay D" <[email protected]> To: Emmanuel Mayssat <[email protected]>; epics <[email protected]> Cc: "Malitsky, Nikolay D" <[email protected]> Sent: Wednesday, March 21, 2012 7:31 PM Subject: RE: hdf5 (h5py) anyone? Hello,
We (nsls-2) are pretty serious about this format and expect to accommodate it under the present archiver system to address both the accelerator control and beamline requirements. In short, I am (very) like the hdf5 conceptual model, but it requires extra services for multi-file data processing. So, we are working on the corresponding solution. -Nikolay From: [email protected] [[email protected]] on behalf of Emmanuel Mayssat [[email protected]]
Sent: Wednesday, March 21, 2012 8:30 PM To: epics Subject: hdf5 (h5py) anyone? hdf5 is a filesystem database (directory file structure, can span files and computers, etc)
h5py v2.0 (python interface) was demonstrated recently at PyCon2012.Is anyone of you using hdf5 file in an epics context (or other)? Regards, -- E |