Experimental Physics and Industrial Control System
|
hdf5 offers three important conceptual components (multi-dimensional datasets,
named data types, and groups) and a few nice technical features like chunks.
As a result, it looks friendly for accommodating historical data (one-dimensional
time series of user-defined data types) and multi-dimensional images.
read/write issues, multi-file data processing, etc require the corresponding extra service.
As I said, it is under consideration ….
-Nikolay
From: Emmanuel Mayssat [[email protected]]
Sent: Thursday, March 22, 2012 3:22 AM
To: Malitsky, Nikolay D; epics
Subject: Re: hdf5 (h5py) anyone?
Interesting... that's exactly why I want to use hdf5 for: archiving!
That is I would like a one-virtual-file database.
Can hdf5 do that? Can you give me pointers to h5 multi-file data processing?
I cannot find anything about it.
I heard that there can be read/write issues that could lead to data corruption.
(even with one single writer and several readers)
Can you confirm that? is there a way to minimize risk?
(I am thinking a 1 file per day archive, writer is on today's file, all the readers have access to history minus today)
So far I use sddslogger, but am looking at other alternatives like this one.
--
Emmanuel
Hello,
We (nsls-2) are pretty serious about this format and expect to accommodate it under
the present archiver system to address both the accelerator control and beamline
requirements. In short, I am (very) like the hdf5 conceptual model, but it requires
extra services for multi-file data processing. So, we are working on the corresponding
solution.
-Nikolay
hdf5 is a filesystem database (directory file structure, can span files and computers, etc)
h5py v2.0 (python interface) was demonstrated recently at PyCon2012.
Is anyone of you using hdf5 file in an epics context (or other)?
Regards,
--
E
|
- Replies:
- Re: hdf5 (h5py) anyone? Emmanuel Mayssat
- References:
- hdf5 (h5py) anyone? Emmanuel Mayssat
- RE: hdf5 (h5py) anyone? Malitsky, Nikolay D
- Re: hdf5 (h5py) anyone? Emmanuel Mayssat
- Navigate by Date:
- Prev:
Re: hdf5 (h5py) anyone? Emmanuel Mayssat
- Next:
RE: hdf5 (h5py) anyone? Mark Rivers
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
<2012>
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
- Navigate by Thread:
- Prev:
Re: hdf5 (h5py) anyone? Emmanuel Mayssat
- Next:
Re: hdf5 (h5py) anyone? Emmanuel Mayssat
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
<2012>
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
|
ANJ, 18 Nov 2013 |
·
Home
·
News
·
About
·
Base
·
Modules
·
Extensions
·
Distributions
·
Download
·
·
Search
·
EPICS V4
·
IRMIS
·
Talk
·
Bugs
·
Documents
·
Links
·
Licensing
·
|