Re: hdf5 (h5py) anyone?

[Emmanuel Mayssat <emayssat@yahoo.com>]

Interesting... that's exactly why I want to use hdf5 for: archiving!

That is I would like a one-virtual-file database.

Can hdf5 do that? Can you give me pointers to h5 multi-file data processing?

I cannot find anything about it.

I heard that there can be read/write issues that could lead to data corruption.

(even with one single writer and several readers)

Can you confirm that? is there a way to minimize risk?

(I am thinking a 1 file per day archive, writer is on today's file, all the readers have access to history minus today)

So far I use sddslogger, but am looking at other alternatives like this one.

--

Emmanuel

---

From: "Malitsky, Nikolay D" <malitsky@bnl.gov>
To: Emmanuel Mayssat <emayssat@yahoo.com>; epics <tech-talk@aps.anl.gov>
Cc: "Malitsky, Nikolay D" <malitsky@bnl.gov>
Sent: Wednesday, March 21, 2012 7:31 PM
Subject: RE: hdf5 (h5py) anyone?

Hello,

We (nsls-2) are pretty serious about this format and expect to accommodate it under
the present archiver system to address both the accelerator control and beamline
requirements.  In short, I am (very) like the hdf5 conceptual model, but it requires
extra services for multi-file data processing. So, we are working on the corresponding
solution.

-Nikolay

---

From: tech-talk-bounces@aps.anl.gov [tech-talk-bounces@aps.anl.gov] on behalf of Emmanuel Mayssat [emayssat@yahoo.com]
Sent: Wednesday, March 21, 2012 8:30 PM
To: epics
Subject: hdf5 (h5py) anyone?

hdf5 is a filesystem database (directory file structure, can span files and computers, etc)

h5py v2.0 (python interface) was demonstrated recently at PyCon2012.
Is anyone of you using hdf5 file in an epics context (or other)?

Regards,
--
E