RE: Area Detector and high performance NVME devices

[Mark Rivers <rivers@cars.uchicago.edu>]

Hi Mark,

> What i want is an effect more like just opening 2 or more distinct files at once. If they are doing totally different things, 
> with totally unrelated life cycles, what is the mutex protecting?

In general the mutex is protecting global structures and variables in the HDF5 library.  Consider an extremely simple example pseudocode:

FILE *globalFileId;
mutex globalLock;
void writeChunk(FILE *fileId, void *pData)
{
  lock(globalLock);
  globalFileId = fileId;
  writeData(pData);
  unlock(globalLock);
}
static void writeData(void *pData)
{
  fwrite(pData, 1, 1000, globalFileId);
}

writeChunk is externally visible and called by user-code, while writeData is private and hidden.  Because writeChunk does not pass fileId directly to writeData, but rather uses a global variable, it needs to be protected by a mutex.  Otherwise this code would not be thread safe.

Obviously the HDF5 library does not do anything this silly, but the principle is the same.  It uses global variables and structures that need to be protected by a mutex.  It does this in such a way that only 1 user-callable HDF5 function can be running at once, i.e. there is no concurrency possible in the library.   The only way to achieve concurrency is to run multiple processes, not multiple threads.  The HDF5 documents are very clear about this, and the performance results I posted yesterday using multiple threads vs multiple processes confirm it.

Mark

________________________________________
From: Engbretson, Mark S. [engbretson@aps.anl.gov]
Sent: Thursday, June 29, 2017 7:08 PM
To: Mark Rivers
Cc: ulrik.pedersen@diamond.ac.uk; tech-talk@aps.anl.gov
Subject: Re: Area Detector and high performance NVME devices

But wait . . . It can not work that way. Threads pop in and out of existence all the time. They may or may not use the HDF5 libraries, but if they do, they certainly would not want to have a dependency on code or constructs that could go "poof" at any instant.

What i want is an effect more like just opening 2 or more distinct files at once. If they are doing totally different things, with totally unrelated life cycles, what is the mutex protecting?

On Jun 29, 2017, at 18:16, Mark Rivers <rivers@cars.uchicago.edu> wrote:

What you propose would not work, because the problem is not in NDFileHDF5.cpp, it is in the underlying HDF5 library.  There are global structures there that you would not be making multiple instances of, so they need to be protected by the mutex.

-----Original Message-----
From: Mark S. Engbretson [mailto:Engbretson@anl.gov]
Sent: Thursday, June 29, 2017 5:05 PM
To: ulrik.pedersen@diamond.ac.uk; Mark Rivers
Cc: tech-talk@aps.anl.gov
Subject: RE: Area Detector and high performance NVME devices

For the sake of argument, what would happen if instead of creating multiple
instances of the HDF File plugin, which is actually the exact same entity, I
instead cloned the NDFileHDF5.cpp sources and made it a couple of unique
plugins named <NDFileHDF5>_A,B,C,and D. I can do this on windows system and
force every thread to have its own unique copy everything, so there is no
conflicts in non thread safe code just as you would not expect any mutex
issues from another process running on the system.

I know that I should technically be able to use constructs such as

void *handle1 = dlmopen(LM_ID_NEWLM, "/path/to/library.so", RTLD_NOW);
void *handle2 = dlmopen(LM_ID_NEWLM, "/path/to/library.so", RTLD_NOW);

Where the references is to the same shared library, but via difference
handles don't conflict.  But there should be easier ways to achieve the same
effect. I.e, with multiple file plugins, there are no common anythings that
should need that mutex lock or could be trashed.



-----Original Message-----
From: ulrik.pedersen@diamond.ac.uk [mailto:ulrik.pedersen@diamond.ac.uk]
Sent: Thursday, June 29, 2017 9:34 AM
To: Engbretson@anl.gov; rivers@cars.uchicago.edu
Cc: tech-talk@aps.anl.gov
Subject: RE: Area Detector and high performance NVME devices