Experimental Physics and
Industrial Control System

Mark Rivers <[email protected]> · Tue, 1 May 2018 21:10:48 +0000

Hi Steve,

> Ok I'll do some more tests tomorrow, with R3-1 and stream mode initially. 
> Just to double check, you think that in that mode the parameters with each image should reflect PV values as they were when the data was collected from the detector?

You are using the HDF5 plugin.  Are you using an XML file to specify the layout of the data and attributes in the HDF5 file?  If so then for each NDAttribute that you write to the HDF5 file you can specify when that attribute is written to the file.  The choices are: "OnFileOpen", "OnFileClose", and  "OnFileWrite".  If you have selected either of the first 2 choices then that attribute will only be written to the file once (on open or close) and not for each image.  So you need to make sure that you either omit the "when" attribute in the XML file (in which case it defaults to "OnFileWrite" which stores it for each image), or explicitly specify "OnFileWrite".

> The change to NDPluginFile at R3-2 sounds like a patch I applied to R3-1 following a previous tech-talk discussion (prior to R3-2 release). 
> Is that the one that would should address the parameters issue I asked about originally with capture mode?

Can you post a link to the tech-talk thread?  I am not sure what discussion and patch you are referring to.

Mark

-----Original Message-----
From: Steve Kinder - UKRI STFC [mailto:[email protected]] 
Sent: Tuesday, May 1, 2018 3:55 PM
To: Mark Rivers <[email protected]>; [email protected]
Subject: Re: NDPluginFile attribute saving

Ok I'll do some more tests tomorrow, with R3-1 and stream mode initially. Just to double check, you think that in that mode the parameters with each image should reflect PV values as they were when the data was collected from the detector?

The change to NDPluginFile at R3-2 sounds like a patch I applied to R3-1 following a previous tech-talk discussion (prior to R3-2 release). Is that the one that would should address the parameters issue I asked about originally with capture mode?

Steve

________________________________
From: Mark Rivers <[email protected]>
Sent: 01 May 2018 21:43:27
To: Kinder, Steve (STFC,DL,TECH); [email protected]
Subject: RE: NDPluginFile attribute saving

> I was using capture mode to maximise performance of the image acquisition. We want to go as fast as possible.
> From what I've read this mode data is placed in buffer and only written to disk when the capture stops. Is this still correct?

It is true that capture mode saves the images in a buffer and then writes to disk when the capture stops.  However, I am not sure that capture mode is any faster than stream mode, which writes to disk as the images come in.  Because the acquisition thread and the file writing thread can be using different cores, the file writing thread should not slow down the acquisition thread in stream mode.  You would need to test this on your machine and detector to be sure, but I suspect you may get the same performance in stream and capture modes.  You can test that today without needing to upgrade to R3-2.  Just be sure to set the QueueSize large enough that you don't drop frames in stream mode.

If you set the QueueSize equal to the number of images you will collect then it is very similar to capture mode except that it is writing the images to disk as they arrive rather than waiting for them to all be collected.  But if the disk cannot keep up the images will be buffered in the queue and you cannot lose any if the QueueSize is set equal to the number of images you will collect.

Mark

-----Original Message-----
From: Steve Kinder - UKRI STFC [mailto:[email protected]]
Sent: Tuesday, May 1, 2018 3:33 PM
To: Mark Rivers <[email protected]>; [email protected]
Subject: Re: NDPluginFile attribute saving

Hi Mark

Many thanks for that. I'm using R3-1 of ADCore, I'll look at the release notes but sounds like I'm going to need an update.

I was using capture mode to maximise performance of the image acquisition. We want to go as fast as possible. From what I've read this mode data is placed in buffer and only written to disk when the capture stops. Is this still correct?

Steve

________________________________
From: Mark Rivers <[email protected]>
Sent: 01 May 2018 18:35:56
To: Kinder, Steve (STFC,DL,TECH); [email protected]
Subject: RE: NDPluginFile attribute saving

Hi Steve,

What version of ADCore are you using?  The way capture mode is handled was changed in R3-2.  See the release notes:

https://github.com/areaDetector/ADCore/blob/master/RELEASE.md

In R3-2 I think it will do what you want, but previously it probably did not handle attributes correctly.

Is there a reason you are using capture mode rather than stream mode?  I recommend using stream mode.  Prior to R2-5 there was a reason to use Capture mode because the QueueSize for a plugin was fixed at iocInit and could not be changed a run-time.  Since R2-5 the QueueSize for all plugins can be changed at run-time.  It is generally better to set a large QueueSize and use stream mode rather than capture mode because:
- It always handles the attributes correctly, even prior to R3-2.
- File writing is overlapped with data collection so it reduces the total time for the experiment
- The memory overhead is less because it only needs to buffer the frames it cannot keep up with writing to disk, rather than buffering all frames in memory.

Capture mode may be deprecated in a future release of ADCore because it complicates the code and offers no real advantages any more.

Mark

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Steve Kinder - UKRI STFC
Sent: Tuesday, May 1, 2018 12:08 PM
To: [email protected]
Subject: NDPluginFile attribute saving

I have an areaDetector file plugin saving a specific HDF5 structure with multiple images in one file. I set attributes to be saved per image via NDAttributesFile. I'm currently using capture mode. At the moment each image seems to have the same parameter values; as at the point the file is written. Is there a way for the attribute values to be as they were at the point each image is collected? Maybe I missed something in creating the plugin, some other setting or the attributes XML file?

Thanks

Steve

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System