Experimental Physics and
Industrial Control System

Mark Rivers <[email protected]> · Wed, 16 Dec 2015 21:06:58 +0000

I agree with most of what Jason said.

> prevent your plugin from modifying the VLI until the plugin array callback chain has completed,

Currently there is no way to do this unless you have control over the downstream plugins and set them all to CallbacksBlock=Yes.  In that case doCallbacksGenericPointer will not return until all downstream plugins are done.

If any plugin has CallbackBlock=No then the only way to know if the plugins are done is to know if the referenceCount on the NDArray is back to 1 (because your plugin still references it but no other plugins do).  However, referenceCount is private to the NDArray class so you can't access it directly.  We would need to make an accessor function for referenceCount in the NDArray class.

Mark

________________________________
From: [email protected] [[email protected]] on behalf of Jason Matthias Abernathy [[email protected]]
Sent: Wednesday, December 16, 2015 2:06 PM
To: [email protected]
Subject: Re: areaDetector plugin: very large output array

It really depends on what is happening downstream.

If downstream needs to *modify* the Very Large Image then a copy is unavoidable. If it's not done by your plugin then at the very least a downstream plugin will copy it before modification (this is the convention).

If downstream only reads the Very Large Image then you need to either:
 a) prevent your plugin from modifying the VLI until the plugin array callback chain has completed, or
 b) accept that modifying the VLI "on the fly" will introduce side effects into downstream plugins.

(b) may be acceptable if somewhat real-time access to the VLI is required and the worker plugin is only making small changes.

Cheers,
Jason

On 15-12-16 11:41 AM, Phil Atkin wrote:
Just to re-iterate: the contents of the large array must be preserved over many cycles.  There can only be a single large array in use; the processing produces a cumulative result in it.  (Imagine, if you will, a plugin that extrapolates each input image to a larger array, then sums the result; this is structurally equivalent to what I need although the details are different).

Does this change the answer?

Phil

On 16/12/2015 17:57, Mark Rivers wrote:

I would first see if the normal mechanism of allocating an array from the pool will work.  I suspect it will work fine.

That mechanism only needs to allocate more than 2 arrays at a time if a downstream plugin is still busy processing the previous array when your plugin does callbacks on the next array.  If that does not happen, and all downstream plugins are done by the time you do your next callback then 2 pool arrays should be enough.  That's only 2 GB of RAM which any modern machine should not have trouble with.

You can also set all downstream plugins to have CallbacksBlock=Yes, at which point there is no queueing of arrays at all, all the plugins are run in the thread for your plugin, and a single array would be enough.

Mark

________________________________
From: [email protected]<mailto:[email protected]> [[email protected]<mailto:[email protected]>] on behalf of Phil Atkin [[email protected]<mailto:[email protected]>]
Sent: Wednesday, December 16, 2015 10:39 AM
To: [email protected]<mailto:[email protected]>
Subject: areaDetector plugin: very large output array

Hi folks,

My plugin needs to build and output a very large image - 1GB would not be surprising.  The contents of the image are gradually accumulated as many images are produced by the detector driver.  I'm not sure how I can make this 'fit' the NDArray/NDArrayPool architecture.  If I make the accumulator image a private area of memory then I could copy to an output NDArray when required - but that requires even more memory.

My ideal approach would be for my plugin to allocate the accumulator array (a 'worker' class currently encapsulates this), and only when the output is required would a pointer to this memory be used to 'publish' the result through an NDArray.

If I created a private NDArray and set its pData pointer to my accumulator image, could I then pass this to doCallbacksGenericPointer instead of the usual this->pArrays[0]?  If so, that call would presumably be inexpensive unless/until a client subscribed to the output port?

Thanks,

Phil
--
[cid:[email protected]]Pixel Analytics is a limited company registered in England. Company number: 7747526; Registered office: 93A New Road, Haslingfield, Cambridge CB23 1LP

--
[cid:[email protected]]Pixel Analytics is a limited company registered in England. Company number: 7747526; Registered office: 93A New Road, Haslingfield, Cambridge CB23 1LP

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System