1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 <2015> 2016 2017 2018 2019 2020 2021 2022 2023 2024 | Index | 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 <2015> 2016 2017 2018 2019 2020 2021 2022 2023 2024 |
<== Date ==> | <== Thread ==> |
---|
Subject: | Re: areaDetector plugin: very large output array |
From: | Phil Atkin <[email protected]> |
To: | Mark Rivers <[email protected]>, "[email protected]" <[email protected]> |
Date: | Wed, 16 Dec 2015 19:41:52 +0000 |
Just to re-iterate: the contents of the large array must be
preserved over many cycles. There can only be a single large array
in use; the processing produces a cumulative result in it.
(Imagine, if you will, a plugin that extrapolates each input image
to a larger array, then sums the result; this is structurally
equivalent to what I need although the details are different). Does this change the answer? Phil On 16/12/2015 17:57, Mark Rivers wrote:
I would first see if the normal mechanism of allocating an array from the pool will work. I suspect it will work fine. That mechanism only needs to allocate more than 2 arrays at a time if a downstream plugin is still busy processing the previous array when your plugin does callbacks on the next array. If that does not happen, and all downstream plugins are done by the time you do your next callback then 2 pool arrays should be enough. That's only 2 GB of RAM which any modern machine should not have trouble with. You can also set all downstream plugins to have CallbacksBlock=Yes, at which point there is no queueing of arrays at all, all the plugins are run in the thread for your plugin, and a single array would be enough. Mark ________________________________ From: [email protected] [[email protected]] on behalf of Phil Atkin [[email protected]] Sent: Wednesday, December 16, 2015 10:39 AM To: [email protected] Subject: areaDetector plugin: very large output array Hi folks, My plugin needs to build and output a very large image - 1GB would not be surprising. The contents of the image are gradually accumulated as many images are produced by the detector driver. I'm not sure how I can make this 'fit' the NDArray/NDArrayPool architecture. If I make the accumulator image a private area of memory then I could copy to an output NDArray when required - but that requires even more memory. My ideal approach would be for my plugin to allocate the accumulator array (a 'worker' class currently encapsulates this), and only when the output is required would a pointer to this memory be used to 'publish' the result through an NDArray. If I created a private NDArray and set its pData pointer to my accumulator image, could I then pass this to doCallbacksGenericPointer instead of the usual this->pArrays[0]? If so, that call would presumably be inexpensive unless/until a client subscribed to the output port? Thanks, Phil -- [cid:[email protected]]Pixel Analytics is a limited company registered in England. Company number: 7747526; Registered office: 93A New Road, Haslingfield, Cambridge CB23 1LP --
Pixel Analytics is a limited company registered in England. Company number: 7747526; Registered office: 93A New Road, Haslingfield, Cambridge CB23 1LP |