Experimental Physics and Industrial Control System
|
Subject: |
Re: areaDetector plugin: very large output array |
From: |
Jason Matthias Abernathy <[email protected]> |
To: |
<[email protected]> |
Date: |
Wed, 16 Dec 2015 12:06:01 -0800 |
It really depends on what is happening downstream.
If downstream needs to *modify* the Very Large Image then a copy is
unavoidable. If it's not done by your plugin then at the very least
a downstream plugin will copy it before modification (this is the
convention).
If downstream only reads the Very Large Image then you need to
either:
a) prevent your plugin from modifying the VLI until the plugin
array callback chain has completed, or
b) accept that modifying the VLI "on the fly" will introduce side
effects into downstream plugins.
(b) may be acceptable if somewhat real-time access to the VLI is
required and the worker plugin is only making small changes.
Cheers,
Jason
On 15-12-16 11:41 AM, Phil Atkin wrote:
Just to re-iterate: the contents of the large array must
be preserved over many cycles. There can only be a single large
array in use; the processing produces a cumulative result in it.
(Imagine, if you will, a plugin that extrapolates each input image
to a larger array, then sums the result; this is structurally
equivalent to what I need although the details are different).
Does this change the answer?
Phil
On 16/12/2015 17:57, Mark Rivers
wrote:
I would first see if the normal mechanism of allocating an array from the pool will work. I suspect it will work fine.
That mechanism only needs to allocate more than 2 arrays at a time if a downstream plugin is still busy processing the previous array when your plugin does callbacks on the next array. If that does not happen, and all downstream plugins are done by the time you do your next callback then 2 pool arrays should be enough. That's only 2 GB of RAM which any modern machine should not have trouble with.
You can also set all downstream plugins to have CallbacksBlock=Yes, at which point there is no queueing of arrays at all, all the plugins are run in the thread for your plugin, and a single array would be enough.
Mark
________________________________
From: [email protected] [[email protected]] on behalf of Phil Atkin [[email protected]]
Sent: Wednesday, December 16, 2015 10:39 AM
To: [email protected]
Subject: areaDetector plugin: very large output array
Hi folks,
My plugin needs to build and output a very large image - 1GB would not be surprising. The contents of the image are gradually accumulated as many images are produced by the detector driver. I'm not sure how I can make this 'fit' the NDArray/NDArrayPool architecture. If I make the accumulator image a private area of memory then I could copy to an output NDArray when required - but that requires even more memory.
My ideal approach would be for my plugin to allocate the accumulator array (a 'worker' class currently encapsulates this), and only when the output is required would a pointer to this memory be used to 'publish' the result through an NDArray.
If I created a private NDArray and set its pData pointer to my accumulator image, could I then pass this to doCallbacksGenericPointer instead of the usual this->pArrays[0]? If so, that call would presumably be inexpensive unless/until a client subscribed to the output port?
Thanks,
Phil
--
[cid:[email protected]]Pixel Analytics is a limited company registered in England. Company number: 7747526; Registered office: 93A New Road, Haslingfield, Cambridge CB23 1LP
--
Pixel
Analytics is a limited company registered in England.
Company number: 7747526; Registered office: 93A New Road,
Haslingfield, Cambridge CB23 1LP
|
- Replies:
- RE: areaDetector plugin: very large output array Mark Rivers
- References:
- areaDetector plugin: very large output array Phil Atkin
- RE: areaDetector plugin: very large output array Mark Rivers
- Re: areaDetector plugin: very large output array Phil Atkin
- Navigate by Date:
- Prev:
Re: areaDetector plugin: very large output array Phil Atkin
- Next:
RE: areaDetector plugin: very large output array Mark Rivers
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
<2015>
2016
2017
2018
2019
2020
2021
2022
2023
2024
- Navigate by Thread:
- Prev:
Re: areaDetector plugin: very large output array Phil Atkin
- Next:
RE: areaDetector plugin: very large output array Mark Rivers
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
<2015>
2016
2017
2018
2019
2020
2021
2022
2023
2024
|
ANJ, 16 Dec 2015 |
·
Home
·
News
·
About
·
Base
·
Modules
·
Extensions
·
Distributions
·
Download
·
·
Search
·
EPICS V4
·
IRMIS
·
Talk
·
Bugs
·
Documents
·
Links
·
Licensing
·
|