1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 <2022> 2023 2024 | Index | 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 <2022> 2023 2024 |
<== Date ==> | <== Thread ==> |
---|
Subject: | Re: Strange memory leak with ADAravis |
From: | Mark Rivers via Tech-talk <tech-talk at aps.anl.gov> |
To: | Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo> |
Cc: | "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov> |
Date: | Wed, 21 Sep 2022 12:01:55 +0000 |
> I noticed that PollUsedMem.SCAN was set to “1 second”, when it was set to 1 second the buffers used and allocated were increasing by around 10-20 frames/s, when I set it to I/O Intr the buffers counts were constant and memory usage was perfectly fine.
SCAN=I/O Intr does not work for PoolUsedMem.SCAN, you need to use a periodic scan rate like "1 second". If you use I/O Intr it is not getting the statistics, so it is just hiding the problem.
You said that when it was 1 second the buffers used and allocated were increasing by 10-20 frames/s. That does not make sense for the screen shot you sent, because the acquisition time was 10 seconds. Your commonPlugins.adl screen shows you only have 2 plugins loaded and active, NDPluginStdArrays and NDPluginStats. In the worst possible case under those conditions, if both plugins and the camera had a memory leak, the number of buffers used and allocated could only increase by 0.1 frames/s * 3 = 0.3 frames/s. Were you really using a 10 second acquire time when you had the problem?
Please change PoolUsedMem.SCAN to 1 second and report again how the buffers used and allocated are increasing under the exposure time conditions you use when you have seen the memory problem in the past.
> Please note that in this setup (while also failing before) I set QSIZE in the st.cmd to 3000 then 10000.
I am confused, your screen shot of commonPlugins.cmd shows that QSIZE was set to 100 (Free column), not 3000 or 10000. For these plugins there is no reason to set a QueueSize that large. Generally you only need a large queue size like that for the file saving plugins, where you want to buffer frames because the camera is collecting faster than the files can be saved to disk. Note that QueueSize is normally in autosave, so the value in use may not be the QSIZE you set in the file. You can change it in the main medm screen for that plugin.
Mark
From: Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo>
Sent: Wednesday, September 21, 2022 3:25 AM To: Mark Rivers <rivers at cars.uchicago.edu> Cc: tech-talk at aps.anl.gov <tech-talk at aps.anl.gov> Subject: RE: Strange memory leak with ADAravis Hello Mark
I mistakenly rebooted the IOC when huge buffers were allocated (+120K), so I attached an image of the working setup. I noticed that PollUsedMem.SCAN was set to “1 second”, when it was set to 1 second the buffers used and allocated were increasing by around 10-20 frames/s, when I set it to I/O Intr the buffers counts were constant and memory usage was perfectly fine. Please note that in this setup (while also failing before) I set QSIZE in the st.cmd to 3000 then 10000.
Best Regards, Abdalla.
From: Mark Rivers <rivers at cars.uchicago.edu>
After this happens please send the screen shots for ADAravis and commonPlugins.
Mark
Sent from my iPhone
|