RE: Strange memory leak with ADAravis

[Abdalla Ahmad via Tech-talk <tech-talk at aps.anl.gov>]

Hello Mark

 

I only used NDStdArrays for the ArrayData PV. Actually since then I haven’t been able to reproduce the problem, I started the same IOC couple of times during the day and sometimes overnight and the IOC is working fine every time. It could be the system got very busy and unstable due to some other process(s) hogging it. I will keep running it in an attempt to reproduce the issue. Please note that empty free list button did not solve the issue when the buffers were huge.

 

Best Regards,

Abdalla.

 

From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Thursday, June 2, 2022 1:46 PM
To: tech-talk at aps.anl.gov; Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo>
Subject: Re: Strange memory leak with ADAravis

 

Hi Abdalla,

 

What areaDetectors plugins do you have running and what is the value of QueueSize for each of them?  The Buffers information you sent is very useful.  It says that the current number of buffers (NDArrays) in use is only 22.  But it has allocated 24664 of them, and they are in the pool, using 28904 MB of memory.  Can you send a screen shot of the commonPlugins.adl screen?

 

I can think of two possibilities for this:

• The system got very busy for a while, and plugins were not able to keep up with the camera.  That will cause each active plugin to fill its queue.  If there were 12 plugins and each had QueueSize of 2000 that would explain it.
• There is a leak in the driver or some plugin which is causing the pool to grow.  But then I would expect PoolUsedBuffers to be large.  I would also expect someone else to have reported that problem by now.

Note that in the Buffers section of the detector screen there is button to Empty Free List.  That processes the  EmptyFreeList record, which will free all of the unused buffers and reduce the memory without restarting the IOC.

 

Mark

 

 

---

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Abdalla Ahmad via Tech-talk <tech-talk at aps.anl.gov>
Sent: Thursday, June 2, 2022 4:57 AM
To: tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Strange memory leak with ADAravis

 

Hi

 

I have setup area detector to control the Basler gigE cameras we have, I pulled the latest tag for all the necessary modules (asyn, ADCore, ADAravis, etc). The setup works fine, I can control the camera and acquire images even at the maximum FPS which is 32. Now I configured the IOC to control 2 cameras, one with exposure time 10 s and one with 32 FPS and left it overnight few days ago. I got to the PC the day after and the RAM and swap were full but the IOC was working, I managed to login to the PC later and launched the GUI and I found strange values in the Buffers section of ADAravis.adl:

 

PoolUsedBuffers: 22

PoolAllocBuffers: 24664

PoolFreeBuffers: 24642

PoolMaxMem: 0 MB

PoolUsedMem: 28904.3

 

The PC has is running CentOS 7 and has 8GB RAM. I re-ran the IOC yesterday and for now it is working fine. What could be wrong with my setup? The network is all gigabit Ethernet and jumbo frames are enabled.

 

Thanks

Abdalla.