EPICS Home EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

Format page
for printing


Search Tech-talk
1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  <2021>  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  <2021>  2022  2023  2024 
<== Date ==> <== Thread ==>
Subject: Re: ADVimba memory leak ?
From: John Dobbins via Tech-talk <tech-talk at aps.anl.gov>
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: "tech-talk at aps.anl.gov \(tech-talk at aps.anl.gov\)" <tech-talk at aps.anl.gov>
Date: Fri, 17 Dec 2021 22:46:07 +0000
Mark,

I'll need to look more carefully to determine if the growth is continuous or in spurts.

John
From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Friday, December 17, 2021 4:41:35 PM
To: John Dobbins <john.dobbins at cornell.edu>
Cc: tech-talk at aps.anl.gov (tech-talk at aps.anl.gov) <tech-talk at aps.anl.gov>
Subject: RE: ADVimba memory leak ?
 

Hi John,

 

I just some testing on one of our Allied Vision cameras.

 

Model                     GT1380

Firmware Version  00.01.54.17562

SDK Version         1.8.2

Driver Version       1.3

ADCore Version    3.11

Operating system   Centos8

 

I ran the following command to get the virtual memory size and the resident memory size, both in KB.

$ date;ps -o vsz,rss,cmd 13422

 

In the output below I have put the date on the same line as the ps output.

 

This is just after the IOC started, acquisition has not been started.

 

                                                      VSZ      RSS      CMD

Fri Dec 17 14:10:26 CST 2021 5993748 76732 /corvette/home/epics/support/areaDetector/ADVimba/iocs/vimbaIOC/bin/linux-x86_64-centos8/vimbaApp st.cmd

Fri Dec 17 14:10:33 CST 2021 5993748 76732 /corvette/home/epics/support/areaDetector/ADVimba/iocs/vimbaIOC/bin/linux-x86_64-centos8/vimbaApp st.cmd

Fri Dec 17 14:12:06 CST 2021 5993748 76732 /corvette/home/epics/support/areaDetector/ADVimba/iocs/vimbaIOC/bin/linux-x86_64-centos8/vimbaApp st.cmd

Fri Dec 17 14:18:16 CST 2021 5993748 76732 /corvette/home/epics/support/areaDetector/ADVimba/iocs/vimbaIOC/bin/linux-x86_64-centos8/vimbaApp st.cmd

 

So it ran for about 8 minutes and there was no increase in VSZ or RSS.  Thus, I do not see the increase of 0.15 MB/min (150 KB/min) that you see when not acquiring.

 

I now started acquisition.  This camera is 1360x1024 pixels.  Each image was thus 1360 KB.  I was acquiring at 5 frames/s, and I had the following plugins active:

NDPluginStdArray, NDPluginPva, NDPluginTransform, NDPluginROI, NDPluginStat, NDFileJPEG, NDFileTIFF.  The file plugins were not saving data.

 

                                                      VSZ      RSS      CMD

Fri Dec 17 14:19:10 CST 2021 6089256 125504 /corvette/home/epics/support/areaDetector/ADVimba/iocs/vimbaIOC/bin/linux-x86_64-centos8/vimbaApp st.cmd

Fri Dec 17 14:19:57 CST 2021 6089256 127336 /corvette/home/epics/support/areaDetector/ADVimba/iocs/vimbaIOC/bin/linux-x86_64-centos8/vimbaApp st.cmd

Fri Dec 17 14:23:26 CST 2021 6155792 134424 /corvette/home/epics/support/areaDetector/ADVimba/iocs/vimbaIOC/bin/linux-x86_64-centos8/vimbaApp st.cmd

Fri Dec 17 14:44:48 CST 2021 6155792 134848 /corvette/home/epics/support/areaDetector/ADVimba/iocs/vimbaIOC/bin/linux-x86_64-centos8/vimbaApp st.cmd

Fri Dec 17 14:45:55 CST 2021 6155792 134848 /corvette/home/epics/support/areaDetector/ADVimba/iocs/vimbaIOC/bin/linux-x86_64-centos8/vimbaApp st.cmd

Fri Dec 17 14:51:39 CST 2021 6155792 134848 /corvette/home/epics/support/areaDetector/ADVimba/iocs/vimbaIOC/bin/linux-x86_64-centos8/vimbaApp st.cmd

 

When acquisition was turned on there was an immediate jump in VSZ and RSS.  This is expected because a number of NDArrays have been allocated. RSS was still increasing 4 minutes after acquisition started.  In the ~21 minutes between 14:23:26 and 14:44:48 RSS increased by only about 400 KB, so less than the 1360 KB in a single image, and less 20 KB/min.  After 14:44:48 there was no further increase in RSS for the 7 minutes that I tested.

 

I then stopped acquisition.

 

                                                      VSZ      RSS      CMD

Fri Dec 17 14:51:57 CST 2021 6142152 121208 /corvette/home/epics/support/areaDetector/ADVimba/iocs/vimbaIOC/bin/linux-x86_64-centos8/vimbaApp st.cmd

Fri Dec 17 14:52:28 CST 2021 6142152 121208 /corvette/home/epics/support/areaDetector/ADVimba/iocs/vimbaIOC/bin/linux-x86_64-centos8/vimbaApp st.cmd

Fri Dec 17 14:59:34 CST 2021 6142152 121208 /corvette/home/epics/support/areaDetector/ADVimba/iocs/vimbaIOC/bin/linux-x86_64-centos8/vimbaApp st.cmd

Fri Dec 17 15:05:02 CST 2021 6142152 121208 /corvette/home/epics/support/areaDetector/ADVimba/iocs/vimbaIOC/bin/linux-x86_64-centos8/vimbaApp st.cmd

 

When acquisition was stopped VSZ and RSS both dropped.  RSS did not increase at all in the 13 minutes I observed after stopping acquisition.

 

Is the increase in RSS size you are seeing continuous, or does it suddenly increase at specific times?

 

Mark

 

 

From: John Dobbins <john.dobbins at cornell.edu>
Sent: Wednesday, December 15, 2021 2:47 PM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: tech-talk at aps.anl.gov (tech-talk at aps.anl.gov) <tech-talk at aps.anl.gov>
Subject: Re: ADVimba memory leak ?

 

Mark,

 

An IOC which is not acquiring from the camera grows at ~ 0.15 MB per minute. (resident memory grows, virtual memory is constant)

 

An IOC acquiring an image (1116x836, 5.5 Hz), no plugins, grows at ~ 0.24 MB per minute

 

[ note:  0.24 MB/min  -->  10 GB after a month]

 

Enabling NDPluginStdArrays, NDPluginOverLay, NDPluginROI, NDPluginStats doesn't seem to produce additional memory growth.

 

John

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of John Dobbins via Tech-talk <tech-talk at aps.anl.gov>
Sent: Monday, December 13, 2021 8:50 PM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: tech-talk at aps.anl.gov (tech-talk at aps.anl.gov) <tech-talk at aps.anl.gov>
Subject: Re: ADVimba memory leak ?

 

Sorry, I should have said, Linux. I will investigate and report. 

 

John

 

From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Monday, December 13, 2021 7:36:10 PM
To: John Dobbins <john.dobbins at cornell.edu>
Cc: tech-talk at aps.anl.gov (tech-talk at aps.anl.gov) <tech-talk at aps.anl.gov>
Subject: RE: ADVimba memory leak ?

 

Hi John,

 

Is this on Linux or Windows?

 

Can you use “top” or other memory monitoring tools to see if memory usage increase corresponds to specific actions, such as stopping and starting the camera, errors about dropped frames, use of specific plugins, etc.?

 

Mark

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of John Dobbins via Tech-talk
Sent: Monday, December 13, 2021 6:26 PM
To: tech-talk at aps.anl.gov
Subject: ADVimba memory leak ?

 

All,

 

We are using ADVimba with Mako G319B cameras. Over a period of three months memory usage has grown by an order of magnitude. Has anyone else encountered this?

 

Firmware Version 00.01.54.21000

SDK Version           1.8.0

Driver Version       1.1

ADCore Version     3.7

 

[ I can try newer version of ADVimba in january.]

 

Regards,

 

John Dobbins

 

Research Support Specialist

Cornell High Energy Synchrotron Source

Cornell University

 

 

 

 

 

References:
ADVimba memory leak ? John Dobbins via Tech-talk
RE: ADVimba memory leak ? Mark Rivers via Tech-talk
Re: ADVimba memory leak ? John Dobbins via Tech-talk
Re: ADVimba memory leak ? John Dobbins via Tech-talk
RE: ADVimba memory leak ? Mark Rivers via Tech-talk
Navigate by Date:
Prev: RE: ADVimba memory leak ? Mark Rivers via Tech-talk
Next: Ask for help about compiling ADAravis on openEuler 20.03 LTS lup--- via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  <2021>  2022  2023  2024 
Navigate by Thread:
Prev: RE: ADVimba memory leak ? Mark Rivers via Tech-talk
Next: Photonic Science X-ray FDS camera Sintschuk, Michael via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  <2021>  2022  2023  2024 
ANJ, 19 Dec 2021 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·