Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020 
<== Date ==> <== Thread ==>

Subject: Re: EPICS 3.15 mlockall()/dbThreadRealtimeLock high memory usage
From: Till Straumann via Tech-talk <tech-talk@aps.anl.gov>
To: Lucas Russo <lerwys@gmail.com>, EPICS Tech Talk <tech-talk@aps.anl.gov>
Date: Wed, 24 Jul 2019 10:53:39 -0700
EPICS IOCs traditionally (back in the days when most of them were running on vxWorks)
have been supporting or even relying on real-time scheduling.

This is also true under linux. However, real-time scheduling doesn't make much sense
if memory has to be paged in before RT threads can do their work. Thus the need to
lock pages in memory.

Under linux the IOC attempts to use RT scheduling and if it is allowed to use the RT
scheduler then it also attempts to mlockall().

The mechanism to prevent an IOC from using these mechanisms is simple: restrict
the processes' privileges to use RT priorities and it will execute as a non-RT process
and will in this case not try to mlockall().

On most linux distros a normal process lacks the privilege to use RT - the sysadmin
has to specifically set this up so select processes may use RT.
If your system is very relaxed or you run IOCs as root then you have to revoke
privileges (ulimit) from processes that should not use RT.

Since these are administrative decisions and can be managed with ordinary linux
sysadmin tools there is no need for special EPICS mechanisms.

See also:
https://wiki-ext.aps.anl.gov/epics/index.php/How_To_Use_Posix_Thread_Priority_Scheduling_under_Linux

HTH
- Till

On 7/24/19 10:24 AM, Lucas Russo via Tech-talk wrote:
Hello everyone,

Recently, I noticed one of my IOCs consuming a lot more memory right after IOC startup (from linux "top" command, RES field, which I know it's not a good estimate sometimes, but still...) after upgrading it from EPICS base 3.14.12.6 to 3.15.6. All of the other EPICS modules was kept in the same version:

asyn: R4-35
areaDetector: R3-6

Even with other IOCs, areaDetector ADSimDetector, for instance, the memory consumption after IOC startup was way higher than before. For example:

EPICS 3.14 ADSimDetector: RES = 356502 MiB
EPICS 3.15 ADSimDetector: RES = 48613 MiB

After doing some debugging and memory profiling I saw that the ADSimDetector IOC running with EPICS 3.15 had the "VMLck" field from /proc/<pid>/status with:

VmLck: 4586544 kB

And the IOC running EPICS 3.14 had:

VmLck: 0  kB

Taking a look at the Release Notes there was a mention of "mlockall" being moved to some other place ("Moved mlockall() into its own epicsThread routine", "Changes made between 3.15.3 and 3.15.4")

I changed the "dbThreadRealtimeLock" variable from 1 (default) to 0 and the memory consumption dropped back to ~48613 MiB. Even after running the IOC for some time, doing some large acquisitions, the memory consumption was not close to the 356502 MiB.

Can anyone help me understand if this is somewhat expected?

And wouldn't it be better to have the variable dbThreadRealtimeLock with the default value of 0, instead of 1?

Cheers and Regards,

Lucas



Replies:
Re: EPICS 3.15 mlockall()/dbThreadRealtimeLock high memory usage Lucas Russo via Tech-talk
Re: EPICS 3.15 mlockall()/dbThreadRealtimeLock high memory usage Ralph Lange via Tech-talk
References:
EPICS 3.15 mlockall()/dbThreadRealtimeLock high memory usage Lucas Russo via Tech-talk

Navigate by Date:
Prev: Re: EPICS 3.15 mlockall()/dbThreadRealtimeLock high memory usage Lucas Russo via Tech-talk
Next: Re: EPICS 3.15 mlockall()/dbThreadRealtimeLock high memory usage Lucas Russo via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020 
Navigate by Thread:
Prev: Re: EPICS 3.15 mlockall()/dbThreadRealtimeLock high memory usage Lucas Russo via Tech-talk
Next: Re: EPICS 3.15 mlockall()/dbThreadRealtimeLock high memory usage Lucas Russo via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020 
ANJ, 24 Jul 2019 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·