Re: EPICS 3.15 mlockall()/dbThreadRealtimeLock high memory usage

[Lucas Russo via Tech-talk <tech-talk@aps.anl.gov>]

Hi Till,

Thanks for the explanation. I was just surprised to see this behavior change when upgrading base versions.

Anyway, It seems I was doing the wrong thing from the start.

Thanks!

Lucas

On Wed, Jul 24, 2019 at 2:54 PM Till Straumann <strauman@slac.stanford.edu> wrote:

EPICS IOCs traditionally (back in the days when most of them were running on vxWorks)
have been supporting or even relying on real-time scheduling.

This is also true under linux. However, real-time scheduling doesn't make much sense
if memory has to be paged in before RT threads can do their work. Thus the need to
lock pages in memory.

Under linux the IOC attempts to use RT scheduling and if it is allowed to use the RT
scheduler then it also attempts to mlockall().

The mechanism to prevent an IOC from using these mechanisms is simple: restrict
the processes' privileges to use RT priorities and it will execute as a non-RT process
and will in this case not try to mlockall().

On most linux distros a normal process lacks the privilege to use RT - the sysadmin
has to specifically set this up so select processes may use RT.
If your system is very relaxed or you run IOCs as root then you have to revoke
privileges (ulimit) from processes that should not use RT.

Since these are administrative decisions and can be managed with ordinary linux
sysadmin tools there is no need for special EPICS mechanisms.

See also:
https://wiki-ext.aps.anl.gov/epics/index.php/How_To_Use_Posix_Thread_Priority_Scheduling_under_Linux

HTH
- Till

On 7/24/19 10:24 AM, Lucas Russo via Tech-talk wrote:

Hello everyone,

Recently, I noticed one of my IOCs consuming a lot more memory right after IOC startup (from linux "top" command, RES field, which I know it's not a good estimate sometimes, but still...) after upgrading it from EPICS base 3.14.12.6 to 3.15.6. All of the other EPICS modules was kept in the same version:

asyn: R4-35

areaDetector: R3-6

Even with other IOCs, areaDetector ADSimDetector, for instance, the memory consumption after IOC startup was way higher than before. For example:

EPICS 3.14 ADSimDetector: RES = 356502 MiB

EPICS 3.15 ADSimDetector: RES = 48613 MiB

After doing some debugging and memory profiling I saw that the ADSimDetector IOC running with EPICS 3.15 had the "VMLck" field from /proc/<pid>/status with:

VmLck: 4586544 kB

And the IOC running EPICS 3.14 had:

VmLck: 0  kB

Taking a look at the Release Notes there was a mention of "mlockall" being moved to some other place ("Moved mlockall() into its own epicsThread routine", "Changes made between 3.15.3 and 3.15.4")

I changed the "dbThreadRealtimeLock" variable from 1 (default) to 0 and the memory consumption dropped back to ~48613 MiB. Even after running the IOC for some time, doing some large acquisitions, the memory consumption was not close to the 356502 MiB.

Can anyone help me understand if this is somewhat expected?

And wouldn't it be better to have the variable dbThreadRealtimeLock with the default value of 0, instead of 1?

Cheers and Regards,

Lucas