Real-time IOC and systemd

["Konrad, Martin via Tech-talk" <tech-talk at aps.anl.gov>]

Hi,
I'm struggling to configure an IOC for real-time scheduling. To ensure
I/O and record processing are happening in a predictable way I want to
configure

- CPU affinity (core 1-3 for the IOC, core 0 for everything else)
- real-time scheduling (SCHED_FIFO for important threads)
- real-time priority (e.g. OS runs cbHigh thread with higher priority)

I read [1] but I need to tweak the instructions provided there a little
since my OS is using systemd which starts procServ which in turn
executes my IOC (standard FRIB configuration using the attached
service file). Systemd allows setting some RT properties. I'm adding the
following file to augment our standard configuration with a few RT settings:

/etc/systemd/system/softioc-masterioc.service.d/10-realtime.conf

[Service]
CPUAffinity = 1-3
CPUSchedulingPolicy=fifo
CPUSchedulingPriority=99
LimitMEMLOCK=infinity

I'm applying these changes to take them for a test drive:
$ sudo systemctl daemon-reload
$ sudo systemctl restart softioc-masterioc.service

Test CPU affinity:
$ taskset -c -p 1
pid 1's current affinity list: 0
$ taskset -c -p `pgrep st.cmd`
pid 6553's current affinity list: 1-3

Test memory lock:
Here comes the first surprise:

$ grep VmLck /proc/`pgrep st.cmd`/status
VmLck:   0 kB

After reading [2] I was expecting the IOC to call mlockall()
automatically. This apparently doesn't work. Linking my IOC against
mcore-utils and calling mcoreMLock() in st.cmd results in the memory
being locked, though:

$ grep VmLck /proc/`pgrep st.cmd`/status
VmLck:   2590972 kB

This suggest that the necessary permissions have been granted to the IOC
successfully. I'm suspecting that they might not be detected correctly
in my environment, though.

Test RT scheduling and priorities:
Here comes another surprise:

$ ps -To pid,tid,policy,rtprio,comm -p `pgrep st.cmd`
  PID   TID POL RTPRIO COMMAND
 6553  6553 FF      99 st.cmd
 6553  6555 FF       1 errlog
 6553  6566 FF       1 PCIISR0000:08:0
 6553  6567 FF       1 taskwd
 6553  6568 FF       1 timerQueue
 6553  6569 FF       1 cbLow
...

Apparently systemd configures the process to run with the correct RT
scheduling policy ("FF" aka SCHED_FIFO). It also sets the real-time
priority of the IOC process correctly. However, for some reason all
child threads seem to have a priority of 1 while I would expect them to
inherit the process' priority. I also tried to apply finer-grained
priorities using mcore-utils with the following rtrules file:

callbackThreads:*:*:1:cb.*
scanThreads:*:*:2:scan.*
stateMachines:*:50:3:snc.*

The rules are loaded successfully but both ps and mcoreThreadShowAll
still report RTPRIO/OSSPRI as "1" for all threads.

To rule out problems with my systemd configuration I replaced the IOC
executable with a little test program [3] with an endless loop. This
works as expected:

$ ps -To pid,tid,policy,rtprio,comm -p `pgrep a.out`
  PID   TID POL RTPRIO COMMAND
19737 19737 FF      99 a.out
19737 19738 FF      80 a.out

Am I using EPICS' RT features the wrong way?

Thanks,

Martin

P.S.: My system runs Debian 10 with Debian's RT kernel:
$ uname -a
Linux gts-fts-mstr-n0001 4.19.0-6-rt-amd64 #1 SMP PREEMPT RT Debian
4.19.67-2+deb10u1 (2019-09-20) x86_64 GNU/Linux

[1]
https://epics-controls.org/resources-and-support/documents/howto-documents/posix-thread-priority/
[2] https://epics.anl.gov/base/R3-15/7-docs/RELEASE_NOTES.html
[3]
https://wiki.linuxfoundation.org/realtime/documentation/howto/applications/application_base#example

-- 
Martin Konrad
Facility for Rare Isotope Beams
Michigan State University
640 South Shaw Lane
East Lansing, MI 48824-1321, USA
Tel. 517-908-7253
Email: konrad at frib.msu.edu

[Unit]
Description=EPICS Soft IOC masterioc
Requires=network.target
After=network.target

[Service]
Environment="EPICS_IOC_LOG_PORT=7005"
Environment="EPICS_IOC_LOG_INET=sys-logstash.cts"
ExecStart=/usr/bin/procServ --foreground --quiet --chdir=/usr/local/lib/iocapps/masterioc/iocBoot/iocgts-fts-mstr-n0001-evg --ignore=^C^D^] --restrict --name masterioc --port 4051 /usr/local/lib/iocapps/masterioc/iocBoot/iocgts-fts-mstr-n0001-evg/st.cmd
Restart=always
User=softioc
RuntimeDirectory=softioc-masterioc

[Install]
WantedBy=multi-user.target