Re: Real-time IOC and systemd

[Benjamin Franksen via Tech-talk <tech-talk at aps.anl.gov>]

Hi Martin

It's been a while since you sent this so my reply may no longer be
useful. Anyway.

It is my experience with systemd that it is best to not rely on it,
except for the well-tested mainstream features. Each day they are adding
more configuration options to cater for yet another special situation.
It is no wonder that this often leads to corner cases when options
interact with each other, and such interactions are (understandably)
often not well-documented.

A more predictable way to do things like setting resource limits is to
use the tools that come with Linux: prlimit, unshare, runuser, etc.
These are /very/ simple tools that do one thing and do it well. For
instance, here is my current ExecStart for a generic soft IOC:

ExecStart=/usr/bin/prlimit --memlock=2000 --rtprio=unlimited unshare -u
sethostname %i\
 runuser -u ioc --\
 procServ --foreground --logfile=/var/log/softIOC/%i.log
--timefmt='+%Y-%m-%d %H:%M:%S.%N%:z'\
 --chdir=${BOOTDIR} --quiet --name=%i --ignore=^D^C^] ${PORT} ./${STCMD}

At first blush this may look a lot less clean than setting options in
the unit file. But this is superficial. These small "chain loader" tools
can be combined in many ways without starting to interact strangely. For
instance, with such a command line it is crystal clear at which point
privileges are dropped.

(NB, sethostname is a simple shell script I wrote to make hostname act
as a chain loader. We use this behind the unshare -u so that soft IOCs
report their own name as their host name, so CA clients can distinguish
between IOCs running on the same machine for CA security.

> cat sethostname
#!/bin/sh
hostname "$1"
shift 1
exec "$@"
)

Cheers
Ben

Am 07.02.20 um 19:47 schrieb Konrad, Martin via Tech-talk:
> Hi,
> I'm struggling to configure an IOC for real-time scheduling. To ensure
> I/O and record processing are happening in a predictable way I want to
> configure
> 
> - CPU affinity (core 1-3 for the IOC, core 0 for everything else)
> - real-time scheduling (SCHED_FIFO for important threads)
> - real-time priority (e.g. OS runs cbHigh thread with higher priority)
> 
> I read [1] but I need to tweak the instructions provided there a little
> since my OS is using systemd which starts procServ which in turn
> executes my IOC (standard FRIB configuration using the attached
> service file). Systemd allows setting some RT properties. I'm adding the
> following file to augment our standard configuration with a few RT settings:
> 
> /etc/systemd/system/softioc-masterioc.service.d/10-realtime.conf
> 
> [Service]
> CPUAffinity = 1-3
> CPUSchedulingPolicy=fifo
> CPUSchedulingPriority=99
> LimitMEMLOCK=infinity
> 
> I'm applying these changes to take them for a test drive:
> $ sudo systemctl daemon-reload
> $ sudo systemctl restart softioc-masterioc.service
> 
> Test CPU affinity:
> $ taskset -c -p 1
> pid 1's current affinity list: 0
> $ taskset -c -p `pgrep st.cmd`
> pid 6553's current affinity list: 1-3
> 
> Test memory lock:
> Here comes the first surprise:
> 
> $ grep VmLck /proc/`pgrep st.cmd`/status
> VmLck:   0 kB
> 
> After reading [2] I was expecting the IOC to call mlockall()
> automatically. This apparently doesn't work. Linking my IOC against
> mcore-utils and calling mcoreMLock() in st.cmd results in the memory
> being locked, though:
> 
> $ grep VmLck /proc/`pgrep st.cmd`/status
> VmLck:   2590972 kB
> 
> This suggest that the necessary permissions have been granted to the IOC
> successfully. I'm suspecting that they might not be detected correctly
> in my environment, though.
> 
> Test RT scheduling and priorities:
> Here comes another surprise:
> 
> $ ps -To pid,tid,policy,rtprio,comm -p `pgrep st.cmd`
>   PID   TID POL RTPRIO COMMAND
>  6553  6553 FF      99 st.cmd
>  6553  6555 FF       1 errlog
>  6553  6566 FF       1 PCIISR0000:08:0
>  6553  6567 FF       1 taskwd
>  6553  6568 FF       1 timerQueue
>  6553  6569 FF       1 cbLow
> ...
> 
> Apparently systemd configures the process to run with the correct RT
> scheduling policy ("FF" aka SCHED_FIFO). It also sets the real-time
> priority of the IOC process correctly. However, for some reason all
> child threads seem to have a priority of 1 while I would expect them to
> inherit the process' priority. I also tried to apply finer-grained
> priorities using mcore-utils with the following rtrules file:
> 
> callbackThreads:*:*:1:cb.*
> scanThreads:*:*:2:scan.*
> stateMachines:*:50:3:snc.*
> 
> The rules are loaded successfully but both ps and mcoreThreadShowAll
> still report RTPRIO/OSSPRI as "1" for all threads.
> 
> To rule out problems with my systemd configuration I replaced the IOC
> executable with a little test program [3] with an endless loop. This
> works as expected:
> 
> $ ps -To pid,tid,policy,rtprio,comm -p `pgrep a.out`
>   PID   TID POL RTPRIO COMMAND
> 19737 19737 FF      99 a.out
> 19737 19738 FF      80 a.out
> 
> Am I using EPICS' RT features the wrong way?
> 
> Thanks,
> 
> Martin
> 
> P.S.: My system runs Debian 10 with Debian's RT kernel:
> $ uname -a
> Linux gts-fts-mstr-n0001 4.19.0-6-rt-amd64 #1 SMP PREEMPT RT Debian
> 4.19.67-2+deb10u1 (2019-09-20) x86_64 GNU/Linux
> 
> [1]
> https://epics-controls.org/resources-and-support/documents/howto-documents/posix-thread-priority/
> [2] https://epics.anl.gov/base/R3-15/7-docs/RELEASE_NOTES.html
> [3]
> https://wiki.linuxfoundation.org/realtime/documentation/howto/applications/application_base#example
>

Attachment: signature.asc
Description: OpenPGP digital signature