Re: RT linux and priorities

[Michael Davidsaver via Core-talk <core-talk at aps.anl.gov>]

Hi Han,

It's good to know that I'm not imagining this problem,
and thank you for sharing your experience.  Between this,
and the lack of PI mutex* thus far.  I'm left with the
impression that IOCs on RT Linux have always been subject
to priority inversions without some amount of patching
and tweaking.

I hope that others will share their experiences with, and patches for, doing this.

By aggregating this information, we can perhaps choose
better defaults, or add some warnings, to make this
process less tricky.


* cf. https://code.launchpad.net/~dirk.zimoch/epics-base/+git/epics-base/+merge/394327


On 2/2/21 6:21 PM, Jeong Han Lee wrote:
> Hi Michael,
> 
> I saw this issue twice when I worked at ESS. Due to this reason, I had
> to change the "rt prio" to more than 80 on each device interface
> (ethernet or EVG).
> An ugly script for this purpose, you can find at
> https://github.com/icshwi/realtime-config/blob/master/rtprio.bash

This seems like a reasonable approach.  As a place to start, here is
a one-liner to raise all IRQ thread priorities to 92.  I pick 92
since epicsThreadPriorityBaseMax is 91 (cf. epicsThread.h).

> for pid in `pgrep '^irq/'`; do readlink /proc/$pid/exe || chrt -f -p 92 $pid; done

The readlink distinguishes between kernel threads, and some other
process which happens to name itself "irq/...".


> For ECMC (Ethercat Motion Controller), at the early stage, we used
> this script to make sure the priority of an allocated ethernet port
> (software ethercat master) should be larger than any EPICS process,
> and later we've embedded this logic within EPICS IOC.
> 
> I forgot almost everything about them now, and I am not sure whether
> ESS still uses this repository or not. However, this repo was my
> struggle to make "motion system works" in my previous life even if I
> am not an expert on this subject.
> 
> HTH,
> Han.
> 
> 
> On Tue, Feb 2, 2021 at 10:01 AM Michael Davidsaver via Core-talk
> <core-talk at aps.anl.gov> wrote:
>>
>>
>> I've not paid serious attention to RT Linux in the past as the cases where
>> I needed RT were already using RTEMS.  Now I have a project where I may
>> want RT latency bounds when receiving high rate UDP traffic (where a short
>> delay can easily mean a socket buffer overflow).
>>
>> While running a test to see that RT priorities were actually being set, I happened
>> to mis-type a 'ps' and list priorities for all threads.  I noticed that some of
>> my (IOC) threads had a higher priority than the OS threaded interrupt handlers.
>>
>> It looks like the ISR threads default to SCHED_FIFO/50, which is also what
>> epicsThreadPriorityMedium currently maps to.  So eg. the 1 second scan task
>> has a higher priority (63) than the NIC ISR thread (50).  This seems like
>> a recipe for some unexpected priority inversions.
>>
>> Has anyone else looked into this?
>>
>>
>> I'm looking at the 'rtprio' and 'cls' columns.  The "[...]" entire are kernel threads.
>>
>>> ps -eLo rtprio,pri,nice,cls,pid,cmd|sort -n
>> ...
>>>     10  50   -  FF 31826 ../../bin/linux-x86_64/pscdemo st.cmd
>>>     10  50   -  FF 31826 ../../bin/linux-x86_64/pscdemo st.cmd
>>>     10  50   -  RR 17801 /usr/lib/firefox-esr/firefox-esr ...
>>>     10  50   -  RR 24728 /usr/lib/firefox-esr/firefox-esr ...
>>>     12  52   -  FF 31826 ../../bin/linux-x86_64/pscdemo st.cmd
>> ...
>>>     35  75   -  FF 31826 ../../bin/linux-x86_64/pscdemo st.cmd
>>>     50  90   -  FF   100 [irq/123-pciehp]
>>>     50  90   -  FF   101 [irq/125-pciehp]
>>>     50  90   -  FF   224 [card0-crtc0]
>>>     50  90   -  FF   225 [card0-crtc1]
>>>     50  90   -  FF   226 [card0-crtc2]
>>>     50  90   -  FF   230 [irq/171-DLL096D]
>>>     50  90   -  FF 26049 [irq/172-mei_me]
>>>     50  90   -  FF 31826 ../../bin/linux-x86_64/pscdemo st.cmd
>> ...
>>>     50  90   -  FF 31826 ../../bin/linux-x86_64/pscdemo st.cmd
>>>     50  90   -  FF   461 [watchdogd]
>>>     50  90   -  FF   753 [irq/176-iwlwifi]
>> ...
>>>     50  90   -  FF   768 [irq/185-iwlwifi]
>>>     50  90   -  FF    99 [irq/122-pciehp]
>>>     58  98   -  FF 31826 ../../bin/linux-x86_64/pscdemo st.cmd
>> ...
>>>     89 129   -  FF 31826 ../../bin/linux-x86_64/pscdemo st.cmd
>>>     99 139   -  FF    12 [migration/0]
>> ...
>>>     99 139   -  FF    45 [migration/7]
>>>     99 139   -  RR  1427 /usr/lib/rtkit/rtkit-daemon
>>
>> Now I'm also wondering how firefox is able to set SCHED_RR/10