Experimental Physics and
Industrial Control System

Jeong Han Lee via Core-talk <core-talk at aps.anl.gov> · Tue, 2 Feb 2021 18:21:04 -0800

Hi Michael,

I saw this issue twice when I worked at ESS. Due to this reason, I had
to change the "rt prio" to more than 80 on each device interface
(ethernet or EVG).
An ugly script for this purpose, you can find at
https://github.com/icshwi/realtime-config/blob/master/rtprio.bash

For ECMC (Ethercat Motion Controller), at the early stage, we used
this script to make sure the priority of an allocated ethernet port
(software ethercat master) should be larger than any EPICS process,
and later we've embedded this logic within EPICS IOC.

I forgot almost everything about them now, and I am not sure whether
ESS still uses this repository or not. However, this repo was my
struggle to make "motion system works" in my previous life even if I
am not an expert on this subject.

HTH,
Han.

On Tue, Feb 2, 2021 at 10:01 AM Michael Davidsaver via Core-talk
<core-talk at aps.anl.gov> wrote:
>
>
> I've not paid serious attention to RT Linux in the past as the cases where
> I needed RT were already using RTEMS.  Now I have a project where I may
> want RT latency bounds when receiving high rate UDP traffic (where a short
> delay can easily mean a socket buffer overflow).
>
> While running a test to see that RT priorities were actually being set, I happened
> to mis-type a 'ps' and list priorities for all threads.  I noticed that some of
> my (IOC) threads had a higher priority than the OS threaded interrupt handlers.
>
> It looks like the ISR threads default to SCHED_FIFO/50, which is also what
> epicsThreadPriorityMedium currently maps to.  So eg. the 1 second scan task
> has a higher priority (63) than the NIC ISR thread (50).  This seems like
> a recipe for some unexpected priority inversions.
>
> Has anyone else looked into this?
>
>
> I'm looking at the 'rtprio' and 'cls' columns.  The "[...]" entire are kernel threads.
>
> > ps -eLo rtprio,pri,nice,cls,pid,cmd|sort -n
> ...
> >     10  50   -  FF 31826 ../../bin/linux-x86_64/pscdemo st.cmd
> >     10  50   -  FF 31826 ../../bin/linux-x86_64/pscdemo st.cmd
> >     10  50   -  RR 17801 /usr/lib/firefox-esr/firefox-esr ...
> >     10  50   -  RR 24728 /usr/lib/firefox-esr/firefox-esr ...
> >     12  52   -  FF 31826 ../../bin/linux-x86_64/pscdemo st.cmd
> ...
> >     35  75   -  FF 31826 ../../bin/linux-x86_64/pscdemo st.cmd
> >     50  90   -  FF   100 [irq/123-pciehp]
> >     50  90   -  FF   101 [irq/125-pciehp]
> >     50  90   -  FF   224 [card0-crtc0]
> >     50  90   -  FF   225 [card0-crtc1]
> >     50  90   -  FF   226 [card0-crtc2]
> >     50  90   -  FF   230 [irq/171-DLL096D]
> >     50  90   -  FF 26049 [irq/172-mei_me]
> >     50  90   -  FF 31826 ../../bin/linux-x86_64/pscdemo st.cmd
> ...
> >     50  90   -  FF 31826 ../../bin/linux-x86_64/pscdemo st.cmd
> >     50  90   -  FF   461 [watchdogd]
> >     50  90   -  FF   753 [irq/176-iwlwifi]
> ...
> >     50  90   -  FF   768 [irq/185-iwlwifi]
> >     50  90   -  FF    99 [irq/122-pciehp]
> >     58  98   -  FF 31826 ../../bin/linux-x86_64/pscdemo st.cmd
> ...
> >     89 129   -  FF 31826 ../../bin/linux-x86_64/pscdemo st.cmd
> >     99 139   -  FF    12 [migration/0]
> ...
> >     99 139   -  FF    45 [migration/7]
> >     99 139   -  RR  1427 /usr/lib/rtkit/rtkit-daemon
>
> Now I'm also wondering how firefox is able to set SCHED_RR/10

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System