Experimental Physics and
Industrial Control System

"Sinclair, John William" <[email protected]> · Tue, 25 Sep 2018 18:54:39 +0000

About one week of investigation revealed the following information related to PREEMPT-RT considerations.

Issues:
 - kernel config
 - boot params
 - system config
 - system runtime config
 - thread config

Kernel config

I have only experimented with kernel version 3.14.65-rt68 so far.

In General setup
   - In Timers subsystem
     - disable Old Idle dynticks config
     - In Timer tick handling
       - select Periodic timer ticks (constant rate, no dynticks)

In Processor type and features
   - disable Intel Low Power Subsystem Support

In Power management and ACPI options
   - disable Cpuidle Driver for Intel Processors
   - In ACPI (Advanced Configuration and Power Interface) Support
        - disable Processor

-----------------------------------------------

Boot parms - example config for isolcpus, yours will depend on requirements

 - nmi_watchdog=0
 - watchdog=0
 - isolcpus={list of cpus to isolate}, e.g. isolcpus=2,3

-----------------------------------------------

System config - example config, yours will depend on requirements

/etc/sysconfig/irqbalance

Add:
IRQBALANCE_BANNED_CPUS=0xc             # this is a bit mask, 0x12 means do not balance irq handlers on cpus 2 and 3

-----------------------------------------------

System runtime config

echo "-1" > /proc/sys/kernel/sched_rt_runtime_us

-----------------------------------------------

Thread config

The irq handler needs to be associated with an isolated cpu via affinity (by setting /proc/irq/#/smp_affinity) and possibly have the real-time priority adjusted (chrt).

The real-time user process will need real-time scheduling enabled and should probably lock all virtual memory into ram via mlockall. Within this process, the thread awakened by the handler may need to be associated with an isolated cpu via affinity (taskset) and will need priority adjusted (chrt).

For the case of a single interrupt handler and thread running on the same isolated cpu I observed the following over a 24 hour period:

Largest interrupt latency = 35 us
Largest "return to user mode" latency = 171 us

During this period other processes were performing the following operations:
  - reading and updating memory
  - open file, write file, close file
  - read file over a network connection
  - each of the above spinning in a loop with no delays

The cpu clock speed was around 3 GHz.

I estimated the above timing numbers using custom hardware that implemented a free running 8ns period counter and a special interrupt generator that would itself capture the current counter value for later reference.

--------------------------------------------------------

It is worth noting that Concurrent sells redhawk linux which currently uses a 4.1.27 version of the kernel and they claim the max interrupt latency with qualified hardware is 10 us. They did not supply timing information for the "return to user mode" latency.

Best,
John Sinclair

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Jeong Han Lee
Sent: Monday, September 24, 2018 7:58 PM
To: EPICS Tech Talk <[email protected]>
Subject: Linux PREEMPT-RT kernel tuning example for EPICS IOC

Hi,

   Recently, I am trying to build a generic configuration for the Kernel configuration via the following script:

  https://github.com/icshwi/e3-tools/blob/master/rt_conf/rt_conf.bash

  while studying the existent documents in RT community. But my main concern is the configuration for various IOC applications. Does anyone have site-wide example for these configurations?

   I would like to get some reasonable examples from EPICS community about the Linux PREEMPT-RT kernel tuning for EPICS IOCs.

  Best,
  Han

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System