Channel Access performance with RTEMS on MVME5500

["Wells, Alex \(DLSLtd,RAL,LSCI\) via Tech-talk" <tech-talk at aps.anl.gov>]

Hello Tech Talk,

I'm in the process of moving some VME crates from VxWorks to RTEMS. This also moves from EPICS 3.14.12.7 to EPICS 7.0.7. I'm seeing significant amounts of CPU activity in the "CAS-UDP" thread, and was hoping someone may be able to explain it.

Averaged over time, RTEMS reports that the CAS-UDP thread is using ~40% of the CPU. This persists even when removing all the processing from all records (thus removing all other possible CPU use).  During initialization of the IOC, and for several minutes after initialization finished, I see 80%+ CPU usage on the IOC overall, mostly on this one thread, and it makes the overall IOC very unresponsive. This same behaviour also happens intermittently during operation, where even record processing seems to become delayed due to Channel Access processing - for example the devIocStats Heartbeat record that should just increase by 1 per second sometimes pauses for multiple seconds.

The network the IOC is on is requesting a fair number of PVs from this IOC. CaSnooper shows between 20 and 40 requests per second to PV(s) on this IOC. Approximately 500 PVs are being requested from the IOC. These requests are spread across approximately 30 separate clients. 

When I compare this to the VxWorks version, running on the same hardware and network, I see less almost 0% CPU usage on the CAS-UDP thread, and no record processing hitches. The CPU also idles significantly lower than for RTEMS.

Our current theory is that the RTEMS network stack for UDP processing is much less efficient, and is struggling with the volume of requests. Is anyone able to confirm/deny this, and does anyone have any potential workarounds?

Thanks,

Alex Wells

Diamond Light Source

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom.