Re: Channel Access performance with RTEMS on MVME5500

[Michael Davidsaver via Tech-talk <tech-talk at aps.anl.gov>]

On 3/18/25 09:41, Wells, Alex (DLSLtd,RAL,LSCI) wrote:
> Hi Michael,
>
> I can confirm the CPU use goes to zero when the network cable is unplugged. We do image and file loading over the network, but once all that has completed I can pull the cable out and observe no CPU use by that thread. The system also becomes fully responsive on the console, which correlates the significant drop in CPU use.
>
> I have only tested RTEMS 5.1, which I believe is using the "legacy" IP stack. In a few months, during the next shutdown, I may be able to test RTEMS6 but we have no particular reason to expect the newer stack to work better, as the MVME5500 is 30 years old now so the new stack is unlikely to be optimized for its use case.
I concur with your assessment.  Part of the reason that RTEMS
is juggling two different IP stacks is that not all of the
NIC drivers exist in the newer libbsd stack.

That fact that the CPU load under vxWorks was apparently less
does suggest that the RTEMS NIC driver could be improved.
Although whether this would be cost-effective for such an old
board though...?

So your practical choices may be limited to isolating this
crate behind a cagateway to limit broadcast traffic.


> Thanks,
> Alex Wells
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> *From:* Michael Davidsaver <mdavidsaver at gmail.com>
> *Sent:* Friday, March 14, 2025 4:39 PM
> *To:* Wells, Alex (DLSLtd,RAL,LSCI) <alex.wells at diamond.ac.uk>
> *Cc:* tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
> *Subject:* Re: Channel Access performance with RTEMS on MVME5500
> Hello Alex,
>
> As a quick test, have you tried unplugging the ethernet cable to ensure that
> the reported CPU usage for CAS-UDP goes to zero?
>
> Also, which RTEMS version(s) have you tested?
>
> Is this the RTEMS "legacy" IP stack, or the newer libbsd stack?
>
>
> On 3/14/25 06:30, Wells, Alex (DLSLtd,RAL,LSCI) via Tech-talk wrote:
> > Hello Tech Talk,
> >
> > I'm in the process of moving some VME crates from VxWorks to RTEMS. This also moves from EPICS 3.14.12.7 to EPICS 7.0.7. I'm seeing significant amounts of CPU activity in the "CAS-UDP" thread, and was hoping someone may be able to explain it.
> >
> > Averaged over time, RTEMS reports that the CAS-UDP thread is using ~40% of the CPU. This persists even when removing all the processing from all records (thus removing all other possible CPU use).  During initialization of the IOC, and for several minutes after initialization finished, I see 80%+ CPU usage on the IOC overall, mostly on this one thread, and it makes the overall IOC very unresponsive. This same behaviour also happens intermittently during operation, where even record processing seems to become delayed due to Channel Access processing - for example the devIocStats Heartbeat record that should just increase by 1 per second sometimes pauses for multiple seconds.
> >
> > The network the IOC is on is requesting a fair number of PVs from this IOC. CaSnooper shows between 20 and 40 requests per second to PV(s) on this IOC. Approximately 500 PVs are being requested from the IOC. These requests are spread across approximately 30 separate clients.
> >
> > When I compare this to the VxWorks version, running on the same hardware and network, I see less almost 0% CPU usage on the CAS-UDP thread, and no record processing hitches. The CPU also idles significantly lower than for RTEMS.
> >
> > Our current theory is that the RTEMS network stack for UDP processing is much less efficient, and is struggling with the volume of requests. Is anyone able to confirm/deny this, and does anyone have any potential workarounds?
> >
> > Thanks,
> > Alex Wells
> > Diamond Light Source
> >
> > This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
> > Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
> > Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom.
> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom.