Re: CPU usage and response time of EPICS Channel Access responses

["Wells, Alex \(DLSLtd,RAL,TEC\) via Tech-talk" <tech-talk at aps.anl.gov>]

Hi Andrew,

Thank you so much for the rapid response. I can indeed wait until next week for a patch, I'll keep my eyes peeled for it.

Although I will note that "slower over the life of the IOC" doesn't quite fit my observations - All of these tests were done against a freshly rebooted IOC, where the only difference is the number of records. I suppose that would end up consuming more memory and thus would have an effect on the time taken to check it.

That explanation does cover another oddity I couldn't explain, which is why during my repeated tests some of them would take almost double the time to respond, but then the time fell back down again.

Thanks again,

Alex Wells

---

From: Johnson, Andrew N. <anj at anl.gov>
Sent: Friday, April 24, 2026 4:19 PM
To: Wells, Alex (DLSLtd,RAL,TEC) <alex.wells at diamond.ac.uk>; Jack Harper - STFC UKRI via Tech-talk <tech-talk at aps.anl.gov>
Subject: Re: CPU usage and response time of EPICS Channel Access responses

 

Hi Alex,

My colleague Ran Hong found the cause for this, and developed a patch to fix it but I don’t think he’s turned it into a PR against Base yet. The CA server asks the OS how much free memory is available before it responds to every search request, and the API that EPICS is using on RTEMS is the wrong one (it’s scanning every memory block on the device every time, which gets slower over the life of the IOC).

He’s travelling back from Paris sometime in the next few days, can you wait until next week for the fix?

- Andrew

-- 

Complexity comes for free, Simplicity you have to work for.

On 4/24/26, 10:12 AM, "Tech-talk" <tech-talk-bounces at aps.anl.gov> wrote:

Hi Tech Talk,

I'm investigating an issue when migrating from VxWorks+EPICS 3.14.12.7 to RTEMS5.1+EPICS 7.0.7. The IOC is on a VME crate running an MVME 5500 board. What I see is the RTEMS IOC taking significantly longer to respond to Channel Access requests, and the response time increases as I put more records into the database. The thing that caused me to notice this was that opening a CSS screen that attempted to view ~50 PVs was taking 20-30 seconds to actually connect to all the PVs, whereas when using VxWorks it was instantaneous.

Under VxWorks, the time taken to complete a "caget" of 9 PVs is pretty constant regardless of how many records there are - all the way from 200 through to 60,000. All the records are simple analog inputs with a constant value. There are no support modules or any scan tasks running.

Under RTEMS, the time taken to complete a "caget" of the same 9 PVs is not constant. As the number of records grows, the IOC takes longer and longer to respond to the request. A table of the data is at the bottom of this email, but in summary at about 10k PVs the IOC takes about 1 second to respond to the request, compared to 0.03 seconds for VxWorks.

I cannot explain why this is happening. From my understanding of EPICS Channel Access, the time it takes to look up and respond to a caget should be near-constant and separate from the number of records in the IOC. There's a small possible number-of-records dependency due to the size of the hash table and collisions therein, but I tested using hash table sizes of 2048 and 65536 and this did not change the results - in fact the larger hash table size caused a 20% slowdown when using 60k records.

The only other observation I have is that these caget commands push the CPU usage of my crate to 100%. This is entirely contained within the CAS-UDP task. This is also something I cannot explain, as I was under the impression that a caget would respond via TCP, not UDP.

I am urgently looking for a solution to this, as I don't think we can run our IOCs on RTEMS if it will take 20 seconds just to open a screen.

Thank you for any assistance,

Alex Wells

RTEMS IOC data

Records	Mean (s)	Median (s)	Max (s)
200	0.031	0.031	0.057
1000	0.035	0.032	0.065
3000	0.096	0.096	0.099
6000	0.256	0.243	0.411
9000	0.445	0.371	0.801
12000	1.028	1.027	1.032
18000	0.931	0.712	1.842
21000	1.59	1.957	2.135
24000	2.216	2.216	2.401
36000	4.281	3.241	6.464
48000	5.651	4.365	8.424
60000	7.586	5.418	10.579

VxWorks IOC data

Records	Mean (s)	Median (s)	Max (s)
200	0.032	0.032	0.037
1000	0.032	0.032	0.037
3000	0.036	0.032	0.07
6000	0.035	0.032	0.07
9000	0.032	0.032	0.074
12000	0.036	0.032	0.071
18000	0.031	0.032	0.059
21000	0.032	0.032	0.038
24000	0.032	0.032	0.077
36000	0.032	0.032	0.056
48000	0.035	0.032	0.065
60000	0.039	0.033	0.073

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom.

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom.