Memory leak in QSRV/pvAccess server, triggered by incoming UDP search requests (not the known 7.0.10 beacon leak)

[Adam Piotrowski via Tech-talk <tech-talk at aps.anl.gov>]

Hi all,

We're seeing a reproducible memory leak in an IOC using QSRV/pvAccess on EPICS 7.0.10. Sharing what we've found so far.

Environment
- EPICS Base 7.0.10 (same problem on 7.0.9), cross-compiled with Buildroot for armv5tejl (Microchip SAM9X60)
- Target has 128MB RAM, no swap
- Runs on a busy control network with a lot of existing PVA/CA broadcast/multicast traffic from other devices

Symptom (on our full production IOC)
- Memory use grows steadily, about 130-145 KB per second, until the kernel kills the process from OOM - usually within 5-10 minutes. A separate non-EPICS process we run on the same device stays flat the whole time.

Minimal test case
We built a stripped-down test IOC to see if the leak still shows up with less running:
- EPICS Base 7.0.10 only, cross-compiled the same way
- QSRV/pvAccess linked in (base.dbd + PVAServerRegister.dbd + qsrv.dbd) - no asyn, no other support modules
- One soft ai record, just so there's a real channel to query
- Started on the same busy network segment as the production unit


We watched RSS with:
while true; do grep VmRSS /proc/$(pidof epicsTest)/status; sleep 3; done
Result: RSS grew from 46,436 kB to 61,088 kB in about 93 seconds - roughly 157 KB/s. Same order of magnitude as what we see on the full production IOC.

Not the known 7.0.10 beacon leak
We knew about the 7.0.10 fix for the PVA beacon leak, but our leak happens even with that fix in place. We checked that the binary is actually linked against the new 7.0.10 libraries, not some old mixed version. It correlates with incoming multicast traffic

We compared two identical units:
- One on a quiet network segment (almost no multicast traffic): memory stays flat.
- One on the busy production network (a lot of multicast traffic): memory leaks at the rate above.
Rough math: about 140 KB/s leaked, about 30 packets/s coming in - roughly 4.5 KB leaked per incoming packet, close to one memory page.

Further isolation on the production unit
We tried a few different configurations to narrow it down further. With QSRV/pvAccess included and normal traffic, it leaks at the ~140 KB/s rate described above. With QSRV/pvAccess removed from the build entirely (CA only), there's no leak. Blocking incoming multicast with iptables while keeping QSRV included also stops the leak, and removing that iptables rule again brings the leak straight back.

Questions

Is this a known issue? We couldn't find an existing report that matches.
Is there a setting to limit or throttle in-flight search responses? We checked the EPICS_PVA(S)_* variables and only found beacon-related ones.
We have a minimal reproduction case (described above) - happy to share the source, or run it under valgrind/heaptrack on a workstation build if that would help narrow this down.
Happy to share more details (ldd output, build config, network stats, the VmRSS log above) if needed.

Thanks,
Adam Piotrowski

Department of Microelectronics and Computer Science (DMCS)
Lodz University of Technology
Wolczanska 221/223, 90-924 Lodz, Poland
apiotro at dmcs.pl
https://urldefense.us/v3/__https://dmcs.p.lodz.pl__;!!G_uCfscf7eWS!buQgLXDLKlmijlBmBO_rBdfupGRXcKfmHh_RP13AiRy2V_ECGWL-tJdP6-fJCnFc-lb9xy9sabqHJX3rGMgg0g$