EPICS IOCs in Kubernetes — Community Experience?

["Williams, Ernest L., Jr via Tech-talk" <tech-talk at aps.anl.gov>]

Hi all,

At SLAC, we have started working on integrating EPICS services and IOCs into Kubernetes, and would be interested to hear from others in the EPICS community who are already running EPICS successfully in Kubernetes.

Our current environment includes EPICS Base 7.0.9, Channel Access, EPICS Base PVAccess, PVXS 1.5.2. And p4p 4.2.2.   IOCs and other EPICS services run in Docker containers as Kubernetes Pods, with Kubernetes Services providing stable network endpoints. Our Kubernetes environment currently uses Cilium for the network layer.

One area I am currently investigating is IOC recovery when Pods are restarted or replaced, particularly how CA and PVA clients rediscover an IOC when its Pod IP changes while the Kubernetes Service remains stable. This has raised some interesting questions around the interaction between EPICS discovery and the Kubernetes network layer.

I would be very interested in hearing from anyone running EPICS IOCs or other EPICS services in Kubernetes:

• What Kubernetes networking architecture are you using (for example, Cilium, Calico, Flannel, or another CNI)?
• Are you running CA, PVA, p4p/Python PVA services, or some combination?
• How are you handling EPICS discovery within the cluster and between Kubernetes and external EPICS networks?
• How do your clients and IOCs behave during Pod replacement, rolling updates, rescheduling, or other events where the Pod IP changes?
• Are you using Kubernetes Services as stable endpoints for EPICS traffic?
• What EPICS/Kubernetes integration problems have you encountered and solved?

I am mainly looking to compare experiences and learn what patterns are working well elsewhere in the community.

Cheers,

Ernest Williams
Dept. Head, Controls Software Engineering
Instrumentation Division, Technology Innovation Directorate
SLAC National Laboratory, Stanford University