RE: EPICS IOCs in Kubernetes — Community Experience?

["Pearson, Matthew via Tech-talk" <tech-talk at aps.anl.gov>]

Hi,

 

Thanks for the link to the paper, it looks excellent.

 

I’ve been interested in the use of containers for a while but I am very concerned about the overhead of setting up and maintaining the software infrastructure, so I’d be interested to learn about the resources used to get to the stage of deploying IOCs this way on a production beamline (I see that the paper discusses some of this too).

 

More specifically, I have some questions on the practicalities:

 

1. Are there people dedicated to maintaining the software/IT infrastructure? Are these controls systems engineers or specialized DevOps type people?
2. Is the workflow understandable for everyone in the group? I mean, if there is an issue can anyone start to investigate or is like a black box process that magically produces containerized IOCs?
3. How do non-developers work with the process? We have non-developers who create OPI screens and even basic IOC applications, and we’re lucky if those even end up version controlled. (I know this is more an organizational issue – but I’m sure other sites have similar issues).
4. Have you had any issues during operations related to the container infrastructure?
5. If you have an issue with an IOC in operations, requiring hands-on diagnosis with equipment in the field, how is the software edited and debugged? (I suppose people can still run up IOCs as needed on the base OS for troubleshooting?)
6. How are security patches managed (or monitored) for the software inside the containers?
7. Is the extra staff and computing/software infrastructure worth the effort, over not doing this at all and just running IOCs in a normal virtualized OS environment?
8. Is the main benefit a workflow driven by everything having to be version controlled and deployed that way (i.e., forcing everything to be version controlled – the GitOps paradigm mentioned in the paper)? Rather than the other benefits of containers like OS isolation?

 

Cheers,

Matt

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Andrea Michelotti via Tech-talk
Sent: Friday, August 28, 2026 3:58 AM
To: Williams, Ernest L., Jr <ernesto at slac.stanford.edu>; EPICS Tech Talk <tech-talk at aps.anl.gov>; Knap, Giles (DLSLtd,RAL,CEO) <giles.knap at diamond.ac.uk>
Subject: [EXTERNAL] Re: EPICS IOCs in Kubernetes — Community Experience?

 

Hi, I'm on holiday right now so can't go into detail, but you should definitely talk to Giles Knap at Diamond Light Source (DLS) — he's been a pioneer in this space with the "epics-containers" project, and has likely already solved (or at least

Hi, I'm on holiday right now so can't go into detail, but you should definitely talk to Giles Knap at Diamond Light Source (DLS) — he's been a pioneer in this space with the "epics-containers" project, and has likely already solved (or at least characterized) the Pod-IP/rediscovery problem you're describing for CA and PVA even if we use a slight different approach. Also worth pinging Claudio Bisegni at SLAC directly if you haven't already — He's working on k8s there and we are in touch.

On our side, we run what we call EPIK8S (EPICS on Kubernetes) successfully in production on four beamlines. Some info here if useful: https://iopscience.iop.org/article/10.1088/1748-0221/21/04/P04010

Happy to see you moving this way — we think it really makes things much easier. Will follow up with more detail when I'm back if useful.

Best,

Andrea Michelotti

--------------------

dr. Andrea Michelotti

Head of Control Systems Service

INFN - Laboratori Nazionali di Frascati

Accelerator Division,

Bldg.2, Room 120

Via Enrico Fermi, 40

00044 Frascati (RM)

e-mail:   andrea.michelotti at infn.it

office:   (+39) 06.9403.2272

mobile:   (+39) 06.9403.8203

fax   :   (+39) 06.9403.2256

Teams :   amichelo at infn.it

LinkedIn: http://it.linkedin.com/in/michelotti

 

Inviato da Outlook per Android

 

---

Da: Tech-talk <tech-talk-bounces at aps.anl.gov> per conto di Williams, Ernest L., Jr via Tech-talk <tech-talk at aps.anl.gov>
Inviato: venerdì, agosto 28, 2026 8:34:13 AM
A: EPICS Tech Talk <tech-talk at aps.anl.gov>
Oggetto: EPICS IOCs in Kubernetes — Community Experience?

 

Hi all,

At SLAC, we have started working on integrating EPICS services and IOCs into Kubernetes, and would be interested to hear from others in the EPICS community who are already running EPICS successfully in Kubernetes.

Our current environment includes EPICS Base 7.0.9, Channel Access, EPICS Base PVAccess, PVXS 1.5.2. And p4p 4.2.2.   IOCs and other EPICS services run in Docker containers as Kubernetes Pods, with Kubernetes Services providing stable network endpoints. Our Kubernetes environment currently uses Cilium for the network layer.

One area I am currently investigating is IOC recovery when Pods are restarted or replaced, particularly how CA and PVA clients rediscover an IOC when its Pod IP changes while the Kubernetes Service remains stable. This has raised some interesting questions around the interaction between EPICS discovery and the Kubernetes network layer.

I would be very interested in hearing from anyone running EPICS IOCs or other EPICS services in Kubernetes:

·       What Kubernetes networking architecture are you using (for example, Cilium, Calico, Flannel, or another CNI)?

·       Are you running CA, PVA, p4p/Python PVA services, or some combination?

·       How are you handling EPICS discovery within the cluster and between Kubernetes and external EPICS networks?

·       How do your clients and IOCs behave during Pod replacement, rolling updates, rescheduling, or other events where the Pod IP changes?

·       Are you using Kubernetes Services as stable endpoints for EPICS traffic?

·       What EPICS/Kubernetes integration problems have you encountered and solved?

I am mainly looking to compare experiences and learn what patterns are working well elsewhere in the community.

 

 

Cheers,

Ernest Williams
Dept. Head, Controls Software Engineering
Instrumentation Division, Technology Innovation Directorate
SLAC National Laboratory, Stanford University