Re: EPICS IOCs in Kubernetes — Community Experience?

[Andrea michelotti via Tech-talk <tech-talk at aps.anl.gov>]

Hi all, I'm back.

A quick answer regarding CNIs and the EPICS protocols reconnection issue discussed in https://github.com/epics-base/epics-base/issues/488.

At INFN, we use RKE2 with Calico and Multus—the latter primarily for GigE Vision devices and access special networks. Since UDP broadcasts are unavailable in a standard K8s infrastructure, we rely on TCP/UDP direct connection. Variables such as PVA_NAME_SERVERS and EPICS_CA_ADDR_LIST are centrally configured and updated whenever IOCs are added to or removed from the control system.

IOC and service lifecycles are managed through ArgoCD, with each IOC deployment represented by a separate Argo CD application.

To allow EPICS clients to reconnect reliably after an IOC restart, we deploy each IOC as a StatefulSet and assign its Service a static ClusterIP, deterministically calculated by hashing the IOC name.

I am also experimenting with a second, potentially more elegant approach on a small beamline: configuring the Argo CD IOC applications so that the Service is preserved when an IOC is restarted or redeployed. This allows the Service to retain the IP address originally assigned by Kubernetes.

I have also copied Claudio Bisegni, a passionate developer who, like Giles and me, enjoys making control systems less boring by constantly creating, experimenting with, and adopting new solutions. Since 2012, Claudio and I have had great fun introducing emerging IT technologies into our custom control system, !CHAOS that in 2023 I phased out definitevely to focus on EPICS... but with steroids. Back then, we thought EPICS was something that belonged in the past—and yet, somewhat ironically, we have both ended up working with EPICS today. :-)

Ciao,
Andrea
 

On 28/08/2026 19:05, Knap, Giles (DLSLtd,RAL,DOD) wrote:

Hi Ernest, Matt,

I've made a brief answer to Ernest on Element and will respond to this email in detail on Tues morning BST.

In short, much of what you both ask is discussed in our documentation here https://epics-containers.github.io/.

Cheers,

giles

---

From: Pearson, Matthew <pearsonmr at ornl.gov>
Sent: 28 August 2026 15:47
To: Andrea Michelotti <andrea.michelotti at lnf.infn.it>; Williams, Ernest L., Jr <ernesto at slac.stanford.edu>; EPICS Tech Talk <tech-talk at aps.anl.gov>; Knap, Giles (DLSLtd,RAL,DOD) <giles.knap at diamond.ac.uk>
Subject: RE: EPICS IOCs in Kubernetes — Community Experience?

 

You don't often get email from pearsonmr at ornl.gov. Learn why this is important

Hi,

 

Thanks for the link to the paper, it looks excellent.

 

I’ve been interested in the use of containers for a while but I am very concerned about the overhead of setting up and maintaining the software infrastructure, so I’d be interested to learn about the resources used to get to the stage of deploying IOCs this way on a production beamline (I see that the paper discusses some of this too).

 

More specifically, I have some questions on the practicalities:

 

1. Are there people dedicated to maintaining the software/IT infrastructure? Are these controls systems engineers or specialized DevOps type people?
2. Is the workflow understandable for everyone in the group? I mean, if there is an issue can anyone start to investigate or is like a black box process that magically produces containerized IOCs?
3. How do non-developers work with the process? We have non-developers who create OPI screens and even basic IOC applications, and we’re lucky if those even end up version controlled. (I know this is more an organizational issue – but I’m sure other sites have similar issues).
4. Have you had any issues during operations related to the container infrastructure?
5. If you have an issue with an IOC in operations, requiring hands-on diagnosis with equipment in the field, how is the software edited and debugged? (I suppose people can still run up IOCs as needed on the base OS for troubleshooting?)
6. How are security patches managed (or monitored) for the software inside the containers?
7. Is the extra staff and computing/software infrastructure worth the effort, over not doing this at all and just running IOCs in a normal virtualized OS environment?
8. Is the main benefit a workflow driven by everything having to be version controlled and deployed that way (i.e., forcing everything to be version controlled – the GitOps paradigm mentioned in the paper)? Rather than the other benefits of containers like OS isolation?

 

Cheers,

Matt

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Andrea Michelotti via Tech-talk
Sent: Friday, August 28, 2026 3:58 AM
To: Williams, Ernest L., Jr <ernesto at slac.stanford.edu>; EPICS Tech Talk <tech-talk at aps.anl.gov>; Knap, Giles (DLSLtd,RAL,CEO) <giles.knap at diamond.ac.uk>
Subject: [EXTERNAL] Re: EPICS IOCs in Kubernetes — Community Experience?

 

Hi, I'm on holiday right now so can't go into detail, but you should definitely talk to Giles Knap at Diamond Light Source (DLS) — he's been a pioneer in this space with the "epics-containers" project, and has likely already solved (or at least

Hi, I'm on holiday right now so can't go into detail, but you should definitely talk to Giles Knap at Diamond Light Source (DLS) — he's been a pioneer in this space with the "epics-containers" project, and has likely already solved (or at least characterized) the Pod-IP/rediscovery problem you're describing for CA and PVA even if we use a slight different approach. Also worth pinging Claudio Bisegni at SLAC directly if you haven't already — He's working on k8s there and we are in touch.

On our side, we run what we call EPIK8S (EPICS on Kubernetes) successfully in production on four beamlines. Some info here if useful: https://iopscience.iop.org/article/10.1088/1748-0221/21/04/P04010

Happy to see you moving this way — we think it really makes things much easier. Will follow up with more detail when I'm back if useful.

Best,

Andrea Michelotti

--------------------

dr. Andrea Michelotti

Head of Control Systems Service

INFN - Laboratori Nazionali di Frascati

Accelerator Division,

Bldg.2, Room 120

Via Enrico Fermi, 40

00044 Frascati (RM)

e-mail:   andrea.michelotti at infn.it

office:   (+39) 06.9403.2272

mobile:   (+39) 06.9403.8203

fax   :   (+39) 06.9403.2256

Teams :   amichelo at infn.it

LinkedIn: http://it.linkedin.com/in/michelotti

 

Inviato da Outlook per Android

 

---

Da: Tech-talk <tech-talk-bounces at aps.anl.gov> per conto di Williams, Ernest L., Jr via Tech-talk <tech-talk at aps.anl.gov>
Inviato: venerdì, agosto 28, 2026 8:34:13 AM
A: EPICS Tech Talk <tech-talk at aps.anl.gov>
Oggetto: EPICS IOCs in Kubernetes — Community Experience?

 

Hi all,

At SLAC, we have started working on integrating EPICS services and IOCs into Kubernetes, and would be interested to hear from others in the EPICS community who are already running EPICS successfully in Kubernetes.

Our current environment includes EPICS Base 7.0.9, Channel Access, EPICS Base PVAccess, PVXS 1.5.2. And p4p 4.2.2.   IOCs and other EPICS services run in Docker containers as Kubernetes Pods, with Kubernetes Services providing stable network endpoints. Our Kubernetes environment currently uses Cilium for the network layer.

One area I am currently investigating is IOC recovery when Pods are restarted or replaced, particularly how CA and PVA clients rediscover an IOC when its Pod IP changes while the Kubernetes Service remains stable. This has raised some interesting questions around the interaction between EPICS discovery and the Kubernetes network layer.

I would be very interested in hearing from anyone running EPICS IOCs or other EPICS services in Kubernetes:

·       What Kubernetes networking architecture are you using (for example, Cilium, Calico, Flannel, or another CNI)?

·       Are you running CA, PVA, p4p/Python PVA services, or some combination?

·       How are you handling EPICS discovery within the cluster and between Kubernetes and external EPICS networks?

·       How do your clients and IOCs behave during Pod replacement, rolling updates, rescheduling, or other events where the Pod IP changes?

·       Are you using Kubernetes Services as stable endpoints for EPICS traffic?

·       What EPICS/Kubernetes integration problems have you encountered and solved?

I am mainly looking to compare experiences and learn what patterns are working well elsewhere in the community.

 

 

Cheers,

Ernest Williams
Dept. Head, Controls Software Engineering
Instrumentation Division, Technology Innovation Directorate
SLAC National Laboratory, Stanford University

 

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom.

-- 
---------------------
dr. Andrea Michelotti
Head of Control Systems Service

INFN - Laboratori Nazionali di Frascati
Accelerator Division,
Bldg.2, Room 120
Via Enrico Fermi, 40
00044 Frascati (RM)

e-mail:   andrea.michelotti at infn.it
office:   (+39) 06.9403.2272
mobile:   (+39) 06.9403.8203
fax   :   (+39) 06.9403.2256
Teams :   amichelo at infn.it
LinkedIn: http://it.linkedin.com/in/michelotti