Re: EPICS IOCs in Kubernetes — Community Experience?

["Knap, Giles \(DLSLtd,RAL,DOD\) via Tech-talk" <tech-talk at aps.anl.gov>]

As promised, some answers to Ernest and Matts Qs re EPICS in Kubernetes.

Matts Qs:

• Are there people dedicated to maintaining the software/IT infrastructure? Are these controls systems engineers or specialized DevOps type people?
• Yes we have a team of 6 people that look after all our clusters.
• We have a cluster per beamline, a big central cluster, another for the accelerator and several staging clusters
• Note that more and more of our infra is moving to K8S so the amount of traditional support required is going down with time
• Is the workflow understandable for everyone in the group? I mean, if there is an issue can anyone start to investigate or is like a black box process that magically produces containerized IOCs?
• Yes, we have many of our traditional EPICS beamline support people converting their beamlines to K8S
• We have extensive documentation here: https://epics-containers.github.io/main/index.html
• On top of that we have internal docs with the specifics of the repos and clusters at DLS
• Many engineers have got up to speed using the docs alone
• How do non-developers work with the process? We have non-developers who create OPI screens and even basic IOC applications, and we’re lucky if those even end up version controlled. (I know this is more an organizational issue – but I’m sure other sites have similar issues).
• For end users we have developed a Web UI called argoCD monitor that allows them to view logs, stop/start IOCs
• It uses the ArgoCD API and is an alternative to the native UI which exposes too much of the underlying detail
• See https://github.com/epics-containers/argocd-monitor#example-screenshot for an idea of what that looks like
• Have you had any issues during operations related to the container infrastructure?
• This has not been an issue recently - one niggle that happened once is that our accelerator network is isolated and can't pull containers during the run - that's by design but we did get it wrong once
• We are protected by having a local container image cache called Harbour running so even when the internet is broken previously pulled images are still available
• This is also how we manage the accelerator - its harbour is internet connected during shutdowns and can pull image updates, then use just the cached images during a run
• If you have an issue with an IOC in operations, requiring hands-on diagnosis with equipment in the field, how is the software edited and debugged? (I suppose people can still run up IOCs as needed on the base OS for troubleshooting?)
• Two things.
• Primary choice for easy access we use a developer container running on a workstation on the beamline subnet
• Second choice is debugging in cluster, we have some interesting solutions for this, even allowing temporary patching and compiling of code changes that survive a container restart (using PVCs and mounts)
• This breaks the immutability of containers so is only used when absolutely needed
• But the good thing is you can go into argoCD monitor and sort your IOC list by tracking branch. Anything that is not on main is to be addressed. Thus we don't forget that we've left IOCs in a debug state.
• How are security patches managed (or monitored) for the software inside the containers?
• This comes with the Infra - we have StackRox running on our clusters, constantly monitoring all live containers against the current CVE databases
• Most CVEs don't affect IOCs because they don't expose any sensitive ports. If they do then we can bump dependencies and rebuild our containers - as everything is defined in manifests it is easy to discover everything that is affected by a given dependency bump. Container builds all happen in CI anyway.
• Is the extra staff and computing/software infrastructure worth the effort, over not doing this at all and just running IOCs in a normal virtualized OS environment?

• I like to think of it as less infrastructure!

• We have tried very hard to make this Standard EPICS (using upstream base and support modules) and standard EPICS build system + a standard Kubernetes

• We keep the glue to a minimum

• We do add our IOC generation thing called ibek (Equivalent to DLS XmlBuilder which I expect you recall, Matt) but this is a completely optional layer on top

• So if your infra ran on K8S already then you don't need anything extra to start deploying IOCs

• We made epics-containers as FOSS with the idea that this could be adopted and shared by multiple facilities, reducing the maintenance burden. We do have INFN and Tau Systems collaborating on the project already.

• But I get that this is a rather idealist view and I can't honestly quantify the cost/reward on this

• Is the main benefit a workflow driven by everything having to be version controlled and deployed that way (i.e., forcing everything to be version controlled – the GitOps paradigm mentioned in the paper)? Rather than the other benefits of containers like OS isolation?
• The continuous deployment of released assets is a big win.
• I assert that a non EPICS expert (who knows a little K8S) can look at one of our beamlines and trace each IOC down to lines of code in github
• But the isolation is also useful, one memory leaking  or CPU hogging IOC no longer affects a whole server and such issues can be found and corrected much more easily
• You also get centralized monitoring and logging
• A very big win for us is that we can upgrade the OS on our servers without needing to recompile all of our IOCs.

Ernest's general questions:

• Re CNI
• We are moving to Cilium and I'm not sure if that's done yet Chris Reynolds ?
• Re EPICS transport
• We use CA and p4p
• Re PV search
• Discovery just works on hostNetwork - its identical to old server architecture network wise
• Re pod moves:
• IOCs that get moved just work in hostNetwork because broadcast reconnects them
• Re Pod replacement:
• we use statefulsets so that the old pod dies and drops any singleton device connections before the new one starts
• Re services for accessing IOCs:
• no services for IOCs as hostNetwork
• Some problems we have solved:
• deciding to go host network (and providing Kyverno rules to make sure thats not a security hole)
• solved almost every issue we have had with CA/PVA and gigE camera streaming protocol
• That's because IOCs networking looks exactly like it did in the old infrastructure
• see https://epics-containers.github.io/main/explanations/epics_protocols.html
• The other thing we did is go for small IOCs, 1 per device typically and then manage them all with ArgoCD - meaning the state of each beamline and its history is described in a git repo. Example beamline: https://github.com/epics-containers/p47-services

Ernest's specific question about when pods are restarted or replaced. The problem is that they may get a new IP address and a CA client that has established a channel with the pod will loose contact, CA client libraries assume that the can use DNS once to get an IP address and do not re-establish the channel when this happens.

• The solution developed at INFN in Rome was to give every IOC a deterministic cluster IP address based on a hash of the Helm Release.
• We have this working for our IOCs that use cluster networking but generally don't use it because our production IOCs are host network
• Here is our implementation https://github.com/epics-containers/ec-helm-charts/blob/090ef8b36d0fee6616d1bd825cca1ae83c28fa17/Charts/ioc-instance/templates/_service.tpl#L24-L26
• This sets the cluster IP of the IOC to a deterministic address based on the helm release name.
• Even if the pod is moved to a different machine it still gets the same address. 

---

From: Knap, Giles (DLSLtd,RAL,DOD) <giles.knap at diamond.ac.uk>
Sent: 28 August 2026 18:05
To: Pearson, Matthew <pearsonmr at ornl.gov>; Andrea Michelotti <andrea.michelotti at lnf.infn.it>; Williams, Ernest L., Jr <ernesto at slac.stanford.edu>; EPICS Tech Talk <tech-talk at aps.anl.gov>
Cc: Knap, Giles (DLSLtd,RAL,DOD) <giles.knap at diamond.ac.uk>; Cobb, Tom (DLSLtd,RAL,DOD) <tom.cobb at diamond.ac.uk>
Subject: Re: EPICS IOCs in Kubernetes — Community Experience?

 

Hi Ernest, Matt,

I've made a brief answer to Ernest on Element and will respond to this email in detail on Tues morning BST.

In short, much of what you both ask is discussed in our documentation here https://epics-containers.github.io/.

Cheers,

giles

---

From: Pearson, Matthew <pearsonmr at ornl.gov>
Sent: 28 August 2026 15:47
To: Andrea Michelotti <andrea.michelotti at lnf.infn.it>; Williams, Ernest L., Jr <ernesto at slac.stanford.edu>; EPICS Tech Talk <tech-talk at aps.anl.gov>; Knap, Giles (DLSLtd,RAL,DOD) <giles.knap at diamond.ac.uk>
Subject: RE: EPICS IOCs in Kubernetes — Community Experience?

 

You don't often get email from pearsonmr at ornl.gov. Learn why this is important

Hi,

 

Thanks for the link to the paper, it looks excellent.

 

I’ve been interested in the use of containers for a while but I am very concerned about the overhead of setting up and maintaining the software infrastructure, so I’d be interested to learn about the resources used to get to the stage of deploying IOCs this way on a production beamline (I see that the paper discusses some of this too).

 

More specifically, I have some questions on the practicalities:

 

1. Are there people dedicated to maintaining the software/IT infrastructure? Are these controls systems engineers or specialized DevOps type people?
2. Is the workflow understandable for everyone in the group? I mean, if there is an issue can anyone start to investigate or is like a black box process that magically produces containerized IOCs?
3. How do non-developers work with the process? We have non-developers who create OPI screens and even basic IOC applications, and we’re lucky if those even end up version controlled. (I know this is more an organizational issue – but I’m sure other sites have similar issues).
4. Have you had any issues during operations related to the container infrastructure?
5. If you have an issue with an IOC in operations, requiring hands-on diagnosis with equipment in the field, how is the software edited and debugged? (I suppose people can still run up IOCs as needed on the base OS for troubleshooting?)
6. How are security patches managed (or monitored) for the software inside the containers?
7. Is the extra staff and computing/software infrastructure worth the effort, over not doing this at all and just running IOCs in a normal virtualized OS environment?
8. Is the main benefit a workflow driven by everything having to be version controlled and deployed that way (i.e., forcing everything to be version controlled – the GitOps paradigm mentioned in the paper)? Rather than the other benefits of containers like OS isolation?

 

Cheers,

Matt

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Andrea Michelotti via Tech-talk
Sent: Friday, August 28, 2026 3:58 AM
To: Williams, Ernest L., Jr <ernesto at slac.stanford.edu>; EPICS Tech Talk <tech-talk at aps.anl.gov>; Knap, Giles (DLSLtd,RAL,CEO) <giles.knap at diamond.ac.uk>
Subject: [EXTERNAL] Re: EPICS IOCs in Kubernetes — Community Experience?

 

Hi, I'm on holiday right now so can't go into detail, but you should definitely talk to Giles Knap at Diamond Light Source (DLS) — he's been a pioneer in this space with the "epics-containers" project, and has likely already solved (or at least

Hi, I'm on holiday right now so can't go into detail, but you should definitely talk to Giles Knap at Diamond Light Source (DLS) — he's been a pioneer in this space with the "epics-containers" project, and has likely already solved (or at least characterized) the Pod-IP/rediscovery problem you're describing for CA and PVA even if we use a slight different approach. Also worth pinging Claudio Bisegni at SLAC directly if you haven't already — He's working on k8s there and we are in touch.

On our side, we run what we call EPIK8S (EPICS on Kubernetes) successfully in production on four beamlines. Some info here if useful: https://iopscience.iop.org/article/10.1088/1748-0221/21/04/P04010

Happy to see you moving this way — we think it really makes things much easier. Will follow up with more detail when I'm back if useful.

Best,

Andrea Michelotti

--------------------

dr. Andrea Michelotti

Head of Control Systems Service

INFN - Laboratori Nazionali di Frascati

Accelerator Division,

Bldg.2, Room 120

Via Enrico Fermi, 40

00044 Frascati (RM)

e-mail:   andrea.michelotti at infn.it

office:   (+39) 06.9403.2272

mobile:   (+39) 06.9403.8203

fax   :   (+39) 06.9403.2256

Teams :   amichelo at infn.it

LinkedIn: http://it.linkedin.com/in/michelotti

 

Inviato da Outlook per Android

 

---

Da: Tech-talk <tech-talk-bounces at aps.anl.gov> per conto di Williams, Ernest L., Jr via Tech-talk <tech-talk at aps.anl.gov>
Inviato: venerdì, agosto 28, 2026 8:34:13 AM
A: EPICS Tech Talk <tech-talk at aps.anl.gov>
Oggetto: EPICS IOCs in Kubernetes — Community Experience?

 

Hi all,

At SLAC, we have started working on integrating EPICS services and IOCs into Kubernetes, and would be interested to hear from others in the EPICS community who are already running EPICS successfully in Kubernetes.

Our current environment includes EPICS Base 7.0.9, Channel Access, EPICS Base PVAccess, PVXS 1.5.2. And p4p 4.2.2.   IOCs and other EPICS services run in Docker containers as Kubernetes Pods, with Kubernetes Services providing stable network endpoints. Our Kubernetes environment currently uses Cilium for the network layer.

One area I am currently investigating is IOC recovery when Pods are restarted or replaced, particularly how CA and PVA clients rediscover an IOC when its Pod IP changes while the Kubernetes Service remains stable. This has raised some interesting questions around the interaction between EPICS discovery and the Kubernetes network layer.

I would be very interested in hearing from anyone running EPICS IOCs or other EPICS services in Kubernetes:

·       What Kubernetes networking architecture are you using (for example, Cilium, Calico, Flannel, or another CNI)?

·       Are you running CA, PVA, p4p/Python PVA services, or some combination?

·       How are you handling EPICS discovery within the cluster and between Kubernetes and external EPICS networks?

·       How do your clients and IOCs behave during Pod replacement, rolling updates, rescheduling, or other events where the Pod IP changes?

·       Are you using Kubernetes Services as stable endpoints for EPICS traffic?

·       What EPICS/Kubernetes integration problems have you encountered and solved?

I am mainly looking to compare experiences and learn what patterns are working well elsewhere in the community.

 

 

Cheers,

Ernest Williams
Dept. Head, Controls Software Engineering
Instrumentation Division, Technology Innovation Directorate
SLAC National Laboratory, Stanford University

 

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom.