Re: Phoebus is unable to see PVs inside docker containers

["Knap, Giles \(DLSLtd,RAL,LSCI\) via Tech-talk" <tech-talk at aps.anl.gov>]

Did you try simply not setting org.phoebus.pv.ca/addr_list (or setting it to your subnet broadcast address)? I don't believe you needed to go to the trouble of modifying iptables to talk to multiple local IOCs - broadcast should work.

My apologies for talking in terms of environment variables (for pvget) instead of phoebus ini file settings in my first response - that may have been a little confusing!

---

From: Srinivas, Dhruv <srinivas1 at llnl.gov>
Sent: 04 April 2025 19:35
To: Knap, Giles (DLSLtd,RAL,LSCI) <giles.knap at diamond.ac.uk>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>; Jörn Dreyer <j.dreyer at hzdr.de>
Subject: RE: Phoebus is unable to see PVs inside docker containers

 

You don't often get email from srinivas1 at llnl.gov. Learn why this is important

Thanks for all the advice everyone, I was able to get it working thanks to it. What ended up working for me was setting the addr_list to localhost in the settings.ini file as well as adding the script in the following link to /etc/network/if-up.d and /etc/network/if-down.d

 

https://docs.epics-controls.org/en/latest/sys-admin/channel-access-reach-multiple-soft-iocs-linux.html

 

Also, thank you for the nuanced explanation of why only the first IOC would connect. That explanation fits perfectly with the issues I was seeing, and it’ll help me debug it in the future.

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Knap, Giles (DLSLtd,RAL,LSCI) via Tech-talk
Sent: Friday, April 4, 2025 12:12 AM
To: tech-talk at aps.anl.gov; Jörn Dreyer <j.dreyer at hzdr.de>; Knap, Giles (DLSLtd,RAL,LSCI) <giles.knap at diamond.ac.uk>
Subject: Re: Phoebus is unable to see PVs inside docker containers

 

I thought I'd follow up by remarking that running multiple IOCs in host network containers has one small nuance.

 

carepeater is started inside the first container that launches. The others don't launch it because the carepeater port on the host is already taken.

 

If you then stop that first IOC, you will take down the carepeater and will have none until the next IOC is started.

 

Most of the time this is not an issue but it can be an explanation for unexpected behaviour with respect to clients retrying connections to PVs.

---

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Knap, Giles (DLSLtd,RAL,LSCI) via Tech-talk <tech-talk at aps.anl.gov>
Sent: 04 April 2025 07:23
To: tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>; Jörn Dreyer <j.dreyer at hzdr.de>
Subject: Re: Phoebus is unable to see PVs inside docker containers

 

HI Dhruv,

 

When you are using network host, your IOCs should behave exactly as if they were running bare metal from a networking perspective. Therefore phoebus should be able to see them with default configuration. This is how we run our IOCs in Kubernetes at DLS and we can see PVA and CA for all of them using UDP broadcast from anywhere in the same subnet.

 

I have also made a branch of my example simulation beamline that demonstrates this using docker, host network and a default configured phoebus. You could try it out by following the steps in the README.md https://github.com/epics-containers/example-services/tree/docker-host-network. Maybe if that works for you, you can investigate what is different in your own setup.

 

The symptoms you describe sound like you are using UDP unicast to your server running docker. I.e. You have its address in EPICS_CA_ADDR_LIST. This will always see the first IOC only. You need to have EPICS_CA_ADDR_LIST unset (and EPICS_CA_AUTO_ADDR_LIST unset or "Y") in order to use broadcast and therefore see all IOCs. If your client is not on the same subnet as your IOCs then  this won't work and you will need a CA Gateway.

 

One other thing to check: I know that host network for docker rootless mode was completely broken last time I tried. I'm guessing that you are using docker in it's default rootful mode but thought it was worth checking that.

 

If you end up needing a gateway then the main branch of the linked repo demonstrates how to do that too with on CA gateway (and a pvagw) forwarding CA (PVA) into the container network.

 

Regards,

giles

---

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Jörn Dreyer via Tech-talk <tech-talk at aps.anl.gov>
Sent: 04 April 2025 04:55
To: tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Re: Phoebus is unable to see PVs inside docker containers

 

Hello Dhruv,

how is your org.phoebus.pv.ca/addr_list formated? It should be a list of IP's seperated by spaces. The observation you have
points to a problem here as it looks like your phoebus sees probably only the first host on that list.
You can check the settings that phoebus sees if you go "Help -> Abaut -> Show Details -> Preference Settings"

You could try to set EPICS_CAS_INTF_ADDR_LIST to be the address of the docker network interface before starting phoebus.
You need to kill caRepeater on the host before starting Poebus. Otherwise the change will not have any effect.

If that does not help, probably the easisest way to go is to implement a gateway on the Docker host.

Hope this helps,

Jörn

Am Donnerstag, 3. April 2025, 19:51:20 Mitteleuropäische Sommerzeit schrieb Srinivas, Dhruv via Tech-talk:
> Hello,
>
> I'm installing several IOCs on a new machine. If I run it locally on the machine itself, then Phoebus is able to see the PVs in said IOCs. However, when I place the IOCs into a docker image then run it using the following command, Phoebus can only see one IOC at a time:
>
> "docker run -dit --network=host --name=<name> <image_name>"
>
> So far I've attempted to set auto_addr_list, addr_list, and name_servers in the Phoebus settings.ini file. So far I've been able to confirm that my setting changes did take effect, but the problem persists.
> Does anyone happen to know how to resolve this issue?
>
> Sincerely,
>
> Dhruv Srinivas
> srinivas1 at llnl.gov<mailto:srinivas1 at llnl.gov>
>
>
>

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom.

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom.

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom.