Re: Help in debugging ACF connection issue

["Hu, Yong via Tech-talk" <tech-talk at aps.anl.gov>]

Hello again, Érico,

I think I have managed to reproduce your ACF connection issue. Here is what I did:

1.  Turn off my laptop's networks completely, including WIFI.  

2. Open a terminal and use tcpdump to monitor EPICS search: # tcpdump -ni any udp port 5064

3.  Manually start the first softIOC 'bpm-fron':  it has an ACF which reads a PV (myCommand:just4TestACF) that is provided by the second softIOC 'bpm-rear' .

4.  Immediately manually start the second softIOC 'bpm-rear'.

Here is my debugging:

0) I wait for a long time, more than 5 minutes. 

1) On 'bpm-front': ascar shows 'connected:never'; interestingly, dbcar shows '3 connected ...'  because I do have 3 dbCA links, something like 'LNK2,"myCommand:just4Test CA"' inside the softIOC 'bpm-front'.

epics> ascar 1

connected:never  read:no  write:no  myCommand:just4TestACF <disconnected>

1 channels 1 not connected

epics> dbcar

CA links in all records

Total 3 CA links; 3 connected, 0 not connected.

2) On 'bpm-rear': there are no clients connected. 

epics> casr

Channel Access Server V4.13

No clients connected.

Here is a snippet of tcpdump which makes sense to me: 'bpm-front' sends UDP search through 127.0.0.1 at varied intervals, eventually every 4.5 minutes (<EPICS_CA_MAX_SEARCH_PERIOD=300-sec) continuously (not timeout as Ralph corrected my wording in another thread).

15:03:31.797239 IP 127.0.0.1.59092 > 127.0.0.1.5064: UDP, length 56

15:03:32.821997 IP 127.0.0.1.59092 > 127.0.0.1.5064: UDP, length 56

15:03:34.869266 IP 127.0.0.1.59092 > 127.0.0.1.5064: UDP, length 56

...

15:12:15.058636 IP 127.0.0.1.59092 > 127.0.0.1.5064: UDP, length 56

15:16:37.198335 IP 127.0.0.1.59092 > 127.0.0.1.5064: UDP, length 56

15:20:59.337359 IP 127.0.0.1.59092 > 127.0.0.1.5064: UDP, length 56

3) test caget and camonitor, which fail:

$ caget myCommand:just4TestACF

Channel connect timed out: 'myCommand:just4TestACF' not found.

$ camonitor myCommand:just4TestACF

myCommand:just4TestACF         *** Not connected (PV not found)

osiLocalAddr(): only loopback found

4) Now, the most intriguing part: turn on laptop's networks (actually just WIFI). caget and camonitor both are working, getting the PV values ; tcpdump output message have changed a little bit because my laptop has a new IP '192.168.12.44'

15:40:12.856529 IP 192.168.12.44.53273 > 192.168.12.255.5064: UDP, length 56

15:40:12.857207 IP 192.168.12.44.5064 > 192.168.12.44.53273: UDP, length 40

However, on 'bpm-front', ascar still shows 'connected:never'. That is exactly what happened to your BPM system. 

Conclusion: just as I mentioned below, most likely your IOCs start before your server's network is fully functional (valid IP address). 

To CA developers: Is this a feature or a bug in the Access security Channel Access (asCa)?

Cheers,

Yong

From: Hu, Yong <yhu at bnl.gov>
Date: Thursday, March 5, 2026 at 12:47 PM

Hi Érico,

Thanks for sharing this interesting subtle CA connection issue. 

Another thing worthy of checking: If you use systemd, make both IOC services start after network-online.target, to avoid your IOCs start before your NICs/IPs are fully functional.

I always think CA clients will continuously attempt to connect until timeout (EPICS_CA_MAX_SEARCH_PERIOD, 300-second). I am surprised it never connects in your asCa case.

Cheers,

Yong

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Érico Nogueira Rolim via Tech-talk <tech-talk at aps.anl.gov>
Date: Tuesday, February 24, 2026 at 1:14 PM

On 2/24/26 07:02, Michael Davidsaver wrote:

On 2/23/26 3:36 PM, Érico Nogueira Rolim via Tech-talk wrote:

Hi, all!

We have had some occasional issues when rebooting servers in our BPM
infrastructure, and I'd like to request guidance on how to best debug it.

We have an IOC for each RF front-end and an IOC for the actual BPM
board, and we have an Access Security rule to block writes to certain
front-end PVs when the fast orbit distortion detection is enabled in the
BPM board.

Sometimes, when these servers are rebooted, the front-end IOC
is unable to connect to the PV from the BPM board IOC

So it never connects at all?  Odd.

Indeed. We rarely need to write to these PVs, so we can go without noticing these issues for months, and it doesn't connect for that whole time.

...
While writing this email, I thought of adding a proxy record in the
front-end DB that connects to the outside PV, and checking the
connection on that. Should it yield different results? Is the connection
path considerably different between DB links and AS links?

This seems an interesting test.  The asCa code [1] creates a separate CA client context to what the dbCa code (aka. CA_LINK) uses [2].  Both should use the same configuration from the environment, and so should yield the same result.

https ://github.com/epics-base/epics-base/blob/3855356d2faef8ccf67ae5d0a368d44ef2527a5c/modules/database/src/ioc/as/asCa.c#L172-L173

https ://github.com/epics-base/epics-base/blob/3855356d2faef8ccf67ae5d0a368d44ef2527a5c/modules/database/src/ioc/db/dbCa.c#L1108-L1109

I will make a patch for this and deploy it when possible, then. I'll report back when I have results!