Re: CAS: Client accept error: Too many open files

[Ralph Lange via Tech-talk <tech-talk at aps.anl.gov>]

Hi Yong,

On Fri, 30 Jan 2026 at 17:40, Hu, Yong via Tech-talk <tech-talk at aps.anl.gov> wrote:

Hi all,

I recently encountered an issue with a misbehaving Linux softIOC that caused Phoebus / CS-Studio to appear broken when opening certain OPIs.

Symptoms: The IOC was periodically logging the following error: CAS: Client accept error: Too many open files.

That sounds like a Linux host with too-low settings for the limits on open file descriptors per process and/or per user.

Whenever I opened an OPI containing PVs from that IOC, all CSS OPI screens turned pink, and all PVs appeared disconnected. However:

• caget and camonitor returned correct PV values.

• The CSS "Probe" tool also displayed PV values normally.

In such low-resource situations, an IOC will start arbitrarily rejecting new client connections (printing the error message you were showing) and accepting other new connections, while existing connections usually continue to work.

 

Troubleshooting: I traced the problematic IOC (xxx.x.xxx.xxx:38841) based on terminal output during the failure. The logs included the following error:

SEVERE [Thread 55] com.cosylab.epics.caj.CAJContext (getTransport) - Failed to create transport for: /xxx.x.xxx.xxx:38841

com.cosylab.epics.caj.impl.ConnectionException: Failed to connect to '/xxx.x.xxx.xxx:38841'.

    at com.cosylab.epics.caj.impl.CAConnector.connect(CAConnector.java:138)

    at com.cosylab.epics.caj.CAJContext.getTransport(CAJContext.java:1559)

That's the client side when the IOC rejects a new connection. Looks consistent.

Rebooting the IOC resolved the issue — at least temporarily.

Also consistent with low resource limits on the IOC host.

Use netstat, lsof and friends on the IOC hosts to check the file descriptor/TCP connection usage situation. The iocStats also shows FD usage.

Looking for Insight: A Google search didn’t yield much, so I asked ChatGPT, which suggested that a shared CA context might lead to a shared failure. That seems plausible, but I’d appreciate input from anyone familiar with Phoebus or CSS internals. Why would OPI screens appear completely broken while the "Probe" tool continues to retrieve PV values correctly?

My guess: inside Phoebus, Probe is a different client than the Display Builder runtime. One gets their connection accepted, the other rejected.

Cheers,
~Ralph