IOC connection problem

[Mark Rivers via Tech-talk <tech-talk at aps.anl.gov>]

Folks,

 

I have a number of Linux machines that each run multiple IOCs.  The IOC and clients are all on a single public subnet, 164.54.160.*.  There is also a private subnet that some Linux machines and many devices are connected to, 10.54.160.*.

 

On the Linux machines I have set the following so that CA searches are only on the public network.  This prevents the warnings about multiple IOCs have the same PV.

EPICS_CA_AUTO_ADDR_LIST=NO

EPICS_CA_ADDR_LIST=164.54.160.255

 

This is working fine for all Linux machines except one, called 13idd.  When I run multiple IOCs on 13idd I observe the following:

• I can reach all IOCs from clients running locally on 13idd
• From other machines I can reach the PVs on the first IOC started on 13idd.
• From other machines when trying to reach PVs on other IOCs on 13idd I get the following error:

 

[epics@13bmd ~]$ /corvette/usr/local/epics/base/bin/linux-x86_64/caget 13IDD:m1

CAC: Unable to connect because "No route to host"

CA.Client.Exception...............................................

    Warning: "Virtual circuit disconnect"

    Context: "13idd.cars.aps.anl.gov:46799"

    Source File: ../cac.cpp line 1237

    Current Time: Wed Jun 19 2024 17:37:31.591554930

..................................................................

Channel connect timed out: '13IDD:m1' not found.

 

The message about “No route to host” happens immediately.

 

That machine is not configured differently from the others, as far as I have been able to tell.

 

This is the configuration of the 2 NICs on 13idd:

 

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 10.54.160.173  netmask 255.255.255.0  broadcast 10.54.160.255

        inet6 fe80::42a6:b7ff:feb4:69ac  prefixlen 64  scopeid 0x20<link>

        ether 40:a6:b7:b4:69:ac  txqueuelen 1000  (Ethernet)

        RX packets 701064543  bytes 258403234658 (240.6 GiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 82557039  bytes 7943801957 (7.3 GiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

 

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 164.54.160.193  netmask 255.255.255.0  broadcast 164.54.160.255

        inet6 fe80::42a6:b7ff:feb4:69ad  prefixlen 64  scopeid 0x20<link>

        ether 40:a6:b7:b4:69:ad  txqueuelen 1000  (Ethernet)

        RX packets 28919766  bytes 3450612626 (3.2 GiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 1894585  bytes 1421681130 (1.3 GiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

 

 

This is the output of epicsPrtEnvParams for the second IOC started on 13idd:

 

epics> epicsPrtEnvParams

EPICS_CA_ADDR_LIST: 164.54.160.255

EPICS_CA_CONN_TMO: 30.0

EPICS_CA_AUTO_ADDR_LIST: NO

EPICS_CA_REPEATER_PORT: 5065

EPICS_CA_SERVER_PORT: 5064

EPICS_CA_MAX_ARRAY_BYTES: 16384

EPICS_CA_AUTO_ARRAY_BYTES: YES

EPICS_CA_MAX_SEARCH_PERIOD: 300.0

EPICS_CA_NAME_SERVERS is undefined

EPICS_CA_MCAST_TTL: 1

EPICS_CAS_INTF_ADDR_LIST is undefined

EPICS_CAS_IGNORE_ADDR_LIST is undefined

EPICS_CAS_AUTO_BEACON_ADDR_LIST is undefined

EPICS_CAS_BEACON_ADDR_LIST is undefined

EPICS_CAS_SERVER_PORT is undefined

EPICS_CA_BEACON_PERIOD: 15.0

EPICS_CAS_BEACON_PERIOD is undefined

EPICS_CAS_BEACON_PORT is undefined

EPICS_BUILD_COMPILER_CLASS: gcc

EPICS_BUILD_OS_CLASS: Linux

EPICS_BUILD_TARGET_ARCH: linux-x86_64

EPICS_TZ: CST6CDT,M3.2.0/2,M11.1.0/2

EPICS_TS_NTP_INET is undefined

EPICS_IOC_IGNORE_SERVERS is undefined

EPICS_IOC_LOG_PORT: 7004

EPICS_IOC_LOG_INET is undefined

EPICS_IOC_LOG_FILE_LIMIT: 1000000

EPICS_IOC_LOG_FILE_NAME is undefined

EPICS_IOC_LOG_FILE_COMMAND is undefined

IOCSH_PS1: epics>

IOCSH_HISTSIZE: 50

IOCSH_HISTEDIT_DISABLE is undefined

 

Any idea what could cause this problem?

 

Thanks,

Mark