modbus instruments don't provide the correct connection status using asyn check

[Antonio Moschini via Tech-talk <tech-talk@aps.anl.gov>]

Hi,

In our internal laboratory, some racks are in use.
Inside the racks some components as Temperature Sensors W&T Web Thermometer NTC 57714 and Raritan PX3-5190CXR44 controlled remotely via modbus interface on port 502 have been installed and configured.
Each HW component with ethernet is connected to managed switch Ruckus icx7150 or icx7550. Each switch present on the racks is cross connected to a central node named "controller".
From EPICS point of view the following configuration has been installed:
- EPICS base R7.0.8
- asyn commit a94a78987a99b1c36bbd641a67341069dc5d627e (HEAD in our repository)
- modbus commit a59270f32d2846fe1a2844c60c796b3dbd3e3254 (HEAD in our repository); other extensions and modules have been installed and compiled.
Now the problem seen:
Each modbus HW components (Thermometer and Raritan) have its own IOC that set/get info from HW and, by means of an asyn record, checks the status of the connection. That record is the following one:

-- Thermometer
record(asyn, TEMP:$(_RX):STATUS:CHECK_CONN) {
    field(DESC, "Check device connection")
    field(DTYP, "asynRecordDevice")
    field(PORT, "$(_ASYN_PORT)")
    field(ADDR, "0")
    field(ASG,  "READ_ONLY")
}
-- Raritan
record(asyn, PWR:$(_RXX):STATUS:CHECK_CONN) {
    field(DESC, "Check device connection")
    field(DTYP, "asynRecordDevice")
    field(PORT, "$(_ASYN_PORT)")
    field(ADDR, "0")
    field(ASG,  "READ_ONLY")
}

Both the previous records are then used, as input, to further CALC record to provide the final status of the component.
st.cmd file contains the proper initialization procedure as listed hereafter

drvAsynIPPortConfigure("$(ASYN_PORT)", "$(IP_ADDRESS):502", 0, 0, 0)
modbusInterposeConfig("$(ASYN_PORT)", 0, 0, 0)

and each db is loaded.

Hereafter the use-cases we have used to check the connection status; all the use cases have been performed using the thermometer:

-- 1st Use case --> correct result
When the IOC is started with the instrument switched on and connected to the proper ethernet port, its status is correctly detected ("nominal"), no error is detected by the driver; using asynReport command the connection status is:

asynReport 1 rack_temp_sensor_1
rack_temp_sensor_1 multiDevice:No canBlock:Yes autoConnect:Yes
    enabled:Yes connected:Yes numberConnects 1
    nDevices 0 nQueued 0 blocked:No
    asynManagerLock:No synchronousLock:No
    exceptionActive:No exceptionUsers 2 exceptionNotifys 0
    traceMask:0x1 traceIOMask:0x0 traceInfoMask:0x1
    Port xxx.xxx.xxx.xxx:502: Connected

-- 2nd Use case --> correct result
When the IOC is started with the instrument switched off or disconnected from the proper ethernet port, its status is correctly detected ("off"), errors from the driver are shown on the iocShell and asynReport provides the disconnection status as well.
epics> 2025/09/11 12:14:03.144 drvModbusAsyn::doModbusIO port temp_r error calling writeRead, error=asynManager::queueLockPort queueRequest failed: port rack_temp_sensor_1 not connected, nwrite=0/6, nread=0
2025/09/11 12:14:03.145 drvModbusAsyn::doModbusIO port alarm_state_r error calling writeRead, error=asynManager::queueLockPort queueRequest failed: port rack_temp_sensor_1 not connected, nwrite=0/6, nread=0
2025/09/11 12:14:03.146 drvModbusAsyn::doModbusIO port device_SN error calling writeRead, error=asynManager::queueLockPort queueRequest failed: port rack_temp_sensor_1 not connected, nwrite=0/6, nread=0
2025/09/11 12:14:07.787 rack_temp_sensor_1 -1 autoConnect could not connect: Can't connect to xxx.xxx.xxx.xxx:502: No route to host

epics> asynReport 1 rack_temp_sensor_1
rack_temp_sensor_1 multiDevice:No canBlock:Yes autoConnect:Yes
    enabled:Yes connected:No numberConnects 0
    nDevices 0 nQueued 0 blocked:No
    asynManagerLock:No synchronousLock:No
    exceptionActive:No exceptionUsers 2 exceptionNotifys 0
    traceMask:0x1 traceIOMask:0x0 traceInfoMask:0x1
    Port xxx.xxx.xxx.xxx:502: Disconnected

-- 3rd Use case  --> wrong result
a) Setup 1st Use-case to have the IOC reporting "nominal" as status;

epics> asynReport 1 rack_temp_sensor_1
rack_temp_sensor_1 multiDevice:No canBlock:Yes autoConnect:Yes
    enabled:Yes connected:Yes numberConnects 1
    nDevices 0 nQueued 0 blocked:No
    asynManagerLock:No synchronousLock:No
    exceptionActive:No exceptionUsers 2 exceptionNotifys 0
    traceMask:0x1 traceIOMask:0x0 traceInfoMask:0x1
    Port xxx.xxx.xxx.xxx:502: Connected

b) connect to the Ruckus via ssh and disable the ethernet port;
epics> 2025/09/11 12:25:21.747 drvModbusAsyn::doModbusIO port alarm_state_r error calling writeRead, error=xxx.xxx.xxx.xxx:502 timeout: Resource temporarily unavailable, nwrite=6/6, nread=0
2025/09/11 12:25:21.869 drvModbusAsyn::doModbusIO port temp_r error calling writeRead, error=asynManager::queueLockPort queueRequest timed out, nwrite=0/6, nread=0
2025/09/11 12:25:23.751 drvModbusAsyn::doModbusIO port device_SN error calling writeRead, error=xxx.xxx.xxx.xxx:502 timeout: Resource temporarily unavailable, nwrite=6/6, nread=0

epics> asynReport 1 rack_temp_sensor_1
rack_temp_sensor_1 multiDevice:No canBlock:Yes autoConnect:Yes
    enabled:Yes connected:Yes numberConnects 1
    nDevices 0 nQueued 0 blocked:No
    asynManagerLock:No synchronousLock:Yes
    exceptionActive:No exceptionUsers 2 exceptionNotifys 0
    traceMask:0x1 traceIOMask:0x0 traceInfoMask:0x1
    Port xxx.xxx.xxx.xxx:502: Connected

The result is not the expected one. The expected result was connected:No

-- 4th Use case  --> wrong result
a) Setup 2nd Use-case to have the IOC reporting "off" as status;
epics> 2025/09/11 12:33:56.454 drvModbusAsyn::doModbusIO port alarm_state_r error calling writeRead, error=asynManager::queueLockPort queueRequest failed: port rack_temp_sensor_1 not connected, nwrite=0/6, nread=0
2025/09/11 12:33:56.458 drvModbusAsyn::doModbusIO port temp_r error calling writeRead, error=asynManager::queueLockPort queueRequest failed: port rack_temp_sensor_1 not connected, nwrite=0/6, nread=0
2025/09/11 12:33:56.486 drvModbusAsyn::doModbusIO port device_SN error calling writeRead, error=asynManager::queueLockPort queueRequest failed: port rack_temp_sensor_1 not connected, nwrite=0/6, nread=0
2025/09/11 12:34:01.103 rack_temp_sensor_1 -1 autoConnect could not connect: Can't connect to xxx.xxx.xxx.xxx:502: No route to host

epics> asynReport 1 rack_temp_sensor_1
rack_temp_sensor_1 multiDevice:No canBlock:Yes autoConnect:Yes
    enabled:Yes connected:No numberConnects 0
    nDevices 0 nQueued 0 blocked:No
    asynManagerLock:No synchronousLock:No
    exceptionActive:No exceptionUsers 2 exceptionNotifys 0
    traceMask:0x1 traceIOMask:0x0 traceInfoMask:0x1
    Port xxx.xxx.xxx:502: Disconnected
b) ) connect to the Ruckus via ssh and enable the ethernet port;
epics> 2025/09/11 12:38:11.738 rack_temp_sensor_1 -1 port is now connected
2025/09/11 12:38:11.903 drvModbusAsyn::doModbusIO port alarm_state_r writeRead status back to normal having had 128 errors, nwrite=6/6, nread=6
2025/09/11 12:38:11.916 drvModbusAsyn::doModbusIO port temp_r writeRead status back to normal having had 128 errors, nwrite=6/6, nread=6
2025/09/11 12:38:12.147 drvModbusAsyn::doModbusIO port device_SN writeRead status back to normal having had 128 errors, nwrite=6/6, nread=10

epics>
epics> asynReport 1 rack_temp_sensor_1
rack_temp_sensor_1 multiDevice:No canBlock:Yes autoConnect:Yes
    enabled:Yes connected:Yes numberConnects 1
    nDevices 0 nQueued 0 blocked:No
    asynManagerLock:No synchronousLock:No
    exceptionActive:No exceptionUsers 2 exceptionNotifys 0
    traceMask:0x1 traceIOMask:0x0 traceInfoMask:0x1
    Port xxx.xxx.xxx.xxx:502: Connected

c)  connect to the Ruckus via ssh and disable the ethernet port;
epics> 2025/09/11 12:39:56.465 drvModbusAsyn::doModbusIO port alarm_state_r error calling writeRead, error=10.100.71.80:502 timeout: Resource temporarily unavailable, nwrite=6/6, nread=0
2025/09/11 12:39:56.862 drvModbusAsyn::doModbusIO port device_SN error calling writeRead, error=asynManager::queueLockPort queueRequest timed out, nwrite=0/6, nread=0
2025/09/11 12:39:58.466 drvModbusAsyn::doModbusIO port temp_r error calling writeRead, error=10.100.71.80:502 timeout: Resource temporarily unavailable, nwrite=6/6, nread=0

epics> asynReport 1 rack_temp_sensor_1
rack_temp_sensor_1 multiDevice:No canBlock:Yes autoConnect:Yes
    enabled:Yes connected:Yes numberConnects 1
    nDevices 0 nQueued 1 blocked:No
    asynManagerLock:No synchronousLock:Yes
    exceptionActive:No exceptionUsers 2 exceptionNotifys 0
    traceMask:0x1 traceIOMask:0x0 traceInfoMask:0x1
    Port xxx.xxx.xxx.xxx:502: Connected

The result is not the expected one. The expected result was connected:No

No problem is seen using an unmanaged switch.

P.S. setting "IGMP snooping" on the switch does not produce any improvement

Does anyone get the same result ? If not how to solve or overcome this misbehavior ?

Thanks in advance

Antonio Moschini

Senior SW Engineer
Telescopes Business Unit
Viale De Gasperi, 2, 20151 Milano (MI) – ITALY


________________________________

<< Le informazioni contenute in questo messaggio di posta elettronica e relativi allegati sono riservate e confidenziali e ne è vietata la diffusione in qualunque modo eseguita. Qualora Lei non fosse la persona a cui il presente messaggio è destinato, La invitiamo ad eliminarlo e a darcene gentile comunicazione e a non copiare per diffondere o utilizzare i suoi contenuti per qualsiasi motivo. Quanto precede ai fini del rispetto del GDPR di cui al Reg. (EU) 679/2016, sulla tutela dei dati personali. Ringraziamo per la collaborazione. Per qualsiasi necessità La preghiamo di contattare il numero +3902380481 >>

<< This e-mail is confidential and may well also be legally privileged. If you have received it in error, you are on notice of its status. Please notify us immediately by reply e-mail and then delete this message from your system. Please do not copy it or use it for any purposes, or disclose its contents to any other person: to do so could be a breach of confidence. The above is in compliance to European Regulation on Data Protection Reg. (EU) 679/2016. Thank you for your co-operation. Please contact our IT Helpdesk on +3902380481 if you need assistance >>