RE: Network reconnection issue using modbus over asyn

[Paul Richardson - STFC UKRI via Tech-talk <tech-talk at aps.anl.gov>]

Hi Mark,

Yes, this issue is caused by disconnecting the network cable at the PLC end, so the connection is not closed cleanly. Asyn.CNCT reports Disconnect, just once, the moment the cable is removed.

If I leave it disconnected for roughly up to a minute, the connection is always re-established and asyn.CNCT reports Connect, just once, as expected.

The problem only seems to occur when I leave the cable disconnected for longer, usually 5 mins or more. Sometimes upon re-connection it works Ok, but more often than not on re-connection the asyn.CNCT status repeatedly flips between Connect and Disconnect. Looking at the socket (using netstat -at) for the PLC port it reports CLOSE_WAIT, interspersed with ESTABLISHED.

I’m currently using asyn4-38 and modbus-R2-4. One of the first things I tried was using more up to date versions, so I downloaded asyn-R4-43 and modbus-R2-11, but the problem still occurred.

 

Regards

Paul

 

 

From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Friday, November 17, 2023 1:32 PM
To: Richardson, Paul (STFC,RAL,CLF) <paul.richardson at stfc.ac.uk>
Cc: tech-talk at aps.anl.gov
Subject: Re: Network reconnection issue using modbus over asyn

 

If I understand correctly this happens when you disconnect the cable. So the PLC has not cleanly closed the connection, correct? 

 

Do you see the problem when you disconnect the cable for a shorter amount of time, say 15 seconds?

 

What you should see is a single message for each asyn port that the connection is down, and then a single message when it is back up again.

 

I will test early next week to verify that is what I see.

What versions of asyn and Modbus are you using?

Mark

 

Sent from my iPhone

On Nov 17, 2023, at 6:50 AM, Paul Richardson - STFC UKRI via Tech-talk <tech-talk at aps.anl.gov> wrote:



Hi,

I’m having problems with an epics IOC re-establishing network connections to a PLIZ PLC, after a network cable disconnection, then reconnection 10 minutes or so later. The IOC is running on a Ubuntu 22.04 PC and communicating to the PLC using modbus interpose layer and asyn over TCP/IP.

After re-connection, the Asyn.CNCT PV repeatedly reports Connect then Disconnect. This can go on for approx. 20 mins before finally the problem resolves itself and the connection remains Connected. During this period of instability, whilst I can happily ping the PLC, the IOC is unable to read any of its values through modbus. This would make sense if the PLC has closed its socket.

When the issue occurs, I’ve examined the network socket for port 502 (the PLC port) using netstat -at | grep 502. This reports the socket is an a CLOSE_WAIT state. I understand this indicates the PLC has closed its connection, but the socket is still open at the IOC end. I’m unsure how to get asyn to recognise this situation and resolve it, maybe by closing the socket and establishing a new one?

I had hoped using asynSetOption("$(PORT)", 0, "disconnectOnReadTimeout", "Y"), would resolve the issue but it didn’t.

Other options I’ve tried include:

Adjusting the linuc tcp_keepalive_time from 7200 s (120 mins) to 20 s, using /sbin/sysctl -e net.ipv4.tcp_keepalive_time = 20, but it didn’t help

disabling/enabling asyn disconnectOnReadTimeout() and moving its position in st.cmd

varying modbusInterposeConfig() read write timeout

enabling/disabling noProcessEos in drvAsynIPPortConfigure()

but the problem remains.

 

Below is a section of my st.cmd

epicsEnvSet("DEVICE_IP", "109.0.201.1")

epicsEnvSet("DEVICE_PORT", "502")

epicsEnvSet("PORT", "PILZ")

 

drvAsynIPPortConfigure("$(PORT)","$(DEVICE_IP):$(DEVICE_PORT)",0,0,1) #IP address of simulated plc

asynSetOption("$(PORT)", 0, "disconnectOnReadTimeout", "Y")

modbusInterposeConfig("$(PORT)",0,0,0)

drvModbusAsynConfigure("BI", "$(PORT)", 0, 3, 107, 56, 0, 100, "pilz")

 

## Load record instances

dbLoadRecords("$(ASYN)/db/asynRecord.db", "P=$(SHORT_NM), R=:Asyn, PORT=$(PORT), ADDR=0, IMAX=20, OMAX=20")

 

I think the cause of the problem is the PLC closes the socket, but asyn in the IOC is leaving it open. How do I get asyn to recognise the CLOSE_WAIT status of the socket, close it then establishes a new one? I’m wondering if this may be an issue with modbus interposed in the asyn?

Any help or advice would be much appreciate, thanks.

 

Kind regards

Paul Richardson