Experimental Physics and
Industrial Control System

Hm,
not a real answer in any way...
CLOSE_WAIT on the IOC means that the remote end (the PLC)
has closed the connection.
And now the IOC should close the connection,
and then estabish a new one.

I have something like this in my code:

  status = pasynOctetSyncIO->read(pasynUser,
                                  indata, part_1_len,
                                  DEFAULT_CONTROLLER_TIMEOUT,
                                  &nread1, &eomReason);
  if (eomReason & ASYN_EOM_END) {;
    status = pasynCommon->disconnect(pasynInterface->drvPvt,
                                       pasynUser);
   }
=====================
I don't know what the modbus driver does, on top of asyn.

But please note that the line
   asynSetOption("$(PORT)", 0, "disconnectOnReadTimeout", "Y")
is useful in general.
But not in this very situation, since we don't have a timeout.
The connection is closed from remote.

Having said that, what happens within this 10 minutes ?
Can you run a Wireshark, filtering on port 502 ?

Or may be asyntrace, or better both  ?

And/or wait for the experts from US get online ?

HTH
/Torsten


On 2023-11-17 13:49, Paul Richardson - STFC UKRI via Tech-talk wrote:

Hi,

I’m having problems with an epics IOC re-establishing network connections to a PLIZ PLC, after a network cable disconnection, then reconnection 10 minutes or so later. The IOC is running on a Ubuntu 22.04 PC and communicating to the PLC using modbus interpose layer and asyn over TCP/IP.

After re-connection, the Asyn.CNCT PV repeatedly reports Connect then Disconnect. This can go on for approx. 20 mins before finally the problem resolves itself and the connection remains Connected. During this period of instability, whilst I can happily ping the PLC, the IOC is unable to read any of its values through modbus. This would make sense if the PLC has closed its socket.

When the issue occurs, I’ve examined the network socket for port 502 (the PLC port) using netstat -at | grep 502. This reports the socket is an a CLOSE_WAIT state. I understand this indicates the PLC has closed its connection, but the socket is still open at the IOC end. I’m unsure how to get asyn to recognise this situation and resolve it, maybe by closing the socket and establishing a new one?

I had hoped using asynSetOption("$(PORT)", 0, "disconnectOnReadTimeout", "Y"), would resolve the issue but it didn’t.

Other options I’ve tried include:

Adjusting the linuc tcp_keepalive_time from 7200 s (120 mins) to 20 s, using /sbin/sysctl -e net.ipv4.tcp_keepalive_time = 20, but it didn’t help

disabling/enabling asyn disconnectOnReadTimeout() and moving its position in st.cmd

varying modbusInterposeConfig() read write timeout

enabling/disabling noProcessEos in drvAsynIPPortConfigure()

but the problem remains.

Below is a section of my st.cmd

epicsEnvSet("DEVICE_IP", "109.0.201.1")

epicsEnvSet("DEVICE_PORT", "502")

epicsEnvSet("PORT", "PILZ")

drvAsynIPPortConfigure("$(PORT)","$(DEVICE_IP):$(DEVICE_PORT)",0,0,1) #IP address of simulated plc

asynSetOption("$(PORT)", 0, "disconnectOnReadTimeout", "Y")

modbusInterposeConfig("$(PORT)",0,0,0)

drvModbusAsynConfigure("BI", "$(PORT)", 0, 3, 107, 56, 0, 100, "pilz")

## Load record instances

dbLoadRecords("$(ASYN)/db/asynRecord.db", "P=$(SHORT_NM), R=:Asyn, PORT=$(PORT), ADDR=0, IMAX=20, OMAX=20")

I think the cause of the problem is the PLC closes the socket, but asyn in the IOC is leaving it open. How do I get asyn to recognise the CLOSE_WAIT status of the socket, close it then establishes a new one? I’m wondering if this may be an issue with modbus interposed in the asyn?

Any help or advice would be much appreciate, thanks.

Kind regards

Paul Richardson

References:

Network reconnection issue using modbus over asyn Paul Richardson - STFC UKRI via Tech-talk

Navigate by Date:

Prev: Network reconnection issue using modbus over asyn Paul Richardson - STFC UKRI via Tech-talk

Next: Re: Network reconnection issue using modbus over asyn Mark Rivers via Tech-talk

Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  <2023>  2024 

Navigate by Thread:

Prev: Network reconnection issue using modbus over asyn Paul Richardson - STFC UKRI via Tech-talk

Next: Re: Network reconnection issue using modbus over asyn Mark Rivers via Tech-talk

Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  <2023>  2024