EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: RE: Connection Timeout Using Asyn and StreamDevice with Ethernet Device
From: Mark Rivers via Tech-talk <[email protected]>
To: 'Kyle Hesse' <[email protected]>
Cc: "[email protected]" <[email protected]>
Date: Thu, 7 Nov 2019 20:57:50 +0000

This is the tough kind of problem.  You may need to run wireshark for a few days capturing all traffic to/from the power meter.

 

Mark

 

 

 

From: Kyle Hesse <[email protected]>
Sent: Thursday, November 7, 2019 2:40 PM
To: Mark Rivers <[email protected]>
Cc: [email protected]
Subject: Re: Connection Timeout Using Asyn and StreamDevice with Ethernet Device

 

Hello Mark,

It is the former, I left the connection open for 10 days and made requests to the meter every five seconds. I have posted the script below.

Yes, a restart of the IOC does not reconnect. I have to cycle power. AutoConnect does try its best though.

2019/11/07 09:56:19.197 Can't connect to 129.57.230.28:23: No route to host ETH1 -1 autoConnect could not connect

2019/11/07 09:56:42.193 Can't connect to 129.57.230.28:23: No route to host ETH1 -1 autoConnect could not connect

2019/11/07 09:57:05.189 Can't connect to 129.57.230.28:23: No route to host ETH1 -1 autoConnect could not connect

2019/11/07 09:57:28.186 Can't connect to 129.57.230.28:23: No route to host ETH1 -1 autoConnect could not connect

2019/11/07 09:57:51.182 Can't connect to 129.57.230.28:23: No route to host ETH1 -1 autoConnect could not connect

2019/11/07 09:58:14.178 Can't connect to 129.57.230.28:23: No route to host ETH1 -1 autoConnect could not connect

2019/11/07 09:58:37.174 Can't connect to 129.57.230.28:23: No route to host ETH1 -1 autoConnect could not connect

Script:

#!/usr/csite/pubtools/bin/expect -f

set IP "129.57.230.28"

spawn telnet $IP

expect "Escape character is '^]'."

 

for {set i 0} {$i < 2000000000} {incr i} {

set cmd ":POWER?\r"

 send "$cmd"

 expect "${cmd}\n\r\n*\r" {}

 sleep 5

}

 


From: Mark Rivers <[email protected]>
Sent: Thursday, November 7, 2019 3:16 PM
To: Kyle Hesse <
[email protected]>
Cc:
[email protected] <[email protected]>
Subject: [EXTERNAL] RE: Connection Timeout Using Asyn and StreamDevice with Ethernet Device

 

Hi Kyle,

 

Ø  As a test, I decided to remove EPICS from the equation and run a Tcl/tk expect script that opens the telnet connection and sends a request every five seconds. This ran for ten days without issue.

 

Does your Tcl/tk script leave the telnet connection open for 10 days, or does it close and re-open the connection every 5 seconds?  If it is the latter then it is not a good comparison to the EPICS IOC.

 

Restarting the IOC will not allow it to reconnect, you have to power-cycle the meter?

 

Mark

 

From: Tech-talk <[email protected]> On Behalf Of Kyle Hesse via Tech-talk
Sent: Thursday, November 7, 2019 10:16 AM
To:
[email protected]
Subject: Connection Timeout Using Asyn and StreamDevice with Ethernet Device

 

Hello,

Thanks in advance for any help you may provide.

Problem:

 

Over the past few weeks we have noticed that our telnet connection on our Mini-Circuits power meter will go unresponsive when using asynDriver after several days of successful use. We can still ping it, the HTML interface over ethernet is responsive, but the telnet port can no longer connect without a device power cycle. As a test, I decided to remove EPICS from the equation and run a Tcl/tk expect script that opens the telnet connection and sends a request every five seconds. This ran for ten days without issue. Any other diagnostic recommendations or theories as to what might be causing this behavior?      

 

Setup:

 

We use a Mini-Circuits PWR-8P-RC Power Meter which is network configured. The control protocol is SCPI ASCII. We communicate through Telnet (port 23) over a TCP/IP connection. ASYN (4-31) with StreamDevice (2-7) is used for the EPICS (3.14.12.3) implementation.

 

PV Example:

 

record(ai,"$(P)$(R)POWER_PEAK_RAW") {

field(SCAN,"5 second")

field(DTYP,"stream")

field(FLNK,"$(P)$(R)POWER_PEAK")

field(INP,"@PWR-xP.proto getPwrPeak $(PORT)")

field(PREC,"3")

field(EGU,"dBm")

}

 

Protocol Example:

 

OutTerminator = CR LF;

InTerminator = LF CR;

writetimeout = 300;

readtimeout = 1000;

replytimeout = 2000;

ExtraInput = Ignore;

 

getPwrPeak {

    out ":POWER?";

    in "%f dBm %*f dBm";

}

 

Port Configuration:

 

drvAsynIPPortConfigure("ETH1",129.57.230.28:23)

 

ERROR Messages:

 

scan5 HOM:PWR1:POWER_PEAK_RAW lockRequest: pasynManager->queueRequest: port ETH1 not connected

 

"Can't connect to 129.57.230.28:23: Connection timed out ETH1 -1 autoConnect could not connect"

 

Diagnostics:

 

This was the response after the device had disconnected and I restarted the softIoc.

 

epics>asynReport(1,ETH1)

    ETH1 multiDevice:No canBlock:Yes autoConnect:Yes

    enabled:Yes connected:No numberConnects 0

    nDevices 0 nQueued 1 blocked:No

    asynManagerLock:No synchronousLock:Yes

    exceptionActive:No exceptionUsers 1 exceptionNotifys 0

    traceMask:0x1 traceIOMask:0x0 traceInfoMask:0x1

    Port 129.57.230.28:23: Disconnected

 

This was a response after the power meter was power cycled.

 

epics> asynReport(1,ETH1)

ETH1 multiDevice:No canBlock:Yes autoConnect:Yes

    enabled:Yes connected:Yes numberConnects 1

    nDevices 0 nQueued 5 blocked:No

    asynManagerLock:No synchronousLock:Yes

    exceptionActive:No exceptionUsers 1 exceptionNotifys 0

    traceMask:0x1 traceIOMask:0x0 traceInfoMask:0x1

    Port 129.57.230.28:23: Connected

 

 

 


References:
Connection Timeout Using Asyn and StreamDevice with Ethernet Device Kyle Hesse via Tech-talk
RE: Connection Timeout Using Asyn and StreamDevice with Ethernet Device Mark Rivers via Tech-talk
Re: Connection Timeout Using Asyn and StreamDevice with Ethernet Device Kyle Hesse via Tech-talk

Navigate by Date:
Prev: Re: Connection Timeout Using Asyn and StreamDevice with Ethernet Device Kyle Hesse via Tech-talk
Next: Re: Connection Timeout Using Asyn and StreamDevice with Ethernet Device Peterson, Kevin M. via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Re: Connection Timeout Using Asyn and StreamDevice with Ethernet Device Kyle Hesse via Tech-talk
Next: Re: Connection Timeout Using Asyn and StreamDevice with Ethernet Device Peterson, Kevin M. via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024 
ANJ, 07 Nov 2019 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·