Gateway error

["Leicester, PJ \(Pete\)" <peter.leicester@diamond.ac.uk>]

Title: Message

 

We have been running the new gateway 2.0 on epics 3.14.8.2 for a month or more with good results. It has been stable even with large transfers.

 

However last evening a problem occurred with the gateway and it had to be restarted. Unfortunately the only information I have is the following gateway log.

Can anyone hazard a guess as to what was going on ?

 

We are running the gateway on Redhat Enterprise Linux 4.

Also we are running the alarm handler through the gateway too which I understand from previous Techtalk emails is a bad thing (DBE_VALUE v DBE_ALARM events subscription conflict) and something we have to address. Could this have caused the problem?

 

Pete Leicester

Diamond Light Source

 

------------------------------------------

 

Feb 27 09:01:02 PV Gateway Version 2.0.0.0 [Jan  9 2006 11:30:43]
EPICS 3.14.8.2 PID=2598 ServerPID=2597
EPICS_CA_ADDR_LIST=172.23.255.255
EPICS_CA_AUTO_ADDR_LIST=NO
EPICS_CA_SERVER_PORT=5064
EPICS_CA_MAX_ARRAY_BYTES=3000000
EPICS_CAS_INTF_ADDR_LIST=172.23.194.129
EPICS_CAS_SERVER_PORT=5064
EPICS_CAS_IGNORE_ADDR_LIST=172.23.194.129
Running as user gate on host serv0013.cs.diamond.ac.uk
Statistics PV prefix is CtoP:serv0013
Feb 28 14:42:43 Warning: Virtual circuit unresponsive 172.23.248.87:5064
Unexpected problem with CA circuit to server "172.23.248.87:5064" was "No route to host" - disconnecting

 

Feb 28 14:58:03 !!! Errlog message received (message is above)
Feb 28 14:58:03 Warning: Virtual circuit disconnect 172.23.248.87:5064
Feb 28 15:36:41 Warning: Virtual circuit unresponsive 172.23.248.60:5064
CAS: TCP socket send to "172.23.198.8:32805" failed because "No route to host"

 

Feb 28 15:50:15 !!! Errlog message received (message is above)
Unexpected problem with CA circuit to server "172.23.248.60:5064" was "No route to host" - disconnecting

 

Feb 28 15:52:01 !!! Errlog message received (message is above)
Feb 28 15:52:01 Warning: Virtual circuit disconnect 172.23.248.60:5064
Feb 28 16:15:46 Warning: Virtual circuit unresponsive 172.23.248.109:5064
CAS: TCP socket send to "172.23.198.8:32813" failed because "No route to host"

 

Feb 28 16:16:40 !!! Errlog message received (message is above)
Unexpected problem with CA circuit to server "172.23.248.109:5064" was "No route to host" - disconnecting

 

Feb 28 16:31:07 !!! Errlog message received (message is above)
Feb 28 16:31:07 Warning: Virtual circuit disconnect 172.23.248.109:5064
Mar 01 10:41:06 Warning: Virtual circuit unresponsive 172.23.248.25:5064
Mar 01 10:41:07 Warning: Virtual circuit unresponsive 172.23.248.27:5064
Mar 01 10:41:09 Warning: Virtual circuit unresponsive 172.23.248.83:5064
Mar 01 10:41:11 Warning: Virtual circuit unresponsive 172.23.248.195:5064
Mar 01 10:41:15 Warning: Virtual circuit unresponsive 172.23.248.164:5064
Mar 01 10:41:15 Warning: Virtual circuit unresponsive 172.23.248.24:5064
Mar 01 10:41:16 Warning: Virtual circuit unresponsive 172.23.248.144:5064
Unexpected problem with CA circuit to server "172.23.248.25:5064" was "No route to host" - disconnecting

 

Mar 01 10:56:27 !!! Errlog message received (message is above)
Mar 01 10:56:27 Warning: Virtual circuit disconnect 172.23.248.25:5064
Unexpected problem with CA circuit to server "172.23.248.83:5064" was "No route to host" - disconnecting

 

Mar 01 10:56:29 !!! Errlog message received (message is above)
Mar 01 10:56:29 Warning: Virtual circuit disconnect 172.23.248.83:5064
CAS: client 172.23.198.11:32812 disconnected because "No route to host"

 

Mar 01 10:56:32 !!! Errlog message received (message is above)
Unexpected problem with CA circuit to server "172.23.248.195:5064" was "No route to host" - disconnecting

 

Mar 01 10:56:32 !!! Errlog message received (message is above)
Mar 01 10:56:32 Warning: Virtual circuit disconnect 172.23.248.195:5064
Unexpected problem with CA circuit to server "172.23.248.164:5064" was "No route to host" - disconnecting

 

Mar 01 10:56:36 !!! Errlog message received (message is above)
Mar 01 10:56:36 Warning: Virtual circuit disconnect 172.23.248.164:5064
Unexpected problem with CA circuit to server "172.23.248.24:5064" was "No route to host" - disconnecting

 

Mar 01 10:56:36 !!! Errlog message received (message is above)
Mar 01 10:56:36 Warning: Virtual circuit disconnect 172.23.248.24:5064
Unexpected problem with CA circuit to server "172.23.248.144:5064" was "No route to host" - disconnecting

 

Mar 01 10:56:37 !!! Errlog message received (message is above)
Mar 01 10:56:37 Warning: Virtual circuit disconnect 172.23.248.144:5064
Unexpected problem with CA circuit to server "172.23.248.27:5064" was "No route to host" - disconnecting

 

Mar 01 10:56:44 !!! Errlog message received (message is above)
Mar 01 10:56:44 Warning: Virtual circuit disconnect 172.23.248.27:5064
CAS: client 172.23.200.84:1090 disconnected because "No route to host"

 

Mar 01 10:57:00 !!! Errlog message received (message is above)
Mar 01 16:54:11 Warning: Virtual circuit unresponsive 172.23.248.24:5064
Mar 01 16:54:32 Warning: Virtual circuit unresponsive 172.23.248.27:5064
Mar 01 16:54:40 Warning: Virtual circuit unresponsive 172.23.248.83:5064
Unexpected problem with CA circuit to server "172.23.248.27:5064" was "Connection reset by peer" - disconnecting

 

Mar 01 16:54:54 !!! Errlog message received (message is above)
Mar 01 16:54:54 Warning: Virtual circuit disconnect 172.23.248.27:5064
Mar 01 16:54:57 Warning: Virtual circuit unresponsive 172.23.248.195:5064
Unexpected problem with CA circuit to server "172.23.248.24:5064" was "Connection reset by peer" - disconnecting

 

Mar 01 16:54:58 !!! Errlog message received (message is above)
Mar 01 16:54:58 Warning: Virtual circuit disconnect 172.23.248.24:5064
Unexpected problem with CA circuit to server "172.23.248.195:5064" was "Connection reset by peer" - disconnecting

 

Mar 01 16:55:18 !!! Errlog message received (message is above)
Mar 01 16:55:18 Warning: Virtual circuit disconnect 172.23.248.195:5064
Unexpected problem with CA circuit to server "172.23.248.83:5064" was "Connection reset by peer" - disconnecting

 

Mar 01 16:55:26 !!! Errlog message received (message is above)
Mar 01 16:55:26 Warning: Virtual circuit disconnect 172.23.248.83:5064
Unexpected problem with CA circuit to server "172.23.252.107:5064" was "Connection reset by peer" - disconnecting

 

Mar 01 17:06:40 !!! Errlog message received (message is above)
Mar 01 17:06:40 Warning: Virtual circuit disconnect 172.23.252.107:5064
Unexpected problem with CA circuit to server "172.23.248.195:5064" was "Connection reset by peer" - disconnecting

 

Mar 01 17:18:05 !!! Errlog message received (message is above)
Mar 01 17:18:05 Warning: Virtual circuit disconnect 172.23.248.195:5064
Unexpected problem with CA circuit to server "172.23.248.195:5064" was "Connection reset by peer" - disconnecting

 

Mar 01 17:20:37 !!! Errlog message received (message is above)
Mar 01 17:20:37 Warning: Virtual circuit disconnect 172.23.248.195:5064

 

 

 

A call to "assert (item.pList == this)" failed in ../../../../src/cas/generic/st/ioBlocked.cc line 112.

 

Mar 01 18:30:04 !!! Errlog message received (message is above)
EPICS Release EPICS R3.14.8.2 $R3-14-8-2$ $2006/01/06 15:55:13$.

 

Mar 01 18:30:04 !!! Errlog message received (message is above)
Current time Wed Mar 01 2006 18:30:04.20652000.

 

Mar 01 18:30:04 !!! Errlog message received (message is above)
Please E-mail this message to Jeff Hill johill@lanl.gov or to tech-talk@aps.anl.gov

 

Mar 01 18:30:04 !!! Errlog message received (message is above)
Calling epicsThreadSuspendSelf()

 

Mar 01 18:30:04 !!! Errlog message received (message is above)
Mar 01 19:28:11 PV Gateway Ending (SIGTERM)

 

 

------------------------------------------

Mr P J Leicester

Senior Software Engineer

Diamond Light Source Ltd.

Rutherford Appleton Laboratory

Chilton

Didcot

OX11 0QX

 

Tel: 01235 778478