Re: CA gateway dies without error message

[Dirk Zimoch <dirk.zimoch@psi.ch>]

Hi Jeff,

There is nothing to trace. The process is not there any more. The 
gateway process just ends without any message. Just as if something has 
called exit().

My start line is:
export EPICS_CA_MAX_ARRAY_BYTES=4000000
ulimit -s 2048
export EPICS_CAS_BEACON_ADDR_LIST=172.21.10.255
caGateway -no_cache -archive -prefix FIN-CCAGW11W-EXP \
	-sip 172.21.10.22 \
	-cip 172.21.11.255 \
	-uid 922 -gid 922 \
	-pvlist GATEWAY.pvlist \
	-access GATEWAY.access \
	-log gateway.log \
	"$@" > gateway.err 2>&1 < /dev/null &

Gateway startup log:
Aug 19 15:32:36 PV Gateway Version 2.0.4.0 [Oct 16 2009 10:54:59]
EPICS 3.14.11 PID=4757
EPICS_CA_ADDR_LIST=172.21.11.255
EPICS_CA_AUTO_ADDR_LIST=NO
EPICS_CA_SERVER_PORT=Not specified
EPICS_CA_MAX_ARRAY_BYTES=4000000
EPICS_CAS_INTF_ADDR_LIST=172.21.10.22
EPICS_CAS_SERVER_PORT=Not specified
EPICS_CAS_IGNORE_ADDR_LIST=Not specified
Running as user cagw on host fin-ccagw11w
Statistics PV prefix is FIN-CCAGW11W-EXP

There is nothing in gateway.err and no error message in gateway.log with 
a time stamp near the crash. The usual goodbye message like "Jul 13 
17:12:40 PV Gateway Ending (SIGTERM)" is missing.

I will try to enable core dumps. Haven't done that yet.

Dirk

Jeff Hill wrote:
> Hi Dirk,
>
> I did put some considerable effort into improving the quality (in terms of
> efficiency, stability, performance, and functionality) of the PCAS and the 
> gateway around this time last year. And I believe that the result was a
> significant step forward. Which version of the gateway are you running? 
>
> Currently, I have no clues on your issue to start with so it's hard to move 
> forward. So I will 2nd Andrew's suggestion of attaching gdb and getting a
> stack 
> trace of the failed thread. One can attach gdb to a running process by
> typing
> "gdb <full path to image> <process id>". If gdb can't be attached to the 
> running process then another alternative is to attach gdb to the core
> file using "gdb  <full path to image> <full path to core file>".
> In gdb type first "bt" to show the 
> thread that crashed and then follow that with "thread apply all bt" so that 
> I know what all threads are up to when the crash occurred. Lots of output
> but 
> exactly what is needed to get the matter resolved. This works better if the 
> code is compiled for debugging, but I might be able to resolve the issue
> with 
> only the stack traces from the optimized version (which runs faster and so
> can have a different set of failure scenarios) - so I am very willing to
> look at traces from an optimized version first if it will save some effort 
> on your end. 
>
> Thanks in advance for your help,
>
> Jeff
> ______________________________________________________
> Jeffrey O. Hill           Email        johill@lanl.gov
> LANL MS H820              Voice        505 665 1831
> Los Alamos NM 87545 USA   FAX          505 665 5107
>
> Message content: TSPA
>
>
> > -----Original Message-----
> > From: tech-talk-bounces@aps.anl.gov [mailto:tech-talk-
> > bounces@aps.anl.gov] On Behalf Of Dirk Zimoch
> > Sent: Thursday, August 19, 2010 7:52 AM
> > To: EPICS
> > Subject: CA gateway dies without error message
> >
> > Hi all,
> >
> > Since upgrading the EPICS base version of the caGateways to 3.14.11,
> > the
> > gateway dies from time to time for no obvious reason. It does not write
> > any log message when it dies (as it would do in a clean shutdown, in
> > case of a signal or when any strange CA event happens.) It also does
> > not
> > write anything to stderr. I cannot remember that I have seen this type
> > of fault with 3.13.8.
> >
> > Any idea?
> >
> > Dirk