RE: EPICS R3.14.11, sequencer 2.0.12 crash on device support write error

["Jeff Hill" <johill@lanl.gov>]

Hi Kay,

> ca_array_put catches all sorts of exceptions, 
> including a "catch ( ... )"
> in ca/oldChannelNotify.cpp line 522.

Yea, it would appear that an exception is being thrown at line 101 in
dbChannelIO::write, but is never caught by the try/catch in ca_array_put.
One can guess that an exception is being thrown when the check of the status
from db_put_field detects a failure. Perhaps there is some issue with
unwinding the stack and running the dtors for stack variables which causes
std::terminate() to be directly called by __cxa_throw (). With g++ a
function __cxa_throw () in libstc++ is invoked to unwind the stack when an
exception gets thrown.

The problem is that I don?t see any reason why __cxa_throw might fail to
successfully unwind the stack, and I would bet that this exception path has
run w/o issues quite frequently in the past. The only dtor that needs to be
run is ~epicsGuardRelease which is quite exceptionally simple. I also see no
sign of functions involved that have exceptions specifications (which might
cause std :: unexpected to be invoked). Perhaps it's premature to suspect a
compiler bug, but it does appear that this path must have run on other
environments. It should be quite easy to test this code path. Just write the
string "off" to an analog output record using a very simple (integrated in
memory into the IOC) ca client. You could run that test on a soft IOC on
Linux for example to see if this is a vxWorks specific issue.

Here is another less likely possibility. Perhaps an exception somehow is
thrown within db_put_field, somehow g++ has an implicit try catch around all
calls to extern "C" and it is calling std :: terminate when an extern "C"
code throws an exception. I have seen MSVC converting a hard exception such
as an access violation into a C++ exception, but don?t recall that I have
ever observed this with g++.

Here is another guess; are you properly building (linking) specifying C++
exception handling runtime support is necessary for your sequencer program
(which is mostly C based bust nonetheless has some c++ library
subcomponents)?

Jeff
______________________________________________________
Jeffrey O. Hill           Email        johill@lanl.gov
LANL MS H820              Voice        505 665 1831
Los Alamos NM 87545 USA   FAX          505 665 5107

Message content: TSPA


> -----Original Message-----
> From: tech-talk-bounces@aps.anl.gov [mailto:tech-talk-
> bounces@aps.anl.gov] On Behalf Of Kasemir, Kay
> Sent: Wednesday, July 07, 2010 9:11 AM
> To: EPICS Tech-Talk
> Subject: EPICS R3.14.11, sequencer 2.0.12 crash on device support write
> error
> 
> Hello:
> 
> I ran into the following issue upgrading from EPICS R3.14.8.2 to
> R3.14.11.
> In case others see this, here's a description and workaround.
> 
> It starts with device support that detects if the underlying I/O hardware
> is
> not present.
> 
> For the AO record, this device support write_ao() routine used to
> 1) Put the record into alarm
>    recGblSetSevr(rec, WRITE_ALARM, INVALID_ALARM);
> 2) return some error code like 1 or 2.
> 
> The return code is critical for the following.
> 
> According to the record reference manual, it's OK to return 0 or anything
> else for errors:
> "When finished, write_ao should ... return one the following values:
>  0: Success.
>  other: Error.
> "
> 
> So far, that had worked OK. If there's a write error, the record goes
> into
> WRITE/INVALID. In addition, a Channel Access client sees an error like
> this:
> 
> caput MyRecord 1
> CA.Client.Exception...............................................
>     Warning: "Channel write request failed"
>     Context: "op=1, channel= MyRecord , type=DBR_DOUBLE, count=1, ctx="
> MyRecord""
>     Source File: ../oldChannelNotify.cpp line 159
>     Current Time: Wed Jul 07 2010 08:53:52.816914000
> 
> 
> 
> After switching from R3.14.8.2 to R3.14.11, however, sequencer code that
> writes to such channels would crash:
> 
> 0x6474e0 (name_of_my_sequence): Unhandled C++ exception resulted in call
> to
> terminate
> Thread name_of_my_sequence (0x6474e0) suspended
> 
> This is on vxWorks 6.7 on an MVME5500. Stack trace:
> >tt 0x6474e0
> 0x00137398 vxTaskEntry  +0x48 : 0x014ef464 ()
> 0x014ef4d0 epicsThreadCreate+0x1e4: sequencer ()
> 0x014284f4 sequencer    +0x19c: 0x0142817c ()
> 0x0142817c ss_entry     +0x498: 0x013ff814 ()
> 0x013ff8d4 name_of_my_sequenceRegistrar+0x138: seq_pvPut ()
> 0x01425900 seq_pvPut    +0xb0 : 0x0142b54c ()
> 0x0142b598 pvVarPutNoBlock+0x50 : caVariable::putNoBlock(pvType, int,
> pvValue *) ()
> 0x0142e100 caVariable::putNoBlock(pvType, int, pvValue *)+0x104:
> ca_array_put (0)
> 0x014a5cb8 ca_array_put +0xa4 : dbChannelIO::write(epicsGuard<epicsMutex>
> &,
> unsigned int, unsigned long, const void  ()
> 0x014851c8 dbChannelIO::write(epicsGuard<epicsMutex> &, unsigned int,
> unsigned long, const void +0x1cc: __cxa_throw ()
> 0x00126ac0 __cxa_throw  +0x70 : std::terminate() ()
> 0x0012699c __cxxabiv1::__unexpected(void (*)())+0x0  :
> __cxxabiv1::__terminate(void (*)()) ()
> 0x00126968 __cxxabiv1::__terminate(void (*)())+0x18 : cplusTerminate() ()
> 0x00145e70 cplusTerminate()+0x50 : taskSuspend ()
> 
> As far as I could track this down in the source code, the sequencer's
> caVariable::putNoBlock calls ca_array_put (pv/pvCa.cc line 292).
> 
> ca_array_put calls pChan->io.write (ca/oldChannelNotify.cpp line 487).
> 
> That in turn calls the database and finally my write_ao() routine.
> 
> Something must throw an exception in response to the non-zero return code
> of
> my write_ao. I don't understand why that causes a problem, because
> ca_array_put catches all sorts of exceptions, including a "catch ( ... )"
> in
> ca/oldChannelNotify.cpp line 522.
> 
> Still, the sequencer crashes with an unhandled C++ exception as long as
> the
> write_ao of the record's device support returns a non-zero status code.
> 
> I stopped trying to debug it, and instead changed my device support to
> always return 0 for OK, because the WRITE/INVALID status/severity already
> reports the underlying I/O hardware problem to some extend.
> 
> Thanks,
> Kay