Re: CA inside IOC - shutdown problem

[Andrew Johnson <anj@aps.anl.gov>]

Heinrich du Toit wrote:
> Everything works fine but at shutdown I get some funny errors at times.

This is a known problem on Linux distributions that use the NPTL (the 
New Posix Thread Library).

> One thing that happens is I get:
> epicsThreadOnceOsd epicsMutexLock failed
>
> And other times I get:
> epicsThread: Unknown C++ exception in thread "CAC-UDP" at Fri Sep 28
> 2007 08:39:57.528629432
> terminate called after throwing an instance of '
'
> FATAL: exception not rethrown
> Aborted (core dumped)

That particular error is an interaction between C++ and the NPTL: NPTL 
uses the C++ exception mechanism to cause Posix thread cleanup routines 
to be called when the process exits, but the thing that it throws can't 
be caught by any C++ code.  Unfortunately in EPICS Base we mix Posix and 
C++, and the result is the above error on shutdown.  The error appears 
to be harmless in my experience.

> But this doesn't happen everytime.
> Seem to be something todo with timing during exit?
> I have a hooked epicsatExit function.
> This makes sure ca_clear_channel is called for every CHID I have.
> I don't call ca_destroy_context as that seem to be done by shutdown
> itself?
>
> Can somebody just give me a tip on what those error messages mean? Then
> maybe I can guess what the bug is.

I'm not sure about the "epicsMutexLock failed" error although it is most 
likely that it's related, but we do know that the IOC code is not good 
at shutting itself down cleanly â it was designed to run on an embedded 
vxWorks CPU whose only job was to run that IOC code, and shutdown wasn't 
considered to be important functionality.  As a result we don't have the 
mechanisms in place in our APIs to notify records or device support to 
shut themselves down.  It's likely that we will not be able to properly 
address this problem in current versions of EPICS, although if/when we 
redesign the record and device APIs shutdown will be one of the design 
requirements.

> Also sometimes (1 out of 30+ times) I get a segfault while starting the
> IOC?  Is it dangerous todo some CA stuff during some stages of the IOC
> startup?

You should probably avoid using the CA client library before the startup 
script's call to iocInit has completed.  After that is fine, but do 
register an epicsAtExit() routine (after iocInit) and have that stop any 
threads and CA calls from occurring afterwards.

- Andrew
--
When a distinguished but elderly scientist states that something is
possible, he is almost certainly right.  When he states that something
is impossible, he is very probably wrong.  -- Arthur C. Clarke