EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Re: libca bug?
From: Matt Newville via Tech-talk <[email protected]>
To: "Johnson, Andrew N." <[email protected]>, Bruno Martins <[email protected]>
Cc: "[email protected]" <[email protected]>
Date: Fri, 3 May 2019 16:57:01 -0500
Hi, 

On Fri, May 3, 2019 at 4:33 PM Johnson, Andrew N. via Tech-talk <[email protected]> wrote:
On 5/3/19 3:40 PM, Michael Davidsaver via Tech-talk wrote:
> On 5/3/19 12:43 PM, Bruno Martins via Tech-talk wrote:
>> I believe that the following are true:
>>
>>   -  pyepics dynamically loads libca, which ends up being the same instance as the one loaded by the IOC itself (for dbCa)
>>   -  pyepics does not share the same context with dbCa if it is run on its own thread and the proper context_create call is performed
>>   -  Even when not sharing libca contexts, the variable caClientCallbackThreadId ends up being the same for pyepics and dbCa.
>>   -  cacExitHandler runs when the IOC is exiting and sets caClientCallbackThreadId=0 [1]
>>   -  Any subsequent libca call by pyepics (during its cleanup) that tries to use caClientCallbackThreadId will segfault.
>>
>> Possible solutions:
>>
>>   -  Register pyepics' finalize_libca with pyDevSup's addHook('AtIocExit',...), but this runs after dbCa cleanup. Can it be made to run before? Can there be a 'BeforeIocExit' hook?
> Unfortunately no.  Though this does suggest a rethink of addHook(),
> and maybe also epicsAtExit() to be able to better express ordering
> requirements.
Is pyepics registing an epicsAtExit() hook? These always run in the
reverse order in which they are registered, so pyepics should be
initializing its CA context and registering such a hook when it first
gets called, then cleaning up inside that hook routine, which will be
called before the hook routine of any subsystem that got initialized
before it.

It is not registering an epicsAtExit() hook.  It does register a Python "at end of Python process" hook that does cleanup of the CA connections.  Specifically, pyepics needs to hold on to a (python) global cache of connections so that they don't get garbage collected away, and that needs to be cleaned up when the process ends.  I can believe that would need to be altered for "embedded in IOC".   In fact, Ken Lauer very recently re-did much of this caching, partly to help avoid problems exchanging PVs between threads in multi-threaded apps.   It may be that this when used within an IOC.

If it would help to register an epicsAtExit() so that connections made by pyepics were cleared before the IOC cleaned up its connections, then sure, let's do that.


>>   -  Remove caClientCallbackThreadId=0 in libca. I have no idea about the implications.
I removed a similar statement that was zeroing the caClientContextId
last february (bug #1743076 on Launchpad, SHA
de442e9584d9214533e5e90af4270e34fd45d947) which was included in
Base-3.15.6 and had a similar effect. I don't know whether that will
resolve your problem, but I would definitely recommend trying it.

>>   -  Something else?
>>
>> Of course, I understand that both the IOC itself and pyepics kind of expect to be the sole users of libca in their processes. This assumption doesn't hold when I have pyepics inside a IOC.
> The IOC isn't making such an assumption.  eg. the sequencer creates its own CA context.
I don't think the two subsystems are sharing a context, the problem as I
see it is that when the cacExitHandler() runs in one thread it zeroes
the common index into the thread-private table, thus making it
impossible for any other thread that's still running in a different
context to read the value from its copy of the thread-local variable
table. I see no reason to yank the ID here at all.
 
OK.  Just to be clear, pyepics will create a context, and then work very hard to make sure all of its threads use that.   It does currently assumes that it is responsible for cleaning up that context on exit. 

@Bruno Martins  OK thanks it's good to know that this is still happening after Ken's other fixes to the caching.   It sounds like to me like adding an epicsAtExit() hook to pyepics'  initialize_libca() might be the right thing to try next.

--Matt 

References:
libca bug? Bruno Martins via Tech-talk
Re: libca bug? Michael Davidsaver via Tech-talk
Re: libca bug? Johnson, Andrew N. via Tech-talk

Navigate by Date:
Prev: Re: libca bug? Bruno Martins via Tech-talk
Next: Re: libca bug? Michael Davidsaver via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Re: libca bug? Bruno Martins via Tech-talk
Next: Re: libca bug? Matt Newville via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024 
ANJ, 03 May 2019 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·