EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Re: libca bug?
From: Bruno Martins via Tech-talk <[email protected]>
To: "Johnson, Andrew N." <[email protected]>
Cc: "[email protected]" <[email protected]>
Date: Fri, 3 May 2019 17:47:45 -0400
Thanks for your inputs!

I will try Andrew's suggestion next week (removing caClientCallbackThreadId=0).

Michael -- I will look at how the sequencer does it. Thanks for the suggestion!

Matt -- I did set the PYEPICS_LIBCA to the proper path and set AUTO_CLEANUP to False. I tried several different things, even manually cleaning up the caches. The whole issue, in my opinion, is what Andrew diagnosed: zeroing the common index makes it impossible to do anything with the libca context afterwards.


Here's a proof of concept that triggers the bug: https://github.com/brunoseivam/libca-bug-poc

Versions:
Debian 9.9
EPICS R3.15.6 (FRIB package)
pyepics 3.4.0 (FRIB package)
pyDevSup: 1.1 (FRIB package)


Thanks,

Bruno


On Fri, May 3, 2019 at 5:33 PM Johnson, Andrew N. via Tech-talk <[email protected]> wrote:
On 5/3/19 3:40 PM, Michael Davidsaver via Tech-talk wrote:
> On 5/3/19 12:43 PM, Bruno Martins via Tech-talk wrote:
>> I believe that the following are true:
>>
>>   -  pyepics dynamically loads libca, which ends up being the same instance as the one loaded by the IOC itself (for dbCa)
>>   -  pyepics does not share the same context with dbCa if it is run on its own thread and the proper context_create call is performed
>>   -  Even when not sharing libca contexts, the variable caClientCallbackThreadId ends up being the same for pyepics and dbCa.
>>   -  cacExitHandler runs when the IOC is exiting and sets caClientCallbackThreadId=0 [1]
>>   -  Any subsequent libca call by pyepics (during its cleanup) that tries to use caClientCallbackThreadId will segfault.
>>
>> Possible solutions:
>>
>>   -  Register pyepics' finalize_libca with pyDevSup's addHook('AtIocExit',...), but this runs after dbCa cleanup. Can it be made to run before? Can there be a 'BeforeIocExit' hook?
> Unfortunately no.  Though this does suggest a rethink of addHook(),
> and maybe also epicsAtExit() to be able to better express ordering
> requirements.
Is pyepics registing an epicsAtExit() hook? These always run in the
reverse order in which they are registered, so pyepics should be
initializing its CA context and registering such a hook when it first
gets called, then cleaning up inside that hook routine, which will be
called before the hook routine of any subsystem that got initialized
before it.

>>   -  Remove caClientCallbackThreadId=0 in libca. I have no idea about the implications.
I removed a similar statement that was zeroing the caClientContextId
last february (bug #1743076 on Launchpad, SHA
de442e9584d9214533e5e90af4270e34fd45d947) which was included in
Base-3.15.6 and had a similar effect. I don't know whether that will
resolve your problem, but I would definitely recommend trying it.

>>   -  Something else?
>>
>> Of course, I understand that both the IOC itself and pyepics kind of expect to be the sole users of libca in their processes. This assumption doesn't hold when I have pyepics inside a IOC.
> The IOC isn't making such an assumption.  eg. the sequencer creates its own CA context.
I don't think the two subsystems are sharing a context, the problem as I
see it is that when the cacExitHandler() runs in one thread it zeroes
the common index into the thread-private table, thus making it
impossible for any other thread that's still running in a different
context to read the value from its copy of the thread-local variable
table. I see no reason to yank the ID here at all.

- Andrew

--
Complexity comes for free, Simplicity you have to work for.


References:
libca bug? Bruno Martins via Tech-talk
Re: libca bug? Michael Davidsaver via Tech-talk
Re: libca bug? Johnson, Andrew N. via Tech-talk

Navigate by Date:
Prev: Re: libca bug? Johnson, Andrew N. via Tech-talk
Next: Re: libca bug? Matt Newville via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Re: libca bug? Johnson, Andrew N. via Tech-talk
Next: Re: libca bug? Matt Newville via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024 
ANJ, 03 May 2019 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·