Re: libca bug?

[Bruno Martins via Tech-talk <tech-talk@aps.anl.gov>]

Thanks for your inputs!

I will try Andrew's suggestion next week (removing caClientCallbackThreadId=0).

Michael -- I will look at how the sequencer does it. Thanks for the suggestion!

Matt -- I did set the PYEPICS_LIBCA to the proper path and set AUTO_CLEANUP to False. I tried several different things, even manually cleaning up the caches. The whole issue, in my opinion, is what Andrew diagnosed: zeroing the common index makes it impossible to do anything with the libca context afterwards.

Here's a proof of concept that triggers the bug: https://github.com/brunoseivam/libca-bug-poc

Here's a backtrace: https://gist.github.com/brunoseivam/dc3d0807c1c249ea2fbcaca2a08c278a

Versions:

Debian 9.9

EPICS R3.15.6 (FRIB package)

pyepics 3.4.0 (FRIB package)

pyDevSup: 1.1 (FRIB package)

Thanks,

Bruno

On Fri, May 3, 2019 at 5:33 PM Johnson, Andrew N. via Tech-talk <tech-talk@aps.anl.gov> wrote:

On 5/3/19 3:40 PM, Michael Davidsaver via Tech-talk wrote:
> On 5/3/19 12:43 PM, Bruno Martins via Tech-talk wrote:
>> I believe that the following are true:
>>
>>   -  pyepics dynamically loads libca, which ends up being the same instance as the one loaded by the IOC itself (for dbCa)
>>   -  pyepics does not share the same context with dbCa if it is run on its own thread and the proper context_create call is performed
>>   -  Even when not sharing libca contexts, the variable caClientCallbackThreadId ends up being the same for pyepics and dbCa.
>>   -  cacExitHandler runs when the IOC is exiting and sets caClientCallbackThreadId=0 [1]
>>   -  Any subsequent libca call by pyepics (during its cleanup) that tries to use caClientCallbackThreadId will segfault.
>>
>> Possible solutions:
>>
>>   -  Register pyepics' finalize_libca with pyDevSup's addHook('AtIocExit',...), but this runs after dbCa cleanup. Can it be made to run before? Can there be a 'BeforeIocExit' hook?
> Unfortunately no.  Though this does suggest a rethink of addHook(),
> and maybe also epicsAtExit() to be able to better express ordering
> requirements.
Is pyepics registing an epicsAtExit() hook? These always run in the
reverse order in which they are registered, so pyepics should be
initializing its CA context and registering such a hook when it first
gets called, then cleaning up inside that hook routine, which will be
called before the hook routine of any subsystem that got initialized
before it.

>>   -  Remove caClientCallbackThreadId=0 in libca. I have no idea about the implications.
I removed a similar statement that was zeroing the caClientContextId
last february (bug #1743076 on Launchpad, SHA
de442e9584d9214533e5e90af4270e34fd45d947) which was included in
Base-3.15.6 and had a similar effect. I don't know whether that will
resolve your problem, but I would definitely recommend trying it.

>>   -  Something else?
>>
>> Of course, I understand that both the IOC itself and pyepics kind of expect to be the sole users of libca in their processes. This assumption doesn't hold when I have pyepics inside a IOC.
> The IOC isn't making such an assumption.  eg. the sequencer creates its own CA context.
I don't think the two subsystems are sharing a context, the problem as I
see it is that when the cacExitHandler() runs in one thread it zeroes
the common index into the thread-private table, thus making it
impossible for any other thread that's still running in a different
context to read the value from its copy of the thread-local variable
table. I see no reason to yank the ID here at all.

- Andrew

--
Complexity comes for free, Simplicity you have to work for.