RE: "epicsMutex pthread_mutex_unlock failed" with pyepics/pcaspy

[Mark Rivers <rivers@cars.uchicago.edu>]

> One odd thing I notice.  It seems that GDB finds the debug symbols for
> libCom, but not for libcas.  It seems unlikely that this would be true
> if both libraries were from the same build.  

I believe there was a problem in base that was causing debug symbols to be loaded for C code but not C++ code.  It was fixed in 3.14.12.4.

Mark




-----Original Message-----
From: tech-talk-bounces@aps.anl.gov [mailto:tech-talk-bounces@aps.anl.gov] On Behalf Of Michael Davidsaver
Sent: Wednesday, May 13, 2015 2:26 PM
To: Jameson Graef Rollins; tech-talk@aps.anl.gov
Subject: Re: "epicsMutex pthread_mutex_unlock failed" with pyepics/pcaspy

On 05/13/2015 02:52 PM, Jameson Graef Rollins wrote:
> ...
> This program utilizes two processes: a "main" daemon process and a
> "worker" sub-process.  The daemon holds the PCAS, while the worker uses
> pyepics to do the work.
> 
> Attached is the gdb bt output for the threads of both the "MAIN" and the
> "WORKER" for one of these wedged nodes.  Like I said, I'm pretty sure
> the problem is in the PCAS, so I think the problem is in the MAIN, but I
> include the WORKER bt for completeness.
> 
> I don't actually have a wedged node right now, so I can't get any more
> info until it happens again.  If there's some other gdb info I can
> retrieve next time this happens, please do let me know.

The thread in question (it's sitting in cantProceed()).  It is indeed in
a PCAS call (see below).

One odd thing I notice.  It seems that GDB finds the debug symbols for
libCom, but not for libcas.  It seems unlikely that this would be true
if both libraries were from the same build.  Also I note that libca is
being loaded from a slightly different path (symlink?).

> Reading symbols from /ligo/apps/ubuntu12/epics-3.14.12.2_long/base/lib/linux-x86_64/libCom.so.3.14...done.
> Reading symbols from /ligo/apps/ubuntu12/epics-3.14.12.2_long/base/lib/linux-x86_64/libcas.so.3.14...(no debugging symbols found)...done.
...
> Reading symbols from /ligo/apps/ubuntu12/epics-3.14.12.2_long/base-3.14.12.2/lib/linux-x86_64/libca.so.3.14

This might just be a quirk of GDB searching though.  Can you check on a
running process ("cat /proc/<pid>/maps") to see which libraries were loaded?

Also, running "ldd -v _pcas.so" might be interesting.

FYI I'd recommend setting SHRLIB_VERSION to something more specific than
"3.14".


> Thread 1 (Thread 0x7f8658d49700 (LWP 26265)):
> #0  0x00007f8658933d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00007f86529ed809 in condWait (mutexId=0x301f080, condId=0x301f0a8) at ../../../src/libCom/osi/os/posix/osdEvent.c:75
> #2  epicsEventWait (pevent=0x301f080) at ../../../src/libCom/osi/os/posix/osdEvent.c:137
> #3  0x00007f86529ec158 in epicsThreadSuspendSelf () at ../../../src/libCom/osi/os/posix/osdThread.c:557
> #4  0x00007f86529e2d05 in cantProceed (msg=<optimized out>) at ../../../src/libCom/misc/cantProceed.c:69
> #5  0x00007f8652568525 in casPVI::getExtServer() const () from /ligo/apps/ubuntu12/epics-3.14.12.2_long/base/lib/linux-x86_64/libcas.so.3.14
> #6  0x00007f8652c2efa3 in PV::postEvent (this=0x30a0200, value=...) at pcaspy/pv.cpp:83
> #7  0x00007f8652c1fdc9 in _wrap_PV_postEvent (args=<optimized out>) at pcaspy/casdef_wrap.cpp:9193
> #8  0x0000000000571b58 in PyEval_EvalFrameEx ()