Experimental Physics and
Industrial Control System

Jameson Graef Rollins <[email protected]> · Wed, 13 May 2015 10:06:17 -0700

On Tue, May 12 2015, Michael Davidsaver <[email protected]> wrote:
> On 05/12/2015 02:48 PM, Jameson Graef Rollins wrote:
>> The most troubling aspect of the problem is that the actual PVs remain
>> active, while the underlying process freezes.  Existing PV subscriptions
>> remain valid and the PCAS continues to respond to new requests.  However
>> the database is no longer being updated so all values are bogus.  In
>> other words, we have **no external indication that the process is in a
>> frozen state**.
>
> Since the process continues running can you attached gdb and get stack
> traces?  This should show the place where the _main_ thread has stopped.

Hi, Michael.  I should have mentioned that I did try doing this at your
suggestion, but wasn't able to devine anything useful myself.  I'm not
quite sure what I'm looking for, though:

(gdb) attach 22881
Attaching to process 22881
Reading symbols from /usr/bin/python2.7...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
...
Loaded symbols for /ligo/apps/ubuntu12/epics-3.14.12.2_long/base-3.14.12.2/lib/linux-x86_64/libdbStaticIoc.so.3.14
Reading symbols from /lib/x86_64-linux-gnu/libtinfo.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libtinfo.so.5
0x00007ffa342d8d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0

The thread backtrace shows four threads:

(gdb) thr app all bt

Thread 4 (Thread 0x7ffa3466a700 (LWP 22916)):
#0  0x00007ffa342d8d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007ffa2e392809 in condWait (mutexId=0x2f4d4f0, condId=0x2f4d518) at ../../../src/libCom/osi/os/posix/osdEvent.c:75
...

Thread 3 (Thread 0x7ffa2cca0700 (LWP 22932)):
#0  0x00007ffa330a7c23 in select () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffa2e380326 in fdManager::process(double) () from /ligo/apps/ubuntu12/epics-3.14.12.2_long/base/lib/linux-x86_64/libCom.so.3.14
...

Thread 2 (Thread 0x7ffa27fff700 (LWP 22933)):
#0  0x00007ffa330a7c23 in select () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00000000004c8bf3 in ?? ()
...

Thread 1 (Thread 0x7ffa346ee700 (LWP 22881)):
#0  0x00007ffa342d8d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007ffa2e392809 in condWait (mutexId=0x2f479e0, condId=0x2f47a08) at ../../../src/libCom/osi/os/posix/osdEvent.c:75
...

It looks like Thread 3 is the main epics thread, which seems to be
waiting for a select.  Is there anything to be divined from this?

jamie.

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System