RE: Sequence monitor not getting callback

["Jeff Hill" <johill@lanl.gov>]

Hi Murali,

So in both of those GDB dumps there is only one event queue processing
thread running, and both of them are parked in the normal place waiting for
a signal indicating that the queue is populated. 

That is probably a strange result if ...

o dbel is indicating at the same time that this same event queue has labor
(queue entries) pending against it. Please note also the task id diagnostic
in the dbel output and cross reference this id to what is seen in the "info
threads" output if possible.

o _and_ the stack trace for the event thread _always_, in several tries,
shows the event processing thread is in a labor pending semaphore wait
state. The event processing thread has the function "event_task" on its call
stack.

o _and_ the cpu wasn't saturated, executing in threads with higher
priorities than the event processing thread, at that time

> In both of these, the client and server ran in the same address space. In
> both of these, I launched the program, made sure we had the issue and then
> attached to the running process using "attach" from within gdb

Was the CPU busy when you captured these stack traces? Did you see any
messages indicating that a thread failed, and might have exited? Please also
send a copy of your standalone ca client test code and some instructions on
how to reproduce the problem so I can take a closer look at the situation.
Also, what version of EPICS is in use (sorry if you have already sent that
information and I lost it somehow in my mailbox). I am starting to worry
that there is an issue specific to SMP Linux. Was this running on a
multi-core CPU?

Thanks,

Jeff

Thread 2 (Thread 0x7f9fd3fde700 (LWP 9411)):
#0  0x0000003939e0b44c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x00007fa02e439860 in condWait (condId=0x7f9fc4000978,
mutexId=0x7f9fc4000950) at ../../../src/libCom/osi/os/posix/osdEvent.c:75
#2  0x00007fa02e439bb8 in epicsEventWait (pevent=0x7f9fc4000950) at
../../../src/libCom/osi/os/posix/osdEvent.c:137
#3  0x00007fa02ed3cc85 in event_task (pParm=0x7f9fec022c58) at
../dbEvent.c:914
#4  0x00007fa02e4381fe in start_routine (arg=0x7f9fc4000aa0) at
../../../src/libCom/osi/os/posix/osdThread.c:282
#5  0x0000003939e077e1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000039396e68ed in clone () from /lib64/libc.so.6

Thread 2595 (Thread 0x7f762b8ae700 (LWP 6653)):
#0  0x0000003939e0b44c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x00007f7648678860 in condWait (condId=0x7f760001a028,
mutexId=0x7f760001a000) at ../../../src/libCom/osi/os/posix/osdEvent.c:75
#2  0x00007f7648678bb8 in epicsEventWait (pevent=0x7f760001a000) at
../../../src/libCom/osi/os/posix/osdEvent.c:137
#3  0x00007f7648f7bc85 in event_task (pParm=0x7f7600018ee8) at
../dbEvent.c:914
#4  0x00007f76486771fe in start_routine (arg=0x7f760001a150) at
../../../src/libCom/osi/os/posix/osdThread.c:282
#5  0x0000003939e077e1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000039396e68ed in clone () from /lib64/libc.so.6
______________________________________________________
Jeffrey O. Hill           Email        johill@lanl.gov
LANL MS H820              Voice        505 665 1831
Los Alamos NM 87545 USA   FAX          505 665 5107

Message content: TSPA

With sufficient thrust, pigs fly just fine. However, this is
not necessarily a good idea. It is hard to be sure where they
are going to land, and it could be dangerous sitting under them
as they fly overhead. -- RFC 1925


> -----Original Message-----
> From: Shankar, Murali [mailto:mshankar@slac.stanford.edu]
> Sent: Wednesday, October 12, 2011 12:43 PM
> To: Shankar, Murali; J. Lewis Muir
> Cc: Jeff Hill; tech-talk@aps.anl.gov
> Subject: RE: Sequence monitor not getting callback
> 
> Here's a couple of files with GDB back traces.
> 
> The file GDB_SequencerTest_Oct12_2011.txt contains "info threads" and
> "thread apply all bt" from a test involving sequence programs.
> 
> The file GDB_OnlyCA_Oct12_2011.txt "info threads" and "thread apply all
> bt" from a test involving only CA.
> 
> In both of these, the client and server ran in the same address space. In
> both of these, I launched the program, made sure we had the issue and then
> attached to the running process using "attach" from within gdb.
> 
> Please let me know if more info is needed or what the next steps are.
> 
> Regards,
> Murali
> 
> 
> -----Original Message-----
> From: Shankar, Murali
> Sent: Wednesday, October 12, 2011 10:40 AM
> To: 'J. Lewis Muir'
> Cc: Jeff Hill; tech-talk@aps.anl.gov
> Subject: RE: Sequence monitor not getting callback
> 
> Ok will do...
> 
> Regards,
> Murali
> 
> 
> -----Original Message-----
> From: J. Lewis Muir [mailto:jlmuir@anl.gov]
> Sent: Wednesday, October 12, 2011 10:40 AM
> To: Shankar, Murali
> Cc: Jeff Hill; tech-talk@aps.anl.gov
> Subject: Re: Sequence monitor not getting callback
> 
> On 10/12/11 12:18 PM, Shankar, Murali wrote:
> > This is on linux so I will pop this into gdb and send you the relevant
> info offline.
> 
> Hi, Murali.
> 
> If there's nothing private, could you just keep everything on
> the mailing list?  It's better for using the list archive in the
> future and it's better for anyone following this problem to be
> able to see what's going on, how it was investigated, and how it
> was solved.
> 
> Thanks!
> 
> Lewis