Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019 
<== Date ==> <== Thread ==>

Subject: RE: asyn+streamDevice hangs with synchronousLock:Yes
From: Mark Rivers <rivers@cars.uchicago.edu>
To: "'Brian McAllister'" <mcallister@mit.edu>
Cc: "tech-talk@aps.anl.gov" <tech-talk@aps.anl.gov>
Date: Mon, 8 Jul 2013 15:00:45 +0000
> The reference to asynManager.c is in registerPort().  It seems odd that
> this function would be called other than during iocInit(), and that it
> would get stuck trying to create what should be a new mutex.

registerPort() not getting called again.  epicsMutexShowAll reports the line in which the mutex was originally created, which in this case happened before iocInit when registerPort() was called.

I had guessed that there was a deadlock involving another mutex created by StreamDevice, but that does not seem to be the case.

It would be helpful if you could run the gdb debugger on the IOC application when it is hung up.

Do that as follows:

- Your IOC is running as a process on the Linux system.  Get its process ID using the "ps" command.

For example, here I will debug the testAsynPortDriver IOC application:
corvette:asyn/iocBoot/ioctestAsynPortDriver>ps ax | grep testAsyn
26058 pts/33   Sl+    0:00 ../../bin/linux-x86/testAsynPortDriver st.cmd
26188 pts/9    S+     0:00 grep --color=auto testAsyn

So the IOC process ID is 26058.

- Start gdb as follows:

gdb [path_to_IOC_application] [process_ID]

For my example above:
corvette:asyn/iocBoot/ioctestAsynPortDriver>gdb ../../bin/linux-x86/testAsynPortDriver 26058

Once the debugger starts issue the following command:

(gdb) thread apply all backtrace

For example, the beginning of the output for my example is:
Thread 22 (Thread 0xf73f7b70 (LWP 26059)):
#0  0xf775d430 in __kernel_vsyscall ()
#1  0x4d3b84f4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x4d301d84 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0xf75b81b8 in condTimedwait (time=0xf73f7218, mutexId=0x97cfa58, condId=0x97cfa70) at ../../../src/libCom/osi/os/posix/osdEvent.c:65
#4  epicsEventWaitWithTimeout (pevent=0x97cfa58, timeout=1.7976931348623157e+308) at ../../../src/libCom/osi/os/posix/osdEvent.c:156
#5  0xf75b0bf5 in epicsEvent::wait(double) () from /usr/local/epics/base-3.14.12.3/lib/linux-x86/libCom.so.3.14
#6  0xf75be805 in timerQueueActive::run() () from /usr/local/epics/base-3.14.12.3/lib/linux-x86/libCom.so.3.14
#7  0xf75af312 in epicsThreadCallEntryPoint () from /usr/local/epics/base-3.14.12.3/lib/linux-x86/libCom.so.3.14
#8  0xf75b5cad in start_routine (arg=0x97cfbd0) at ../../../src/libCom/osi/os/posix/osdThread.c:385
#9  0x4d3b4a2e in start_thread () from /lib/libpthread.so.0
#10 0x4d2f381e in clone () from /lib/libc.so.6

Thread 21 (Thread 0xf7376b70 (LWP 26060)):
#0  0xf775d430 in __kernel_vsyscall ()
#1  0x4d3b84f4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x4d301d84 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0xf75b81b8 in condTimedwait (time=0xf7376218, mutexId=0x97cff88, condId=0x97cffa0) at ../../../src/libCom/osi/os/posix/osdEvent.c:65
#4  epicsEventWaitWithTimeout (pevent=0x97cff88, timeout=1.7976931348623157e+308) at ../../../src/libCom/osi/os/posix/osdEvent.c:156
#5  0xf75b0bf5 in epicsEvent::wait(double) () from /usr/local/epics/base-3.14.12.3/lib/linux-x86/libCom.so.3.14
#6  0xf75be805 in timerQueueActive::run() () from /usr/local/epics/base-3.14.12.3/lib/linux-x86/libCom.so.3.14
#7  0xf75af312 in epicsThreadCallEntryPoint () from /usr/local/epics/base-3.14.12.3/lib/linux-x86/libCom.so.3.14
#8  0xf75b5cad in start_routine (arg=0x97d0100) at ../../../src/libCom/osi/os/posix/osdThread.c:385
#9  0x4d3b4a2e in start_thread () from /lib/libpthread.so.0
#10 0x4d2f381e in clone () from /lib/libc.so.6


That will give a stack trace of every thread, and we can see if there are multiple threads waiting on mutexes, and what code those threads are executing.

Mark


-----Original Message-----
From: brian@hotrats.mit.edu [mailto:brian@hotrats.mit.edu] On Behalf Of Brian McAllister
Sent: Monday, July 08, 2013 9:48 AM
To: Mark Rivers
Cc: Dirk Zimoch; tech-talk@aps.anl.gov
Subject: Re: asyn+streamDevice hangs with synchronousLock:Yes 

>>> On 7/8/2013 at 3:26:08 -0000, Mark Rivers wrote:

  > That is interesting.  When it is hung and you type

  > epicsMutexShowAll, 1

  > repeatedly, do you always see the same thing, i.e.

  >> ellCount(&mutexList) 117 ellCount(&freeList) 12
  >> epicsMutexId 0x894b988 source ../../asyn/asynDriver/asynManager.c line 1868
  >> epicsMutexId 0x89a2710 source ../../../src/libCom/osi/epicsMutex.cpp line 222

Yes.  Once it gets stuck, that report doesn't change.

The reference to asynManager.c is in registerPort().  It seems odd that
this function would be called other than during iocInit(), and that it
would get stuck trying to create what should be a new mutex.

- brian


Replies:
RE: asyn+streamDevice hangs with synchronousLock:Yes Mark Rivers
References:
RE: asyn+streamDevice hangs with synchronousLock:Yes Mark Rivers
Re: asyn+streamDevice hangs with synchronousLock:Yes Brian McAllister

Navigate by Date:
Prev: Re: asyn+streamDevice hangs with synchronousLock:Yes Brian McAllister
Next: RE: asyn+streamDevice hangs with synchronousLock:Yes Mark Rivers
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019 
Navigate by Thread:
Prev: Re: asyn+streamDevice hangs with synchronousLock:Yes Brian McAllister
Next: RE: asyn+streamDevice hangs with synchronousLock:Yes Mark Rivers
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019 
ANJ, 20 Apr 2015 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·