RE: proper thread cleanup on Linux?

["Jeff Hill" <johill@lanl.gov>]

FWIW, it appears that I was being confused by features (bugs) in GDB. I just noticed that gdb reports 437 threads, but that the epicsThread library doesn't think that 400 threads are running? Closer inspection reveals that almost all of the threads listed by gdb's "info threads" have the same thread id! Gotta love programmer productivity tools that actually increase your work load :-)

(gdb) call epicsThreadShowAll (10)
NAME EPICS ID PTHREAD ID OSIPRI OSSPRI STATE
_main_ 0x986b040 0 0 0 OK
errlog 0x98a0f88 3085802416 10 0 OK
ipToAsciiProxy 0xb7b00f68 3085319088 10 0 OK

(gdb) thread 11
[Switching to thread 11 (Thread -1209164880 (LWP 4717))]#0 0x001427a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
(gdb) bt
#0 0x001427a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0x00da4b26 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/tls/libpthread.so.0
#2 0x00c16b17 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/libc.so.6
#3 0x0029bbeb in condWait (condId=0x98a2610, mutexId=0x98a25f8)
at ../../../src/libCom/osi/os/posix/osdEvent.c:75
#4 0x0029bf9e in epicsEventWait (pevent=0x98a25f8)
at ../../../src/libCom/osi/os/posix/osdEvent.c:137
#5 0x00280382 in errlogThread () at ../../../src/libCom/error/errlog.c:492
#6 0x0029a53b in start_routine (arg=0x98a0f88)
at ../../../src/libCom/osi/os/posix/osdThread.c:282
#7 0x00da2371 in start_thread () from /lib/tls/libpthread.so.0
#8 0x00c09ffe in clone () from /lib/tls/libc.so.6

(gdb) thread 4
[Switching to thread 4 (Thread -1209164880 (LWP 4706))]#0 0x001427a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
(gdb) bt
#0 0x001427a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0x00da4b26 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/tls/libpthread.so.0
#2 0x00c16b17 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/libc.so.6
#3 0x0029bbeb in condWait (condId=0x98a2610, mutexId=0x98a25f8)
at ../../../src/libCom/osi/os/posix/osdEvent.c:75
#4 0x0029bf9e in epicsEventWait (pevent=0x98a25f8)
at ../../../src/libCom/osi/os/posix/osdEvent.c:137
#5 0x00280382 in errlogThread () at ../../../src/libCom/error/errlog.c:492
#6 0x0029a53b in start_routine (arg=0x98a0f88)
at ../../../src/libCom/osi/os/posix/osdThread.c:282
#7 0x00da2371 in start_thread () from /lib/tls/libpthread.so.0
#8 0x00c09ffe in clone () from /lib/tls/libc.so.6

 (gdb) info threads
  437 Thread -1209648208 (LWP 5145)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
* 435 Thread -1209648208 (LWP 5143)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  427 Thread -1209648208 (LWP 5135)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  425 Thread -1209648208 (LWP 5133)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  417 Thread -1209648208 (LWP 5125)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  415 Thread -1209648208 (LWP 5123)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  407 Thread -1209648208 (LWP 5115)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  405 Thread -1209648208 (LWP 5113)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  397 Thread -1209648208 (LWP 5105)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  395 Thread -1209648208 (LWP 5103)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  387 Thread -1209648208 (LWP 5095)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  385 Thread -1209648208 (LWP 5093)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  377 Thread -1209648208 (LWP 5084)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  375 Thread -1209648208 (LWP 5082)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  367 Thread -1209648208 (LWP 5074)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  365 Thread -1209648208 (LWP 5072)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  357 Thread -1209648208 (LWP 5064)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  355 Thread -1209648208 (LWP 5062)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  347 Thread -1209648208 (LWP 5054)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  345 Thread -1209648208 (LWP 5052)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  337 Thread -1209648208 (LWP 5044)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  335 Thread -1209648208 (LWP 5042)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  327 Thread -1209648208 (LWP 5034)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  325 Thread -1209648208 (LWP 5032)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  317 Thread -1209648208 (LWP 5024)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  315 Thread -1209648208 (LWP 5022)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  307 Thread -1209648208 (LWP 5014)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  305 Thread -1209648208 (LWP 5012)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  297 Thread -1209648208 (LWP 5004)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  295 Thread -1209648208 (LWP 5002)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  287 Thread -1209648208 (LWP 4994)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  285 Thread -1209648208 (LWP 4992)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  277 Thread -1209648208 (LWP 4984)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  275 Thread -1209648208 (LWP 4982)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  267 Thread -1209648208 (LWP 4974)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  265 Thread -1209648208 (LWP 4972)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  257 Thread -1209648208 (LWP 4964)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  255 Thread -1209648208 (LWP 4962)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  247 Thread -1209648208 (LWP 4954)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  245 Thread -1209648208 (LWP 4952)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  237 Thread -1209648208 (LWP 4944)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  235 Thread -1209648208 (LWP 4942)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  227 Thread -1209648208 (LWP 4934)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  225 Thread -1209648208 (LWP 4932)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  217 Thread -1209648208 (LWP 4924)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  215 Thread -1209648208 (LWP 4922)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  210 Thread -1209648208 (LWP 4916)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  205 Thread -1209648208 (LWP 4911)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  200 Thread -1209648208 (LWP 4906)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  195 Thread -1209648208 (LWP 4901)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  187 Thread -1209648208 (LWP 4893)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  167 Thread -1209648208 (LWP 4873)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  165 Thread -1209648208 (LWP 4871)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  157 Thread -1209648208 (LWP 4863)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  137 Thread -1209648208 (LWP 4843)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  110 Thread -1209648208 (LWP 4816)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  102 Thread -1209648208 (LWP 4808)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  82 Thread -1209648208 (LWP 4788)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  77 Thread -1209648208 (LWP 4783)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  75 Thread -1209648208 (LWP 4781)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  67 Thread -1209648208 (LWP 4773)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  65 Thread -1209648208 (LWP 4771)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  57 Thread -1209648208 (LWP 4763)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  55 Thread -1209648208 (LWP 4761)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  47 Thread -1209648208 (LWP 4753)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  45 Thread -1209648208 (LWP 4751)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  37 Thread -1209648208 (LWP 4743)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  35 Thread -1209648208 (LWP 4741)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  27 Thread -1209648208 (LWP 4733)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  25 Thread -1209648208 (LWP 4731)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  17 Thread -1209648208 (LWP 4723)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  15 Thread -1209648208 (LWP 4721)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  11 Thread -1209164880 (LWP 4717)  <function called from gdb>
  9 Thread -1209648208 (LWP 4715)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  7 Thread -1209648208 (LWP 4713)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  5 Thread -1209648208 (LWP 4707)  0x002b8a58 in listLock ()
   from /export/home/hill/epics/R3.14/epics/base/lib/linux-x86/libCom.so.3.14
  4 Thread -1209164880 (LWP 4706)  <function called from gdb>
  1 Thread -1208150336 (LWP 4701)  0x001427a2 in _dl_sysinfo_int80 ()
   from /lib/ld-linux.so.2
(gdb)

Jeff
______________________________________________________
Jeffrey O. Hill           Email        johill@lanl.gov
LANL MS H820              Voice        505 665 1831
Los Alamos NM 87545 USA   FAX          505 665 5107

Message content: TSPA


> -----Original Message-----
> From: core-talk-bounces@aps.anl.gov [mailto:core-talk-bounces@aps.anl.gov]
> On Behalf Of Jeff Hill
> Sent: Tuesday, August 25, 2009 5:46 PM
> To: 'Andrew Johnson'; core-talk@aps.anl.gov
> Subject: RE: proper thread cleanup on Linux?
> 
> 
> As you might have seen in my updated mantis entry I am _not_ reproducing
> with blockingSockTest (which I wrote, but unfortunately neglected to use
> first). After further investigation (on newer Linux with better debug
> symbols) I do see some threads hanging around that look like this - which
> might be a clue. At the moment this isn’t much of a lead but maybe someone
> else knows what this means.
> 
> [Switching to thread 300 (Thread -1208153168 (LWP 5007))]#0  0x00da1841 in
> __nptl_death_event () from /lib/tls/libpthread.so.0
> (gdb) bt
> #0  0x00da1841 in __nptl_death_event () from /lib/tls/libpthread.so.0
> #1  0x00da24b4 in start_thread () from /lib/tls/libpthread.so.0
> #2  0x00c09ffe in clone () from /lib/tls/libc.so.6
> 
> On machines with lots of resources my regression tests pass w/o issue.
> This appears to be one of those silent but deadly, from a performance
> perspective, situations where resources are consumed until the less
> capable system becomes sluggish. That would happen only if circuits come
> and go frequently, and in that context the gateway or a control room medm
> comes to mind.
> 
> Jeff
> ______________________________________________________
> Jeffrey O. Hill           Email        johill@lanl.gov
> LANL MS H820              Voice        505 665 1831
> Los Alamos NM 87545 USA   FAX          505 665 5107
> 
> Message content: TSPA
> 
> 
> > -----Original Message-----
> > From: Andrew Johnson [mailto:anj@aps.anl.gov]
> > Sent: Tuesday, August 25, 2009 4:53 PM
> > To: core-talk@aps.anl.gov
> > Cc: Jeff Hill
> > Subject: Re: proper thread cleanup on Linux?
> >
> > Hi Jeff,
> >
> > On Tuesday 25 August 2009 17:09:14 Jeff Hill wrote:
> > > After further testing on newer linux versions where the debugger seems
> > to
> > > actually function correctly I now strongly suspect that this issue is
> > > caused by this change. The symptom is that two threads that CA creates
> > to
> > > manage TCP circuits never shutdown. They need to shutdown typically
> when
> > > the TCP circuit disconnects or when the last channel on the circuit is
> > > deleted. The bug was introduced in R3.14.9. See mantis 363.
> >
> > Can you write a version of blockingSockTest that demonstrates this and
> > correctly responds with socketSigAlarmRequired?  Unfortunately you'll
> have
> > to
> > test it against R3.14.9 or R3.14.10 — I eviscerated the sigAlarm
> routines
> > for
> > this release since our use of this signal broke the posix timer library
> > (which relies on SIG_ALARM) that some external libraries use.  We'll
> have
> > to
> > switch to using a different signal if we're going to bring it back (we
> > probably should make the signal number we use configurable).
> >
> > - Andrew
> > --
> > The best FOSS code is written to be read by other humans -- Harold Welte