EPICS RE: Problems with stream device 2.8.8

Experimental Physics and Industrial Control System

1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 <2019> 2020 2021 2022 2023 2024	Index	1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 <2019> 2020 2021 2022 2023 2024
<== Date ==>		<== Thread ==>

1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 <2019> 2020 2021 2022 2023 2024

Index

1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 <2019> 2020 2021 2022 2023 2024

<== Date ==>

<== Thread ==>

Subject:

RE: Problems with stream device 2.8.8

From:

Mark Rivers via Tech-talk <[email protected]>

To:

"'Johnson, Andrew N.'" <[email protected]>

Cc:

"[email protected]" <[email protected]>

Date:

Mon, 8 Jul 2019 17:42:05 +0000

I have been communicating with Abdalla about this offline. He sent the attached gdb output when trying to exit.

This is from my last message to him:

It looks like the problem is these 2 threads:

Thread 5 (Thread 0x7fffedfe9700 (LWP 30806)):

#0 0x00007ffff5be38ed in connect () from /lib64/libc.so.6

#1 0x00007ffff76dddb4 in connectIt (pasynUser=0x713308, drvPvt=0x71def0) at ../../asyn/drvAsynSerial/drvAsynIPPort.c:476

#2 asynCommonConnect (drvPvt=0x71def0, pasynUser=0x713308) at ../../asyn/drvAsynSerial/drvAsynIPPort.c:520

#3 0x00007ffff76d08fb in portConnectProcessCallback (pasynUser=0x713308) at ../../asyn/asynDriver/asynManager.c:3076

#4 0x00007ffff76d39c7 in portThread (pport=0x711ed0) at ../../asyn/asynDriver/asynManager.c:820

#5 0x00007ffff67128ec in start_routine (arg=0x712f10) at ../../../src/libCom/osi/os/posix/osdThread.c:403

#6 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0

#7 0x00007ffff5be2bad in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7ffff7fd9740 (LWP 30798)):

#0 0x00007ffff569f51d in __lll_lock_wait () from /lib64/libpthread.so.0

#1 0x00007ffff569ae36 in _L_lock_870 () from /lib64/libpthread.so.0

#2 0x00007ffff569ad2f in pthread_mutex_lock () from /lib64/libpthread.so.0

#3 0x00007ffff6714b66 in mutexLock (id=0x712150) at ../../../src/libCom/osi/os/posix/osdMutex.c:46

#4 epicsMutexOsdLock (pmutex=0x712150) at ../../../src/libCom/osi/os/posix/osdMutex.c:130

#5 0x00007ffff76cf38b in lockPort (pasynUser=0x714d08) at ../../asyn/asynDriver/asynManager.c:1741

#6 0x00007ffff76dcd66 in cleanup (arg=0x71def0) at ../../asyn/drvAsynSerial/drvAsynIPPort.c:246

#7 0x00007ffff6708dd3 in epicsExitCallAtExitsPvt (pep=<optimized out>) at ../../../src/libCom/misc/epicsExit.c:95

#8 epicsExitCallAtExits () at ../../../src/libCom/misc/epicsExit.c:113

#9 0x00007ffff6709178 in epicsExit (status=0) at ../../../src/libCom/misc/epicsExit.c:181

#10 0x0000000000405d0d in main (argc=<optimized out>, argv=<optimized out>) at ../srMain.cpp:21

Thread 1 is drvAsynIPPort trying to close the socket. It is hanging on the call to pasynManager->lockPort(), which tries to lock pport->synchronousLock. The reason is that thread 5 is the asyn port thread trying to connect to the device, and it has the mutex pport->synchronousLock locked. I suspect that thread 5 is trying to connect frequently because many records are talking to the device, and the device has Autoconnect=Yes. If that thread never unlocks the mutex for long then this problem will occur.

I need to try to reproduce this and see if there is a solution.

Next time it happens before you type "exit" please type "epicsMutexShowAll 1".

Mark

From: [email protected] <[email protected]> On Behalf Of Johnson, Andrew N. via Tech-talk
Sent: Monday, July 8, 2019 11:50 AM
To: [email protected]
Subject: Re: Problems with stream device 2.8.8

Hi Abdalla,

On 7/7/19 5:46 AM, Abdalla Ahmad via Tech-talk wrote:

Exit command renders the IOC shell frozen, i.e. the IOC does not exit.

We faced #2 before, but I want to ask is it asyn or stream device problem? Because I want to try different versions but I’m not sure whether asyn or stream to change.

We can't answer that, but you might get some more hints by running the command
var atExitDebug 1
on the IOC shell before exiting. It causes a message to be printed before each of the registered epicsAtExit() routines gets run, so you should be able to see which routine is hanging.

- Andrew

--

Complexity comes for free, Simplicity you have to work for.

Program received signal SIGINT, Interrupt. 0x00007ffff569f51d in __lll_lock_wait () from /lib64/libpthread.so.0 Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.170-4.el7.x86_64 elfutils-libs-0.170-4.el7.x86_64 glibc-2.17-222.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libcap-2.22-9.el7.x86_64 libgcc-4.8.5-28.el7_5.1.x86_64 libstdc++-4.8.5-28.el7_5.1.x86_64 ncurses-libs-5.9-14.20130511.el7_4.x86_64 readline-6.2-10.el7.x86_64 sssd-client-1.16.0-19.el7_5.5.x86_64 systemd-libs-219-57.el7_5.1.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64 Thread 58 (Thread 0x7fffc70f5700 (LWP 30860)): #0 0x00007ffff5be3a5b in recv () from /lib64/libc.so.6 #1 0x00007ffff6c0663a in camsgtask (pParm=0x7c9958) at ../../../src/ioc/rsrv/camsgtask.c:72 #2 0x00007ffff67128ec in start_routine (arg=0x7ce6c0) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #3 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 57 (Thread 0x7fffc71f6700 (LWP 30859)): #0 0x00007ffff569c995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff671502b in epicsEventWait (pevent=0x7ce170) at ../../../src/libCom/osi/os/posix/osdEvent.c:103 #2 0x00007ffff670e119 in epicsEventMustWait (id=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:125 #3 0x00007ffff6bdfc2c in event_task (pParm=0x7cd850) at ../../../src/ioc/db/dbEvent.c:1007 #4 0x00007ffff67128ec in start_routine (arg=0x7ce330) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #5 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 56 (Thread 0x7fffc73f7700 (LWP 30858)): #0 0x00007ffff5be3a5b in recv () from /lib64/libc.so.6 #1 0x00007ffff6c0663a in camsgtask (pParm=0x7fffc8000d50) at ../../../src/ioc/rsrv/camsgtask.c:72 #2 0x00007ffff67128ec in start_routine (arg=0x7c8e20) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #3 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 55 (Thread 0x7fffc74f8700 (LWP 30857)): #0 0x00007ffff569c995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff671502b in epicsEventWait (pevent=0x7c88d0) at ../../../src/libCom/osi/os/posix/osdEvent.c:103 #2 0x00007ffff670e119 in epicsEventMustWait (id=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:125 #3 0x00007ffff6bdfc2c in event_task (pParm=0x7fffd002e050) at ../../../src/ioc/db/dbEvent.c:1007 #4 0x00007ffff67128ec in start_routine (arg=0x7c8a90) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #5 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 54 (Thread 0x7fffc76f9700 (LWP 30855)): #0 0x00007ffff5be3a5b in recv () from /lib64/libc.so.6 #1 0x00007ffff6c0663a in camsgtask (pParm=0x7fffc8000e68) at ../../../src/ioc/rsrv/camsgtask.c:72 #2 0x00007ffff67128ec in start_routine (arg=0x7c8260) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #3 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 53 (Thread 0x7fffc77fa700 (LWP 30854)): #0 0x00007ffff569c995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff671502b in epicsEventWait (pevent=0x7c7d10) at ../../../src/libCom/osi/os/posix/osdEvent.c:103 #2 0x00007ffff670e119 in epicsEventMustWait (id=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:125 #3 0x00007ffff6bdfc2c in event_task (pParm=0x7fffd002e8d0) at ../../../src/ioc/db/dbEvent.c:1007 #4 0x00007ffff67128ec in start_routine (arg=0x7c7ed0) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #5 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 52 (Thread 0x7fffc79fb700 (LWP 30853)): #0 0x00007ffff5be3a5b in recv () from /lib64/libc.so.6 #1 0x00007ffff6c0663a in camsgtask (pParm=0x7fffc8000f80) at ../../../src/ioc/rsrv/camsgtask.c:72 #2 0x00007ffff67128ec in start_routine (arg=0x7c76e0) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #3 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 51 (Thread 0x7fffc7afc700 (LWP 30852)): #0 0x00007ffff569c995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff671502b in epicsEventWait (pevent=0x7c7190) at ../../../src/libCom/osi/os/posix/osdEvent.c:103 #2 0x00007ffff670e119 in epicsEventMustWait (id=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:125 #3 0x00007ffff6bdfc2c in event_task (pParm=0x7fffd002f150) at ../../../src/ioc/db/dbEvent.c:1007 #4 0x00007ffff67128ec in start_routine (arg=0x7c7350) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #5 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 50 (Thread 0x7fffc7cfd700 (LWP 30851)): #0 0x00007ffff5be3a5b in recv () from /lib64/libc.so.6 #1 0x00007ffff6c0663a in camsgtask (pParm=0x7fffc8001098) at ../../../src/ioc/rsrv/camsgtask.c:72 #2 0x00007ffff67128ec in start_routine (arg=0x7c6b60) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #3 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 49 (Thread 0x7fffc7dfe700 (LWP 30850)): #0 0x00007ffff569c995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff671502b in epicsEventWait (pevent=0x7c6610) at ../../../src/libCom/osi/os/posix/osdEvent.c:103 #2 0x00007ffff670e119 in epicsEventMustWait (id=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:125 #3 0x00007ffff6bdfc2c in event_task (pParm=0x7fffd002f9d0) at ../../../src/ioc/db/dbEvent.c:1007 #4 0x00007ffff67128ec in start_routine (arg=0x7c67d0) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #5 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 48 (Thread 0x7fffc7fff700 (LWP 30849)): #0 0x00007ffff5be3a5b in recv () from /lib64/libc.so.6 #1 0x00007ffff6c0663a in camsgtask (pParm=0x7fffc80011b0) at ../../../src/ioc/rsrv/camsgtask.c:72 #2 0x00007ffff67128ec in start_routine (arg=0x7c5fe0) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #3 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 47 (Thread 0x7fffcc1cd700 (LWP 30848)): #0 0x00007ffff569c995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff671502b in epicsEventWait (pevent=0x7c5a90) at ../../../src/libCom/osi/os/posix/osdEvent.c:103 #2 0x00007ffff670e119 in epicsEventMustWait (id=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:125 #3 0x00007ffff6bdfc2c in event_task (pParm=0x7fffd0030250) at ../../../src/ioc/db/dbEvent.c:1007 #4 0x00007ffff67128ec in start_routine (arg=0x7c5c50) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #5 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 46 (Thread 0x7fffcc3ce700 (LWP 30847)): #0 0x00007ffff5be3a5b in recv () from /lib64/libc.so.6 #1 0x00007ffff6c0663a in camsgtask (pParm=0x7fffc80012c8) at ../../../src/ioc/rsrv/camsgtask.c:72 #2 0x00007ffff67128ec in start_routine (arg=0x7c5420) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #3 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 45 (Thread 0x7fffcc4cf700 (LWP 30846)): #0 0x00007ffff569c995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff671502b in epicsEventWait (pevent=0x7c4ed0) at ../../../src/libCom/osi/os/posix/osdEvent.c:103 #2 0x00007ffff670e119 in epicsEventMustWait (id=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:125 #3 0x00007ffff6bdfc2c in event_task (pParm=0x7fffd0030ad0) at ../../../src/ioc/db/dbEvent.c:1007 #4 0x00007ffff67128ec in start_routine (arg=0x7c5090) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #5 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 44 (Thread 0x7fffcc6d0700 (LWP 30845)): #0 0x00007ffff5be3a5b in recv () from /lib64/libc.so.6 #1 0x00007ffff6c0663a in camsgtask (pParm=0x7fffc80013e0) at ../../../src/ioc/rsrv/camsgtask.c:72 #2 0x00007ffff67128ec in start_routine (arg=0x62d820) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #3 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7fffcc7d1700 (LWP 30844)): #0 0x00007ffff569c995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff671502b in epicsEventWait (pevent=0x7c3ad0) at ../../../src/libCom/osi/os/posix/osdEvent.c:103 #2 0x00007ffff670e119 in epicsEventMustWait (id=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:125 #3 0x00007ffff6bdfc2c in event_task (pParm=0x7fffd0031350) at ../../../src/ioc/db/dbEvent.c:1007 #4 0x00007ffff67128ec in start_routine (arg=0x62d700) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #5 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fffcf477700 (LWP 30832)): #0 0x00007ffff569c995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff671502b in epicsEventWait (pevent=0x7fffd00324f0) at ../../../src/libCom/osi/os/posix/osdEvent.c:103 #2 0x00007ffff670e119 in epicsEventMustWait (id=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:125 #3 0x00007ffff6bdfc2c in event_task (pParm=0x7fffd0031bd0) at ../../../src/ioc/db/dbEvent.c:1007 #4 0x00007ffff67128ec in start_routine (arg=0x7fffd00326b0) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #5 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fffcf779700 (LWP 30830)): #0 0x00007ffff569c995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff671502b in epicsEventWait (pevent=0x7fffc8020750) at ../../../src/libCom/osi/os/posix/osdEvent.c:103 #2 0x00007ffff670df9c in epicsEvent::wait (this=this@entry=0x7fffc8020600) at ../../../src/libCom/osi/epicsEvent.cpp:66 #3 0x00007ffff670ad92 in (anonymous namespace)::ipAddrToAsciiGlobal::run (this=0x7fffc80201c0) at ../../../src/libCom/misc/ipAddrToAsciiAsynchronous.cpp:304 #4 0x00007ffff670cce9 in epicsThreadCallEntryPoint (pPvt=0x7fffc8020610) at ../../../src/libCom/osi/epicsThread.cpp:83 #5 0x00007ffff67128ec in start_routine (arg=0x7fffc8020980) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #6 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fffcf9fb700 (LWP 30828)): #0 0x00007ffff5ba956d in nanosleep () from /lib64/libc.so.6 #1 0x00007ffff6713918 in epicsThreadSleep (seconds=seconds@entry=10.24) at ../../../src/libCom/osi/os/posix/osdThread.c:715 #2 0x00007ffff6c0bc1b in rsrv_online_notify_task (pParm=<optimized out>) at ../../../src/ioc/rsrv/online_notify.c:99 #3 0x00007ffff67128ec in start_routine (arg=0x7c1440) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #4 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fffcfafc700 (LWP 30827)): #0 0x00007ffff5be3ac3 in recvfrom () from /lib64/libc.so.6 #1 0x00007ffff6c0b7e6 in cast_server (pParm=<optimized out>) at ../../../src/ioc/rsrv/cast_server.c:167 #2 0x00007ffff67128ec in start_routine (arg=0x7b6800) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #3 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fffcfbfd700 (LWP 30826)): #0 0x00007ffff5be385d in accept () from /lib64/libc.so.6 #1 0x00007ffff671119b in epicsSocketAccept (sock=sock@entry=14, pAddr=pAddr@entry=0x7fffcfbfcdf0, addrlen=addrlen@entry=0x7fffcfbfcde0) at ../../../src/libCom/osi/os/posix/osdSock.c:98 #2 0x00007ffff6c06452 in req_server (pParm=<optimized out>) at ../../../src/ioc/rsrv/caservertask.c:91 #3 0x00007ffff67128ec in start_routine (arg=0x7b66b0) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #4 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fffcfdfe700 (LWP 30825)): #0 0x00007ffff569c995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff671502b in epicsEventWait (pevent=0x7b5920) at ../../../src/libCom/osi/os/posix/osdEvent.c:103 #2 0x00007ffff670e119 in epicsEventMustWait (id=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:125 #3 0x00007ffff6bcae65 in asCaTask () at ../../../src/ioc/as/asCa.c:208 #4 0x00007ffff67128ec in start_routine (arg=0x7b5990) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #5 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fffcffff700 (LWP 30824)): #0 0x00007ffff569cd42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff67151a9 in epicsEventWaitWithTimeout (pevent=0x7a17a0, timeout=0.099923820999999996) at ../../../src/libCom/osi/os/posix/osdEvent.c:130 #2 0x00007ffff6bdcb1b in periodicTask (arg=0x7a16d0) at ../../../src/ioc/db/dbScan.c:759 #3 0x00007ffff67128ec in start_routine (arg=0x79fc60) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #4 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fffec227700 (LWP 30823)): #0 0x00007ffff569cd42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff67151a9 in epicsEventWaitWithTimeout (pevent=0x7a1660, timeout=0.19986989499999999) at ../../../src/libCom/osi/os/posix/osdEvent.c:130 #2 0x00007ffff6bdcb1b in periodicTask (arg=0x7a1590) at ../../../src/ioc/db/dbScan.c:759 #3 0x00007ffff67128ec in start_routine (arg=0x79f900) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #4 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fffec428700 (LWP 30822)): #0 0x00007ffff569cd42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff67151a9 in epicsEventWaitWithTimeout (pevent=0x7a1520, timeout=0.49992281999999999) at ../../../src/libCom/osi/os/posix/osdEvent.c:130 #2 0x00007ffff6bdcb1b in periodicTask (arg=0x7a1450) at ../../../src/ioc/db/dbScan.c:759 #3 0x00007ffff67128ec in start_routine (arg=0x79f5a0) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #4 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fffec629700 (LWP 30821)): #0 0x00007ffff569cd42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff67151a9 in epicsEventWaitWithTimeout (pevent=0x7a13e0, timeout=0.99982646900000005) at ../../../src/libCom/osi/os/posix/osdEvent.c:130 #2 0x00007ffff6bdcb1b in periodicTask (arg=0x79ea10) at ../../../src/ioc/db/dbScan.c:759 #3 0x00007ffff67128ec in start_routine (arg=0x79f240) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #4 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fffec82a700 (LWP 30820)): #0 0x00007ffff569cd42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff67151a9 in epicsEventWaitWithTimeout (pevent=0x79e9a0, timeout=1.9998809209999999) at ../../../src/libCom/osi/os/posix/osdEvent.c:130 #2 0x00007ffff6bdcb1b in periodicTask (arg=0x79e8d0) at ../../../src/ioc/db/dbScan.c:759 #3 0x00007ffff67128ec in start_routine (arg=0x79eee0) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #4 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fffeca2b700 (LWP 30819)): #0 0x00007ffff569cd42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff67151a9 in epicsEventWaitWithTimeout (pevent=0x79e860, timeout=4.9999342770000004) at ../../../src/libCom/osi/os/posix/osdEvent.c:130 #2 0x00007ffff6bdcb1b in periodicTask (arg=0x79e790) at ../../../src/ioc/db/dbScan.c:759 #3 0x00007ffff67128ec in start_routine (arg=0x79eb80) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #4 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fffecc2c700 (LWP 30818)): #0 0x00007ffff569cd42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff67151a9 in epicsEventWaitWithTimeout (pevent=0x79e720, timeout=9.9999356739999996) at ../../../src/libCom/osi/os/posix/osdEvent.c:130 #2 0x00007ffff6bdcb1b in periodicTask (arg=0x79e650) at ../../../src/ioc/db/dbScan.c:759 #3 0x00007ffff67128ec in start_routine (arg=0x7a4050) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #4 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fffece2d700 (LWP 30817)): #0 0x00007ffff569c995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff671502b in epicsEventWait (pevent=0x7a1840) at ../../../src/libCom/osi/os/posix/osdEvent.c:103 #2 0x00007ffff670e119 in epicsEventMustWait (id=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:125 #3 0x00007ffff6bdc9f0 in onceTask (arg=<optimized out>) at ../../../src/ioc/db/dbScan.c:661 #4 0x00007ffff67128ec in start_routine (arg=0x7a18b0) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #5 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fffecf2e700 (LWP 30816)): #0 0x00007ffff569cd42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff67151a9 in epicsEventWaitWithTimeout (pevent=0x79d160, timeout=1.7976931348623157e+308) at ../../../src/libCom/osi/os/posix/osdEvent.c:130 #2 0x00007ffff670dfec in epicsEvent::wait (this=this@entry=0x79cf30, timeOut=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:75 #3 0x00007ffff671d7df in timerQueueActive::run (this=0x79ce70) at ../../../src/libCom/timer/timerQueueActive.cpp:95 #4 0x00007ffff670cce9 in epicsThreadCallEntryPoint (pPvt=0x79cf40) at ../../../src/libCom/osi/epicsThread.cpp:83 #5 0x00007ffff67128ec in start_routine (arg=0x79d390) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #6 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fffed02f700 (LWP 30815)): #0 0x00007ffff569cd42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff67151a9 in epicsEventWaitWithTimeout (pevent=0x77a530, timeout=9.9802927819999994) at ../../../src/libCom/osi/os/posix/osdEvent.c:130 #2 0x00007ffff670dfec in epicsEvent::wait (this=this@entry=0x77a300, timeOut=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:75 #3 0x00007ffff671d7df in timerQueueActive::run (this=0x77a240) at ../../../src/libCom/timer/timerQueueActive.cpp:95 #4 0x00007ffff670cce9 in epicsThreadCallEntryPoint (pPvt=0x77a310) at ../../../src/libCom/osi/epicsThread.cpp:83 #5 0x00007ffff67128ec in start_routine (arg=0x77a760) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #6 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fffed431700 (LWP 30813)): #0 0x00007ffff569c995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff671502b in epicsEventWait (pevent=0x775720) at ../../../src/libCom/osi/os/posix/osdEvent.c:103 #2 0x00007ffff670e119 in epicsEventMustWait (id=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:125 #3 0x00007ffff6be66ce in callbackTask (arg=<optimized out>) at ../../../src/ioc/db/callback.c:168 #4 0x00007ffff67128ec in start_routine (arg=0x779680) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #5 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fffed632700 (LWP 30812)): #0 0x00007ffff569c995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff671502b in epicsEventWait (pevent=0x771460) at ../../../src/libCom/osi/os/posix/osdEvent.c:103 #2 0x00007ffff670e119 in epicsEventMustWait (id=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:125 #3 0x00007ffff6be66ce in callbackTask (arg=<optimized out>) at ../../../src/ioc/db/callback.c:168 #4 0x00007ffff67128ec in start_routine (arg=0x7753c0) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #5 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7ffff4e14700 (LWP 30811)): #0 0x00007ffff569c995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff671502b in epicsEventWait (pevent=0x76d3f0) at ../../../src/libCom/osi/os/posix/osdEvent.c:103 #2 0x00007ffff670e119 in epicsEventMustWait (id=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:125 #3 0x00007ffff6be66ce in callbackTask (arg=<optimized out>) at ../../../src/ioc/db/callback.c:168 #4 0x00007ffff67128ec in start_routine (arg=0x771350) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #5 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fffed733700 (LWP 30810)): #0 0x00007ffff569cd42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff67151a9 in epicsEventWaitWithTimeout (pevent=0x76ce50, timeout=1.7976931348623157e+308) at ../../../src/libCom/osi/os/posix/osdEvent.c:130 #2 0x00007ffff670dfec in epicsEvent::wait (this=this@entry=0x76cc50, timeOut=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:75 #3 0x00007ffff671d7df in timerQueueActive::run (this=0x76cb90) at ../../../src/libCom/timer/timerQueueActive.cpp:95 #4 0x00007ffff670cce9 in epicsThreadCallEntryPoint (pPvt=0x76cc60) at ../../../src/libCom/osi/epicsThread.cpp:83 #5 0x00007ffff67128ec in start_routine (arg=0x76d080) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #6 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fffed8b5700 (LWP 30808)): #0 0x00007ffff569c995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff671502b in epicsEventWait (pevent=0x715670) at ../../../src/libCom/osi/os/posix/osdEvent.c:103 #2 0x00007ffff670e119 in epicsEventMustWait (id=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:125 #3 0x00007ffff76d38fc in portThread (pport=0x7151a0) at ../../asyn/asynDriver/asynManager.c:788 #4 0x00007ffff67128ec in start_routine (arg=0x7156e0) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #5 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7ffff7ed6700 (LWP 30807)): #0 0x00007ffff569cd42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff67151a9 in epicsEventWaitWithTimeout (pevent=0x7fffe0000a10, timeout=timeout@entry=6) at ../../../src/libCom/osi/os/posix/osdEvent.c:130 #2 0x00007ffff671ada4 in twdTask (arg=<optimized out>) at ../../../src/libCom/taskwd/taskwd.c:125 #3 0x00007ffff67128ec in start_routine (arg=0x7fffe0000af0) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #4 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fffedfe9700 (LWP 30806)): #0 0x00007ffff5be38ed in connect () from /lib64/libc.so.6 #1 0x00007ffff76dddb4 in connectIt (pasynUser=0x713308, drvPvt=0x71def0) at ../../asyn/drvAsynSerial/drvAsynIPPort.c:476 #2 asynCommonConnect (drvPvt=0x71def0, pasynUser=0x713308) at ../../asyn/drvAsynSerial/drvAsynIPPort.c:520 #3 0x00007ffff76d08fb in portConnectProcessCallback (pasynUser=0x713308) at ../../asyn/asynDriver/asynManager.c:3076 #4 0x00007ffff76d39c7 in portThread (pport=0x711ed0) at ../../asyn/asynDriver/asynManager.c:820 #5 0x00007ffff67128ec in start_routine (arg=0x712f10) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #6 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fffee0ea700 (LWP 30805)): #0 0x00007ffff569cd42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff67151a9 in epicsEventWaitWithTimeout (pevent=0x711930, timeout=1.7976931348623157e+308) at ../../../src/libCom/osi/os/posix/osdEvent.c:130 #2 0x00007ffff670dfec in epicsEvent::wait (this=this@entry=0x754940, timeOut=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:75 #3 0x00007ffff671d7df in timerQueueActive::run (this=0x754880) at ../../../src/libCom/timer/timerQueueActive.cpp:95 #4 0x00007ffff670cce9 in epicsThreadCallEntryPoint (pPvt=0x754950) at ../../../src/libCom/osi/epicsThread.cpp:83 #5 0x00007ffff67128ec in start_routine (arg=0x711b60) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #6 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7ffff7fd7700 (LWP 30803)): #0 0x00007ffff569cd42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff67151a9 in epicsEventWaitWithTimeout (pevent=0x754200, timeout=1.7976931348623157e+308) at ../../../src/libCom/osi/os/posix/osdEvent.c:130 #2 0x00007ffff670dfec in epicsEvent::wait (this=this@entry=0x71e2e0, timeOut=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:75 #3 0x00007ffff671d7df in timerQueueActive::run (this=0x71e220) at ../../../src/libCom/timer/timerQueueActive.cpp:95 #4 0x00007ffff670cce9 in epicsThreadCallEntryPoint (pPvt=0x71e2f0) at ../../../src/libCom/osi/epicsThread.cpp:83 #5 0x00007ffff67128ec in start_routine (arg=0x754430) at ../../../src/libCom/osi/os/posix/osdThread.c:403 #6 0x00007ffff5698e25 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ffff5be2bad in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7ffff7fd9740 (LWP 30798)): #0 0x00007ffff569f51d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007ffff569ae36 in _L_lock_870 () from /lib64/libpthread.so.0 #2 0x00007ffff569ad2f in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007ffff6714b66 in mutexLock (id=0x712150) at ../../../src/libCom/osi/os/posix/osdMutex.c:46 #4 epicsMutexOsdLock (pmutex=0x712150) at ../../../src/libCom/osi/os/posix/osdMutex.c:130 #5 0x00007ffff76cf38b in lockPort (pasynUser=0x714d08) at ../../asyn/asynDriver/asynManager.c:1741 #6 0x00007ffff76dcd66 in cleanup (arg=0x71def0) at ../../asyn/drvAsynSerial/drvAsynIPPort.c:246 #7 0x00007ffff6708dd3 in epicsExitCallAtExitsPvt (pep=<optimized out>) at ../../../src/libCom/misc/epicsExit.c:95 #8 epicsExitCallAtExits () at ../../../src/libCom/misc/epicsExit.c:113 #9 0x00007ffff6709178 in epicsExit (status=0) at ../../../src/libCom/misc/epicsExit.c:181 #10 0x0000000000405d0d in main (argc=<optimized out>, argv=<optimized out>) at ../srMain.cpp:21 A debugging session is active. Inferior 1 [process 30798] will be killed. Quit anyway? (y or n)