Argonne National Laboratory

Experimental Physics and
Industrial Control System

2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  <20142015  2016  2017  2018  2019  2020  2021  Index 2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  <20142015  2016  2017  2018  2019  2020  2021 
<== Date ==> <== Thread ==>

Subject: 3.15.0.2: Shutdown issues
From: Ralph Lange <Ralph.Lange@gmx.de>
To: EPICS Core-Talk <core-talk@aps.anl.gov>
Date: Fri, 14 Nov 2014 10:16:07 +0100
Hi,

We sometimes see:

Dear Ralph,

         I observe  some errors on   IOC  while exit from the IOC shell with exit command.

 

 

 

 

A call to 'assert(status == epicsMutexLockOK)'

    by thread 'CAS-client' failed in ../../../src/ioc/db/dbEvent.c line 490.

[      0x3817ee8b5d]: /lib64/libc.so.6(clone+0x6d)

EPICS Release EPICS R3.15.0.2 $$Date$$.

 

 

 

A call to 'assert(status == epicsMutexLockOK)'

    by thread 'errlog' failed in ../../../src/libCom/log/logClient.c line 241.

[    0x7f14388c8363]: /opt/codac-5.0/epics/base/lib/linux-x86_64/libCom.so.3.15(epicsStackTrace+0x73)

[    0x7f14388c24ed]: /opt/codac-5.0/epics/base/lib/linux-x86_64/libCom.so.3.15(epicsAssert+0x4d)

[    0x7f1438d87c8a]: /opt/codac-5.0/epics/base/lib/linux-x86_64/libdbCore.so.3.15(db_event_disable+0x7a)

[    0x7f1438d87ca6]: /opt/codac-5.0/epics/base/lib/linux-x86_64/libdbCore.so.3.15(db_cancel_event+0x16)

[    0x7f1438da9c37]: /opt/codac-5.0/epics/base/lib/linux-x86_64/libdbCore.so.3.15(casAttachThreadToClient+0xc7)

[    0x7f1438daa326]: /opt/codac-5.0/epics/base/lib/linux-x86_64/libdbCore.so.3.15(destroy_tcp_client+0x56)

[    0x7f1438dab53a]: /opt/codac-5.0/epics/base/lib/linux-x86_64/libdbCore.so.3.15(camsgtask+0x7a)

[    0x7f14388c3b24]: /opt/codac-5.0/epics/base/lib/linux-x86_64/libCom.so.3.15(epicsThreadSuspendSelf+0x174)

[      0x38186079d1]: /lib64/libpthread.so.0(<no symbol information>)

[      0x3817ee8b5d]: /lib64/libc.so.6(clone+0x6d)

EPICS Release EPICS R3.15.0.2 $$Date$$.

Dumping a stack trace of thread 'errlog':

Local time is 2014-11-12 09:55:17.723540409 UTC

Please E-mail this message to the author or to tech-talk@aps.anl.gov

Calling epicsThreadSuspendSelf()

[    0x7f14388c8363]: /opt/codac-5.0/epics/base/lib/linux-x86_64/libCom.so.3.15(epicsStackTrace+0x73)

[    0x7f14388c24ed]: /opt/codac-5.0/epics/base/lib/linux-x86_64/libCom.so.3.15(epicsAssert+0x4d)

[    0x7f14388b6a1a]: /opt/codac-5.0/epics/base/lib/linux-x86_64/libCom.so.3.15(logClientSend+0x9a)

[    0x7f14388ae3da]: /opt/codac-5.0/epics/base/lib/linux-x86_64/libCom.so.3.15(errlogGetSevEnumString+0x24a)

[    0x7f14388c3b24]: /opt/codac-5.0/epics/base/lib/linux-x86_64/libCom.so.3.15(epicsThreadSuspendSelf+0x174)

[      0x38186079d1]: /lib64/libpthread.so.0(<no symbol information>)

[      0x3817ee8b5d]: /lib64/libc.so.6(clone+0x6d)

EPICS Release EPICS R3.15.0.2 $$Date$$.

Local time is 2014-11-12 09:55:17.723897076 UTC

Please E-mail this message to the author or to tech-talk@aps.anl.gov

Calling epicsThreadSuspendSelf()

 

 

 

 

^C

[screen is terminating]


My current interpretation is:
  • rsrv is shutting down one of its CAS client threads (TCP handler). While removing all subscriptions of that client, being inside db_event_disable() it finds the lockset mutex went away (line 490 of dbEvent.c).
  • The log client tries to send a message to the log server, and finds the client mutex went away (line 241 of logClient.c).
    errlogGetSevEnumString+0x24a is errlogThread(), the call to logSendClient() is via *plistenerNod->listener (line 577 of errlog.c).

I think the two are unrelated. Could it be that the errlog issue is caused by the rsrv issue being logged?

Looking at the interaction between iocLog and the errlog facility, I find that iocLog registers a listener with errlog (line 94 of iocLog.c), but never unregisters. Called as an exit hook, logClientDestroy() destroys the client mutex (line 157 of logClient.c), while the client is still registered as a listener with errlog. That's obviously a bug I will add to the tracker and fix.

Any ideas on what to do about rsrv shutdown?

Thanks a lot,
~Ralph


Replies:
Re: 3.15.0.2: Shutdown issues Michael Davidsaver

Navigate by Date:
Prev: Jenkins build is back to normal : epics-base-3.15-cyg64 #171 APS Jenkins
Next: Re: 3.15.0.2: Shutdown issues Michael Davidsaver
Index: 2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  <20142015  2016  2017  2018  2019  2020  2021 
Navigate by Thread:
Prev: Jenkins build is back to normal : epics-base-3.15-cyg64 #171 APS Jenkins
Next: Re: 3.15.0.2: Shutdown issues Michael Davidsaver
Index: 2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  <20142015  2016  2017  2018  2019  2020  2021 
ANJ, 14 Nov 2014 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·