RE: EPICS stopped

["Tagger, Jueri via Tech-talk" <tech-talk at aps.anl.gov>]

Another occasion. Looks like it is beginning to occur more often(?);  Completely random, no correlation to any IOC or EPICS version:

A call to 'assert(ev_que->evque[ev_que->putix] == EVENTQEMPTY)'
    by thread 'CAS-client' failed in ../../../src/ioc/db/dbEvent.c line 769.
Dumping a stack trace of thread 'CAS-client':
[ 0x81a4673]: ../../bin/linux-x86/srVaC(epicsBackTrace+0x23)
[ 0x81a44d6]: ../../bin/linux-x86/srVaC(epicsStackTrace+0x66)
[ 0x819e9f2]: ../../bin/linux-x86/srVaC(epicsAssert+0x62)
[ 0x812bc60]: ../../bin/linux-x86/srVaC(db_queue_event_log+0x100)
[ 0x812bdc0]: ../../bin/linux-x86/srVaC(db_post_single_event+0x50)
[ 0x8151863]: ../../bin/linux-x86/srVaC(event_add_action+0x213)
[ 0x8152ece]: ../../bin/linux-x86/srVaC(camessage+0x9e)
[ 0x814f0db]: ../../bin/linux-x86/srVaC(camsgtask+0x16b)
[ 0x819fdd6]: ../../bin/linux-x86/srVaC(start_routine+0x106)
[0xb76f2c39]: /lib/i386-linux-gnu/i686/cmov/libpthread.so.0(start_thread+0xc9)
[0xb74e4c6e]: /lib/i386-linux-gnu/i686/cmov/libc.so.6(clone+0x5e)
EPICS Release EPICS R3.15.5.
Local time is 2021-06-02 12:44:59.256136892 EDT
Please E-mail this message to the author or to tech-talk at aps.anl.gov
Calling epicsThreadSuspendSelf()
Thread CAS-client (0xb3b00480) suspended
callbackRequest: cbLow ring buffer full
dbCa:exceptionCallback stat "Virtual circuit unresponsive" channel "unknown" context "1wire-cr-rga.cs.nsls2.local:5064"
 nativeType DBR_invalid requestType DBR_invalid nativeCount 0 requestCount 0 noReadAccess noWriteAccess


-----Original Message-----
From: Michael Davidsaver <mdavidsaver at gmail.com> 
Sent: Thursday, May 20, 2021 1:36 PM
To: Hu, Yong <yhu at bnl.gov>; Tagger, Jueri <jtagger at bnl.gov>
Cc: tech-talk at aps.anl.gov
Subject: Re: EPICS stopped

On 5/20/21 8:38 AM, Hu, Yong wrote:
> Juri,
> 
> It seems the same problem happened twice and the first incident happened last May. 
> Try Mark's suggestion: 
> https://epics.anl.gov/tech-talk/2020/msg00895.php

fyi. I think "ring buffer full" is a symptom, and that increasing the size won't help.

I think the underlying cause is a race condition arising from incomplete locking of the monitor queue data structures in RSRV related to an access security rule recompute being triggered by a dynamic rule.  I'm not immediately certain what to do about this though.  Blindly adding locking could easily lead to a deadlock.
I've made some notes on https://urldefense.com/v3/__https://bugs.launchpad.net/epics-base/*bug/541371__;Kw!!P4SdNyxKAPE!Xy3JRw-XRk-UOPyuOPG1QKdbUKgl5UXRmS_mtf79pvmflO6bzAGZjaDdS0gMg00$  .

I don't expect to have time to look into this further during the next few weeks.


> Yong
> 
> On 5/19/21, 3:55 PM, "Tech-talk on behalf of Tagger, Jueri via Tech-talk" <tech-talk-bounces at aps.anl.gov on behalf of tech-talk at aps.anl.gov> wrote:
> 
>     Indeed! Found it:
> 
>     A call to 'assert(ev_que->evque[ev_que->putix] == EVENTQEMPTY)'
>         by thread 'CAS-client' failed in ../../../src/ioc/db/dbEvent.c line 781.
>     Dumping a stack trace of thread 'CAS-client':
>     [    0x7f3a04a9d4bb]: /home/jtagger/EPICS/R3_15_8-x86_64/base-3.15.8/lib/linux-x86_64/libCom.so.3.15.8(epicsStackTrace+0x4b)
>     [    0x7f3a04a96c2a]: /home/jtagger/EPICS/R3_15_8-x86_64/base-3.15.8/lib/linux-x86_64/libCom.so.3.15.8(epicsAssert+0x3a)
>     [    0x7f3a04f5fedb]: /home/jtagger/EPICS/R3_15_8-x86_64/base-3.15.8/lib/linux-x86_64/libdbCore.so.3.15.8(db_queue_event_log+0xcb)
>     [    0x7f3a04f6012d]: /home/jtagger/EPICS/R3_15_8-x86_64/base-3.15.8/lib/linux-x86_64/libdbCore.so.3.15.8(db_post_single_event+0x3d)
>     [    0x7f3a04f8a1e1]: /home/jtagger/EPICS/R3_15_8-x86_64/base-3.15.8/lib/linux-x86_64/libdbCore.so.3.15.8(event_add_action+0x1c1)
>     [    0x7f3a04f8b72e]: /home/jtagger/EPICS/R3_15_8-x86_64/base-3.15.8/lib/linux-x86_64/libdbCore.so.3.15.8(camessage+0x9e)
>     [    0x7f3a04f881e3]: /home/jtagger/EPICS/R3_15_8-x86_64/base-3.15.8/lib/linux-x86_64/libdbCore.so.3.15.8(camsgtask+0xc3)
>     [    0x7f3a04a97d0c]: /home/jtagger/EPICS/R3_15_8-x86_64/base-3.15.8/lib/linux-x86_64/libCom.so.3.15.8(start_routine+0xdc)
>     [    0x7f3a03c72064]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xc4)
>     [    0x7f3a03f6f62d]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)
>     EPICS Release EPICS R3.15.8.
>     Local time is 2021-05-19 08:18:56.245430055 EDT
>     Please E-mail this message to the author or to tech-talk at aps.anl.gov
>     Calling epicsThreadSuspendSelf()
>     Thread CAS-client (0x7f3978023160) suspended
>     callbackRequest: cbLow ring buffer full
>     epics> date
>     2021/05/19 11:05:33.287078
>