Experimental Physics and
Industrial Control System

Michael Davidsaver via Tech-talk <tech-talk at aps.anl.gov> · Thu, 20 May 2021 10:36:17 -0700

On 5/20/21 8:38 AM, Hu, Yong wrote:
> Juri, 
> 
> It seems the same problem happened twice and the first incident happened last May. 
> Try Mark's suggestion: https://epics.anl.gov/tech-talk/2020/msg00895.php 

fyi. I think "ring buffer full" is a symptom, and that increasing the size won't help.

I think the underlying cause is a race condition arising from incomplete locking
of the monitor queue data structures in RSRV related to an access security rule
recompute being triggered by a dynamic rule.  I'm not immediately certain what to
do about this though.  Blindly adding locking could easily lead to a deadlock.
I've made some notes on https://bugs.launchpad.net/epics-base/+bug/541371 .

I don't expect to have time to look into this further during the next few weeks.

> Yong
> 
> On 5/19/21, 3:55 PM, "Tech-talk on behalf of Tagger, Jueri via Tech-talk" <tech-talk-bounces at aps.anl.gov on behalf of tech-talk at aps.anl.gov> wrote:
> 
>     Indeed! Found it:
> 
>     A call to 'assert(ev_que->evque[ev_que->putix] == EVENTQEMPTY)'
>         by thread 'CAS-client' failed in ../../../src/ioc/db/dbEvent.c line 781.
>     Dumping a stack trace of thread 'CAS-client':
>     [    0x7f3a04a9d4bb]: /home/jtagger/EPICS/R3_15_8-x86_64/base-3.15.8/lib/linux-x86_64/libCom.so.3.15.8(epicsStackTrace+0x4b)
>     [    0x7f3a04a96c2a]: /home/jtagger/EPICS/R3_15_8-x86_64/base-3.15.8/lib/linux-x86_64/libCom.so.3.15.8(epicsAssert+0x3a)
>     [    0x7f3a04f5fedb]: /home/jtagger/EPICS/R3_15_8-x86_64/base-3.15.8/lib/linux-x86_64/libdbCore.so.3.15.8(db_queue_event_log+0xcb)
>     [    0x7f3a04f6012d]: /home/jtagger/EPICS/R3_15_8-x86_64/base-3.15.8/lib/linux-x86_64/libdbCore.so.3.15.8(db_post_single_event+0x3d)
>     [    0x7f3a04f8a1e1]: /home/jtagger/EPICS/R3_15_8-x86_64/base-3.15.8/lib/linux-x86_64/libdbCore.so.3.15.8(event_add_action+0x1c1)
>     [    0x7f3a04f8b72e]: /home/jtagger/EPICS/R3_15_8-x86_64/base-3.15.8/lib/linux-x86_64/libdbCore.so.3.15.8(camessage+0x9e)
>     [    0x7f3a04f881e3]: /home/jtagger/EPICS/R3_15_8-x86_64/base-3.15.8/lib/linux-x86_64/libdbCore.so.3.15.8(camsgtask+0xc3)
>     [    0x7f3a04a97d0c]: /home/jtagger/EPICS/R3_15_8-x86_64/base-3.15.8/lib/linux-x86_64/libCom.so.3.15.8(start_routine+0xdc)
>     [    0x7f3a03c72064]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xc4)
>     [    0x7f3a03f6f62d]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)
>     EPICS Release EPICS R3.15.8.
>     Local time is 2021-05-19 08:18:56.245430055 EDT
>     Please E-mail this message to the author or to tech-talk at aps.anl.gov
>     Calling epicsThreadSuspendSelf()
>     Thread CAS-client (0x7f3978023160) suspended
>     callbackRequest: cbLow ring buffer full
>     epics> date
>     2021/05/19 11:05:33.287078
> 

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System