Experimental Physics and
Industrial Control System

Mark Rivers via Tech-talk <tech-talk at aps.anl.gov> · Tue, 8 Jun 2021 17:27:22 +0000

One possibility is that the thread the pulls the message out of the queue has locked up for some reason.

One possibility is a deadlock.  When this happens you could try typing

epicsMutexShowAll 1

several times.  If the same mutexes are locked each time you type that, then there is likely a deadlock.

What OS is this?

What version of EPICS base?

Is the driver or device support using asyn?  If so, what version?

Mark

________________________________
From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Daykin, Evan via Tech-talk <tech-talk at aps.anl.gov>
Sent: Tuesday, June 8, 2021 12:10 PM
To: EPICS tech-talk
Subject: cbLow consumption randomly stopping

Hello tech-talk,

We have an IOC repeatedly locking up with the infamous “cbLow ring buffer full” message. The IOC consumes data from our CAEN Pico-8 card via an EPICS driver at 100Hz (about 1200-1215 events per second total) , and at some point on the order of hours to days, seems to stop consuming callbacks from the queue. I wrote a script to track this, and it seems to all be happening in one shot. Below is some data I captured from the last time this happened. Before this, everything was happy for about 4 hours, but sometimes it takes up to a week for this to happen.

Time

High-water

Items in q

Q capacity

%

Oflws

6/7/2021 19:22:40

13

0

12000

0

0

6/7/2021 19:22:41

13

0

12000

0

0

6/7/2021 19:22:42

13

0

12000

0

0

6/7/2021 19:22:44

13

0

12000

0

0

6/7/2021 19:22:45

571

571

12000

4.8

0

6/7/2021 19:22:46

1778

1778

12000

14.8

0

6/7/2021 19:22:47

2991

2991

12000

24.9

0

6/7/2021 19:22:48

4204

4204

12000

35

0

6/7/2021 19:22:49

5406

5406

12000

45

0

6/7/2021 19:22:50

6619

6619

12000

55.2

0

6/7/2021 19:22:51

7834

7834

12000

65.3

0

6/7/2021 19:22:52

9048

9048

12000

75.4

0

6/7/2021 19:22:53

10249

10249

12000

85.4

0

6/7/2021 19:22:54

11462

11462

12000

95.5

0

6/7/2021 19:22:56

12000

12000

12000

100

1

6/7/2021 19:22:58

12000

12000

12000

100

1

Has anyone else experienced this? If so, what is the remedy? I can’t seem to correlate it to anything unusual happening in the OS/kernel.

Thanks,

Evan

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System