cbLow consumption randomly stopping

["Daykin, Evan via Tech-talk" <tech-talk at aps.anl.gov>]

Hello tech-talk,

 

We have an IOC repeatedly locking up with the infamous “cbLow ring buffer full” message. The IOC consumes data from our CAEN Pico-8 card via an EPICS driver at 100Hz (about 1200-1215 events per second total) , and at some point on the order of hours to days, seems to stop consuming callbacks from the queue. I wrote a script to track this, and it seems to all be happening in one shot. Below is some data I captured from the last time this happened. Before this, everything was happy for about 4 hours, but sometimes it takes up to a week for this to happen.

 

Time	High-water	Items in q		Q capacity	%	Oflws
6/7/2021 19:22:40	13	0	12000		0		0
6/7/2021 19:22:41	13	0	12000		0		0
6/7/2021 19:22:42	13	0	12000		0		0
6/7/2021 19:22:44	13	0	12000		0		0
6/7/2021 19:22:45	571	571	12000		4.8		0
6/7/2021 19:22:46	1778	1778	12000		14.8		0
6/7/2021 19:22:47	2991	2991	12000		24.9		0
6/7/2021 19:22:48	4204	4204	12000		35		0
6/7/2021 19:22:49	5406	5406	12000		45		0
6/7/2021 19:22:50	6619	6619	12000		55.2		0
6/7/2021 19:22:51	7834	7834	12000		65.3		0
6/7/2021 19:22:52	9048	9048	12000		75.4		0
6/7/2021 19:22:53	10249	10249	12000		85.4		0
6/7/2021 19:22:54	11462	11462	12000		95.5		0
6/7/2021 19:22:56	12000	12000	12000		100		1
6/7/2021 19:22:58	12000	12000	12000		100		1

Has anyone else experienced this? If so, what is the remedy? I can’t seem to correlate it to anything unusual happening in the OS/kernel.

 

Thanks,

Evan