RE: CAS-client thread issues in areaDetector IOC

[Mark Rivers via Tech-talk <tech-talk@aps.anl.gov>]

Hi Jakub,

 

The errors are in the Channel Access server from EPICS base.  Questions:

-          Are your working cameras using the same version of base as the camera you are testing?

-          Are you sure you have rebuilt everything that your camera IOC uses with this version of base?

 

This looks to me like it could be a memory corruption problem, e.g. array out of bounds.  Those errors can remain hidden until you make small changes to your code.

 

I suggest running the IOC under valgrind to see if it detects memory allocation issues.

 

Mark

 

 

From: Tech-talk <tech-talk-bounces@aps.anl.gov> On Behalf Of Wlodek, Jakub via Tech-talk
Sent: Thursday, January 23, 2020 9:44 AM
To: tech-talk@aps.anl.gov
Subject: CAS-client thread issues in areaDetector IOC

 

Hi all,

 

I am currently ironing out a few bugs in an areaDetector driver I wrote a while ago for USB webcams, and doing some code cleanup (removing redundant code etc.), and I have encountered an

error I have not seen before, and I'm not sure what I have done to introduce it. I have ~15 beamline cameras working with this driver for time frames ranging from several months to a year, so I am

fairly certain this is a new issue. Essentially, after the IOC runs normally for  some time, I get the following stack trace dumped to the EPICS shell:

 

epicsEventTrigger: pthread_mutex_lock failed: Invalid argument

epicsEventMustTriggerThread CAS-client (0x7f4e4c042f00) can't proceed, suspending.

Dumping a stack trace of thread 'CAS-client':

[    0x55bf0e032393]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(epicsStackTrace+0x73)

[    0x55bf0e022dc5]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(cantProceed+0xc5)

[    0x55bf0dfabfb3]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(db_close_events+0x33)

[    0x55bf0dfd38df]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(destroy_tcp_client+0x8f)

[    0x55bf0dfd44fd]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(camsgtask+0x13d)

[    0x55bf0e02d7c8]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(start_routine+0xf8)

[    0x7f4f922c46db]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb)

[    0x7f4f9105988f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)

Thread CAS-client (0x7f4e4c042f00) suspended

2020/01/23 10:28:32.930 ADUVC::acquireStop Stopping aquisition

2020/01/23 10:28:32.930 ADUVC::writeInt32 function=8 value=0

epicsEventTrigger: pthread_mutex_lock failed: Invalid argument

epicsEventMustTriggerThread CAS-client (0x7f4e4c0433d0) can't proceed, suspending.

Dumping a stack trace of thread 'CAS-client':

[    0x55bf0e032393]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(epicsStackTrace+0x73)

[    0x55bf0e022dc5]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(cantProceed+0xc5)

[    0x55bf0dfabfb3]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(db_close_events+0x33)

[    0x55bf0dfd38df]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(destroy_tcp_client+0x8f)

[    0x55bf0dfd44fd]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(camsgtask+0x13d)

[    0x55bf0e02d7c8]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(start_routine+0xf8)

[    0x7f4f922c46db]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb)

[    0x7f4f9105988f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)

Thread CAS-client (0x7f4e4c0433d0) suspended

Segmentation fault (core dumped)

 

Note how it does not crash after the first stack dump, and the number of times this gets printed varies, usually up to 10 times. I running the IOC with all

plugins except for NDStdArrays disabled, and capturing 640x480 images @ 30 fps. The currently updated code is available here: https://github.com/jwlodek/ADUVC, with the four

most recent commits being the code cleanup changes since the last known-working version. Could anyone help shine some light on what might be the issue here?

Thanks,

Jakub Wlodek