CAS-client thread issues in areaDetector IOC

["Wlodek, Jakub via Tech-talk" <tech-talk@aps.anl.gov>]

Hi all,

I am currently ironing out a few bugs in an areaDetector driver I wrote a while ago for USB webcams, and doing some code cleanup (removing redundant code etc.), and I have encountered an

error I have not seen before, and I'm not sure what I have done to introduce it. I have ~15 beamline cameras working with this driver for time frames ranging from several months to a year, so I am

fairly certain this is a new issue. Essentially, after the IOC runs normally for  some time, I get the following stack trace dumped to the EPICS shell:

epicsEventTrigger: pthread_mutex_lock failed: Invalid argument

epicsEventMustTriggerThread CAS-client (0x7f4e4c042f00) can't proceed, suspending.

Dumping a stack trace of thread 'CAS-client':

[    0x55bf0e032393]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(epicsStackTrace+0x73)

[    0x55bf0e022dc5]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(cantProceed+0xc5)

[    0x55bf0dfabfb3]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(db_close_events+0x33)

[    0x55bf0dfd38df]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(destroy_tcp_client+0x8f)

[    0x55bf0dfd44fd]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(camsgtask+0x13d)

[    0x55bf0e02d7c8]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(start_routine+0xf8)

[    0x7f4f922c46db]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb)

[    0x7f4f9105988f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)

Thread CAS-client (0x7f4e4c042f00) suspended

2020/01/23 10:28:32.930 ADUVC::acquireStop Stopping aquisition

2020/01/23 10:28:32.930 ADUVC::writeInt32 function=8 value=0

epicsEventTrigger: pthread_mutex_lock failed: Invalid argument

epicsEventMustTriggerThread CAS-client (0x7f4e4c0433d0) can't proceed, suspending.

Dumping a stack trace of thread 'CAS-client':

[    0x55bf0e032393]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(epicsStackTrace+0x73)

[    0x55bf0e022dc5]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(cantProceed+0xc5)

[    0x55bf0dfabfb3]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(db_close_events+0x33)

[    0x55bf0dfd38df]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(destroy_tcp_client+0x8f)

[    0x55bf0dfd44fd]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(camsgtask+0x13d)

[    0x55bf0e02d7c8]: /epics/src/support/areaDetector/ADUVC/iocs/adUVCIOC/bin/linux-x86_64/adUVCApp(start_routine+0xf8)

[    0x7f4f922c46db]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb)

[    0x7f4f9105988f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)

Thread CAS-client (0x7f4e4c0433d0) suspended

Segmentation fault (core dumped)

Note how it does not crash after the first stack dump, and the number of times this gets printed varies, usually up to 10 times. I running the IOC with all

plugins except for NDStdArrays disabled, and capturing 640x480 images @ 30 fps. The currently updated code is available here: https://github.com/jwlodek/ADUVC, with the four

most recent commits being the code cleanup changes since the last known-working version. Could anyone help shine some light on what might be the issue here?

Thanks,

Jakub Wlodek