Re: CAS-client thread issues in areaDetector IOC

["Wlodek, Jakub via Tech-talk" <tech-talk at aps.anl.gov>]

Hi all,

After updating my machine and rebooting it, then recompiling all of EPICS + modules, I ran a test by starting a simDetector IOC and letting it sit overnight.

I didn't activate any plugins or press any buttons, and when I checked on the IOC this morning, the error message was displayed several times, along with

a Segmentation Fault crash:

epics>

epics> epicsEventTrigger: pthread_mutex_lock failed: Invalid argument

epicsEventMustTriggerThread CAS-client (0x7fd6e8042940) can't proceed, suspending.

Dumping a stack trace of thread 'CAS-client':

[    0x56549c7567e3]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(epicsStackTrace+0x73)

[    0x56549c747215]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(cantProceed+0xc5)

[    0x56549c6d0403]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(db_close_events+0x33)

[    0x56549c6f7d2f]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(destroy_tcp_client+0x8f)

[    0x56549c6f894d]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(camsgtask+0x13d)

[    0x56549c751c18]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(start_routine+0xf8)

[    0x7fd82d7016db]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb)

[    0x7fd82c49688f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)

Thread CAS-client (0x7fd6e8042940) suspended

epicsEventTrigger: pthread_mutex_lock failed: Invalid argument

epicsEventMustTriggerThread CAS-client (0x7fd6e8042bf0) can't proceed, suspending.

Dumping a stack trace of thread 'CAS-client':

[    0x56549c7567e3]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(epicsStackTrace+0x73)

[    0x56549c747215]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(cantProceed+0xc5)

[    0x56549c6d0403]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(db_close_events+0x33)

[    0x56549c6f7d2f]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(destroy_tcp_client+0x8f)

[    0x56549c6f894d]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(camsgtask+0x13d)

[    0x56549c751c18]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(start_routine+0xf8)

[    0x7fd82d7016db]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb)

[    0x7fd82c49688f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)

Thread CAS-client (0x7fd6e8042bf0) suspended

epicsEventTrigger: pthread_mutex_unlock failed: Invalid argument

epicsEventMustTriggerThread CAS-client (0x7fd6e8043490) can't proceed, suspending.

Dumping a stack trace of thread 'CAS-client':

[    0x56549c7567e3]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(epicsStackTrace+0x73)

[    0x56549c747215]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(cantProceed+0xc5)

[    0x56549c6d0403]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(db_close_events+0x33)

[    0x56549c6f7d2f]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(destroy_tcp_client+0x8f)

[    0x56549c6f894d]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(camsgtask+0x13d)

[    0x56549c751c18]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(start_routine+0xf8)

[    0x7fd82d7016db]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb)

[    0x7fd82c49688f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)

Thread CAS-client (0x7fd6e8043490) suspended

epicsEventTrigger: pthread_mutex_unlock failed: Invalid argument

epicsEventMustTriggerThread CAS-client (0x7fd6e8043da0) can't proceed, suspending.

Dumping a stack trace of thread 'CAS-client':

[    0x56549c7567e3]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(epicsStackTrace+0x73)

[    0x56549c747215]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(cantProceed+0xc5)

[    0x56549c6d0403]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(db_close_events+0x33)

[    0x56549c6f7d2f]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(destroy_tcp_client+0x8f)

[    0x56549c6f894d]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(camsgtask+0x13d)

[    0x56549c751c18]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(start_routine+0xf8)

[    0x7fd82d7016db]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb)

[    0x7fd82c49688f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)

Thread CAS-client (0x7fd6e8043da0) suspended

epicsEventTrigger: pthread_mutex_unlock failed: Invalid argument

epicsEventMustTriggerThread CAS-client (0x7fd6e80446b0) can't proceed, suspending.

Dumping a stack trace of thread 'CAS-client':

[    0x56549c7567e3]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(epicsStackTrace+0x73)

[    0x56549c747215]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(cantProceed+0xc5)

[    0x56549c6d0403]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(db_close_events+0x33)

[    0x56549c6f7d2f]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(destroy_tcp_client+0x8f)

[    0x56549c6f894d]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(camsgtask+0x13d)

[    0x56549c751c18]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(start_routine+0xf8)

[    0x7fd82d7016db]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb)

[    0x7fd82c49688f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)

epicsEventTrigger: pthread_mutex_unlock failed: Invalid argument

epicsEventMustTriggerThread CAS-client (0x7fd6e8044fc0) can't proceed, suspending.

Dumping a stack trace of thread 'CAS-client':

[    0x56549c7567e3]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(epicsStackTrace+0x73)

[    0x56549c747215]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(cantProceed+0xc5)

[    0x56549c6d0403]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(db_close_events+0x33)

[    0x56549c6f7d2f]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(destroy_tcp_client+0x8f)

[    0x56549c6f894d]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(camsgtask+0x13d)

[    0x56549c751c18]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(start_routine+0xf8)

[    0x7fd82d7016db]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb)

[    0x7fd82c49688f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)

Thread CAS-client (0x7fd6e80446b0) suspended

Thread CAS-client (0x7fd6e8044fc0) suspended

Segmentation fault (core dumped)

I still haven't been able to reproduce this on any other machines, so I may just assume that something on my test machine is broken, and I'll just move development to a

different machine until I can sort this issue out.

Jakub

---

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Wlodek, Jakub via Tech-talk <tech-talk at aps.anl.gov>
Sent: Wednesday, January 29, 2020 3:13 PM
To: Mark Rivers <rivers at cars.uchicago.edu>; Michael Davidsaver <mdavidsaver at gmail.com>
Cc: tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Re: CAS-client thread issues in areaDetector IOC

 

Hi all,

After testing on another Ubuntu 18 machine, I couldn't reproduce the issue. It seems that it is limited to my original specific machine. I will try updating all of the packages on it,

recompiling, and then I will use a python script to test, as suggested.

If I find the cause of the issue I will further update this thread.

Thanks,

Jakub

---

From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Monday, January 27, 2020 1:57 PM
To: Wlodek, Jakub <jwlodek at bnl.gov>; Michael Davidsaver <mdavidsaver at gmail.com>
Cc: tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: RE: CAS-client thread issues in areaDetector IOC

 

I suggest using the simDetector and a Python script for testing.  Start the simDetector with no plugins enabled, and have Python just press start and stop in a loop.  If that does not fail then enable plugins one at a time until you find a minimal configuration is that will generate the error.  Then others can try to reproduce.

 

I just did a lot of clicking on the simDetector with Ubuntu 18.04 and base 7.0.3.1 and cannot make it fail.

 

Mark

 

 

 

From: Wlodek, Jakub <jwlodek at bnl.gov>
Sent: Monday, January 27, 2020 11:35 AM
To: Michael Davidsaver <mdavidsaver at gmail.com>
Cc: Mark Rivers <rivers at cars.uchicago.edu>; tech-talk at aps.anl.gov
Subject: Re: CAS-client thread issues in areaDetector IOC

 

Hi Mark

 

Is this error consistent or intermittent?

 

I observe it every time I start an IOC, but not immediately after startup, it shows up after some time with normal operation (acquire start/stop, enable/disable plugins etc.)

I can't seem to nail down a specific series of steps that leads to the error, but it seems consistent. I also re cloned and rebuilt everything again from scratch in a different location

to make sure something wasn't messed up in that specific set of sources, and I saw the same issue.

 

Regards,

Jakub

---

From: Michael Davidsaver <mdavidsaver at gmail.com>
Sent: Monday, January 27, 2020 12:28 PM
To: Wlodek, Jakub <jwlodek at bnl.gov>
Cc: Mark Rivers <rivers at cars.uchicago.edu>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Re: CAS-client thread issues in areaDetector IOC

 

On 1/27/20 7:45 AM, Wlodek, Jakub wrote:
> ...
> In addition, the following tests showed some error message, but were seemingly re-run successfully:

We have some tests which intentionally provoke error to ensure that
they can be handled correctly.  Unless you see any "fail" instead of "ok",
then the tests are passing and we'll have to look elsewhere.



> iocshTest.t ................... 1/19 Command no_such_command not found.
> iocsh Error: Break
> Can't open no_such_file.cmd: No such file or directory
> .
> .
> .
> iocshTest.t ................... ok 
>
> chfPluginTest.t ............ 1/1433 chfConfigParseStart: plugin pvt alloc failed
> chfPluginTest.t ............ ok       
>
> netget.t ................ 1/3 **** The executable "caRepeater" couldn't be located
> **** because of errno = "No such file or directory".
> **** You may need to modify your PATH environment variable.
> **** Unable to start "CA Repeater" process.
> netget.t ................ ok  
>
> There were also several tests that listed json parsing errors.
>
> Regards,
> Jakub
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> *From:* Michael Davidsaver <mdavidsaver at gmail.com>
> *Sent:* Friday, January 24, 2020 4:52 PM
> *To:* Wlodek, Jakub <jwlodek at bnl.gov>
> *Cc:* Mark Rivers <rivers at cars.uchicago.edu>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
> *Subject:* Re: CAS-client thread issues in areaDetector IOC
>  
> On 1/24/20 10:15 AM, Mark Rivers wrote:
>> Ø  Could it be that there is a problem with something in the newest release of epics-base?
> ...
>> You are running 7.0.3 now.  If you go back to 7.0.3.1 does the problem re-appear?
>
> If the assert() failure does re-appear, could you run the base unittests and report any which don't pass?
>
>> make runtests