Re: CAS-client thread issues in areaDetector IOC

["Wlodek, Jakub via Tech-talk" <tech-talk at aps.anl.gov>]

Hi all,

After some testing, I think I am able to reproduce the problem. Essentially, I had a CS-Studio client opened for both the simDetector IOC and the ADUVC IOC, with a misconfigured

CA_MAX_ARRAY_BYTES (it had too small a value). I hadn't noticed because I use ImageJ for testing the camera IOCs.

Closing the CSS window and running the IOC with a simple python script to set values etc. I did not see the error despite running overnight.

I tried running again after fixing the CSS issue, and now I don't see the problem. I wonder how the client is able to have such an effect on the IOC process, and why I only saw the issue running with

base 7.0.3.1?

I also tried the following to make sure the issue was tied to this: I started the IOC without the misconfigured CSS open, and ran the script to control it (this gave no errors). Then, while the IOC was running,

I opened the CSS window, and promptly saw the error message display in the IOC shell several times, followed by a Segmentation Fault, as before.

Thanks for the help with this problem,

Jakub

---

From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Friday, January 31, 2020 7:22 PM
To: Wlodek, Jakub <jwlodek at bnl.gov>; Wlodek, Jakub via Tech-talk <tech-talk at aps.anl.gov>; Michael Davidsaver <mdavidsaver at gmail.com>
Subject: RE: CAS-client thread issues in areaDetector IOC

 

Hi Jakub,

 

The error seems to be consistently in the CAS-client.  This suggests perhaps there is some CA client that is causing it.  Can you try shutting down all CA clients that are accessing this IOC?  You can use “casr 1” to determine what clients are connected.

 

Mark

 

 

From: Wlodek, Jakub <jwlodek at bnl.gov>
Sent: Friday, January 31, 2020 8:22 AM
To: Wlodek, Jakub via Tech-talk <tech-talk at aps.anl.gov>; Mark Rivers <rivers at cars.uchicago.edu>; Michael Davidsaver <mdavidsaver at gmail.com>
Subject: Re: CAS-client thread issues in areaDetector IOC

 

Hi all,

 

After updating my machine and rebooting it, then recompiling all of EPICS + modules, I ran a test by starting a simDetector IOC and letting it sit overnight.

I didn't activate any plugins or press any buttons, and when I checked on the IOC this morning, the error message was displayed several times, along with

a Segmentation Fault crash:

 

epics>

epics> epicsEventTrigger: pthread_mutex_lock failed: Invalid argument

epicsEventMustTriggerThread CAS-client (0x7fd6e8042940) can't proceed, suspending.

Dumping a stack trace of thread 'CAS-client':

[    0x56549c7567e3]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(epicsStackTrace+0x73)

[    0x56549c747215]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(cantProceed+0xc5)

[    0x56549c6d0403]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(db_close_events+0x33)

[    0x56549c6f7d2f]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(destroy_tcp_client+0x8f)

[    0x56549c6f894d]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(camsgtask+0x13d)

[    0x56549c751c18]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(start_routine+0xf8)

[    0x7fd82d7016db]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb)

[    0x7fd82c49688f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)

Thread CAS-client (0x7fd6e8042940) suspended

epicsEventTrigger: pthread_mutex_lock failed: Invalid argument

epicsEventMustTriggerThread CAS-client (0x7fd6e8042bf0) can't proceed, suspending.

Dumping a stack trace of thread 'CAS-client':

[    0x56549c7567e3]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(epicsStackTrace+0x73)

[    0x56549c747215]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(cantProceed+0xc5)

[    0x56549c6d0403]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(db_close_events+0x33)

[    0x56549c6f7d2f]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(destroy_tcp_client+0x8f)

[    0x56549c6f894d]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(camsgtask+0x13d)

[    0x56549c751c18]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(start_routine+0xf8)

[    0x7fd82d7016db]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb)

[    0x7fd82c49688f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)

Thread CAS-client (0x7fd6e8042bf0) suspended

epicsEventTrigger: pthread_mutex_unlock failed: Invalid argument

epicsEventMustTriggerThread CAS-client (0x7fd6e8043490) can't proceed, suspending.

Dumping a stack trace of thread 'CAS-client':

[    0x56549c7567e3]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(epicsStackTrace+0x73)

[    0x56549c747215]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(cantProceed+0xc5)

[    0x56549c6d0403]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(db_close_events+0x33)

[    0x56549c6f7d2f]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(destroy_tcp_client+0x8f)

[    0x56549c6f894d]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(camsgtask+0x13d)

[    0x56549c751c18]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(start_routine+0xf8)

[    0x7fd82d7016db]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb)

[    0x7fd82c49688f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)

Thread CAS-client (0x7fd6e8043490) suspended

epicsEventTrigger: pthread_mutex_unlock failed: Invalid argument

epicsEventMustTriggerThread CAS-client (0x7fd6e8043da0) can't proceed, suspending.

Dumping a stack trace of thread 'CAS-client':

[    0x56549c7567e3]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(epicsStackTrace+0x73)

[    0x56549c747215]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(cantProceed+0xc5)

[    0x56549c6d0403]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(db_close_events+0x33)

[    0x56549c6f7d2f]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(destroy_tcp_client+0x8f)

[    0x56549c6f894d]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(camsgtask+0x13d)

[    0x56549c751c18]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(start_routine+0xf8)

[    0x7fd82d7016db]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb)

[    0x7fd82c49688f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)

Thread CAS-client (0x7fd6e8043da0) suspended

epicsEventTrigger: pthread_mutex_unlock failed: Invalid argument

epicsEventMustTriggerThread CAS-client (0x7fd6e80446b0) can't proceed, suspending.

Dumping a stack trace of thread 'CAS-client':

[    0x56549c7567e3]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(epicsStackTrace+0x73)

[    0x56549c747215]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(cantProceed+0xc5)

[    0x56549c6d0403]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(db_close_events+0x33)

[    0x56549c6f7d2f]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(destroy_tcp_client+0x8f)

[    0x56549c6f894d]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(camsgtask+0x13d)

[    0x56549c751c18]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(start_routine+0xf8)

[    0x7fd82d7016db]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb)

[    0x7fd82c49688f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)

epicsEventTrigger: pthread_mutex_unlock failed: Invalid argument

epicsEventMustTriggerThread CAS-client (0x7fd6e8044fc0) can't proceed, suspending.

Dumping a stack trace of thread 'CAS-client':

[    0x56549c7567e3]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(epicsStackTrace+0x73)

[    0x56549c747215]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(cantProceed+0xc5)

[    0x56549c6d0403]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(db_close_events+0x33)

[    0x56549c6f7d2f]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(destroy_tcp_client+0x8f)

[    0x56549c6f894d]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(camsgtask+0x13d)

[    0x56549c751c18]: /epics/src/support/areaDetector/ADSimDetector/iocs/simDetectorIOC/bin/linux-x86_64/simDetectorApp(start_routine+0xf8)

[    0x7fd82d7016db]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb)

[    0x7fd82c49688f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)

Thread CAS-client (0x7fd6e80446b0) suspended

Thread CAS-client (0x7fd6e8044fc0) suspended

Segmentation fault (core dumped)

 

I still haven't been able to reproduce this on any other machines, so I may just assume that something on my test machine is broken, and I'll just move development to a

different machine until I can sort this issue out.

 

Jakub

 

---

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Wlodek, Jakub via Tech-talk <tech-talk at aps.anl.gov>
Sent: Wednesday, January 29, 2020 3:13 PM
To: Mark Rivers <rivers at cars.uchicago.edu>; Michael Davidsaver <mdavidsaver at gmail.com>
Cc: tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Re: CAS-client thread issues in areaDetector IOC

 

Hi all,

 

After testing on another Ubuntu 18 machine, I couldn't reproduce the issue. It seems that it is limited to my original specific machine. I will try updating all of the packages on it,

recompiling, and then I will use a python script to test, as suggested.

 

If I find the cause of the issue I will further update this thread.

 

Thanks,

Jakub

---

From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Monday, January 27, 2020 1:57 PM
To: Wlodek, Jakub <jwlodek at bnl.gov>; Michael Davidsaver <mdavidsaver at gmail.com>
Cc: tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: RE: CAS-client thread issues in areaDetector IOC

 

I suggest using the simDetector and a Python script for testing.  Start the simDetector with no plugins enabled, and have Python just press start and stop in a loop.  If that does not fail then enable plugins one at a time until you find a minimal configuration is that will generate the error.  Then others can try to reproduce.

 

I just did a lot of clicking on the simDetector with Ubuntu 18.04 and base 7.0.3.1 and cannot make it fail.

 

Mark

 

 

 

From: Wlodek, Jakub <jwlodek at bnl.gov>
Sent: Monday, January 27, 2020 11:35 AM
To: Michael Davidsaver <mdavidsaver at gmail.com>
Cc: Mark Rivers <rivers at cars.uchicago.edu>; tech-talk at aps.anl.gov
Subject: Re: CAS-client thread issues in areaDetector IOC

 

Hi Mark

 

Is this error consistent or intermittent?

 

I observe it every time I start an IOC, but not immediately after startup, it shows up after some time with normal operation (acquire start/stop, enable/disable plugins etc.)

I can't seem to nail down a specific series of steps that leads to the error, but it seems consistent. I also re cloned and rebuilt everything again from scratch in a different location

to make sure something wasn't messed up in that specific set of sources, and I saw the same issue.

 

Regards,

Jakub

---

From: Michael Davidsaver <mdavidsaver at gmail.com>
Sent: Monday, January 27, 2020 12:28 PM
To: Wlodek, Jakub <jwlodek at bnl.gov>
Cc: Mark Rivers <rivers at cars.uchicago.edu>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Re: CAS-client thread issues in areaDetector IOC

 

On 1/27/20 7:45 AM, Wlodek, Jakub wrote:
> ...
> In addition, the following tests showed some error message, but were seemingly re-run successfully:

We have some tests which intentionally provoke error to ensure that
they can be handled correctly.  Unless you see any "fail" instead of "ok",
then the tests are passing and we'll have to look elsewhere.



> iocshTest.t ................... 1/19 Command no_such_command not found.
> iocsh Error: Break
> Can't open no_such_file.cmd: No such file or directory
> .
> .
> .
> iocshTest.t ................... ok 
>
> chfPluginTest.t ............ 1/1433 chfConfigParseStart: plugin pvt alloc failed
> chfPluginTest.t ............ ok       
>
> netget.t ................ 1/3 **** The executable "caRepeater" couldn't be located
> **** because of errno = "No such file or directory".
> **** You may need to modify your PATH environment variable.
> **** Unable to start "CA Repeater" process.
> netget.t ................ ok  
>
> There were also several tests that listed json parsing errors.
>
> Regards,
> Jakub
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> *From:* Michael Davidsaver <mdavidsaver at gmail.com>
> *Sent:* Friday, January 24, 2020 4:52 PM
> *To:* Wlodek, Jakub <jwlodek at bnl.gov>
> *Cc:* Mark Rivers <rivers at cars.uchicago.edu>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
> *Subject:* Re: CAS-client thread issues in areaDetector IOC
>  
> On 1/24/20 10:15 AM, Mark Rivers wrote:
>> Ø  Could it be that there is a problem with something in the newest release of epics-base?
> ...
>> You are running 7.0.3 now.  If you go back to 7.0.3.1 does the problem re-appear?
>
> If the assert() failure does re-appear, could you run the base unittests and report any which don't pass?
>
>> make runtests