Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020 
<== Date ==> <== Thread ==>

Subject: Segmentation Fault when exiting certain Area Detector
From: "Gofron, Kazimierz via Tech-talk" <tech-talk@aps.anl.gov>
To: "tech-talk@aps.anl.gov" <tech-talk@aps.anl.gov>
Cc: "Wlodek, Jakub" <jwlodek@bnl.gov>
Date: Wed, 23 Jan 2019 20:37:17 +0000
Hi Mark,

I am not sure which version of GraphicsMegic is used. The GraphicsMagic version in ADSupport states that GM verson is 1.3.25

" This directory contains the GraphicsMagick code (1.3.25). It has been modified to build under the EPICS build system."
https://github.com/areaDetector/ADSupport/blob/master/supportApp/GraphicsMagickSrc/README.epics.md#L1

Should we be using a different version, or is it a simple typo in the README file?

Kazimierz

Message: 2
Date: Wed, 23 Jan 2019 19:58:25 +0000
From: Mark Rivers <rivers@cars.uchicago.edu>
To: "'Wlodek, Jakub'" <jwlodek@bnl.gov>
Cc: "tech-talk@aps.anl.gov" <tech-talk@aps.anl.gov>
Subject: RE: Segmentation Fault when exiting certain Area Detector
	IOCs
Message-ID: <1937f9dbbe7e4dde845bdcf684d1e716@cars.uchicago.edu>
Content-Type: text/plain; charset="utf-8"

Since I cannot reproduce the problem I think you are going to have to debug the GraphicsMagick issue.  You could also just ignore it for now, since it is only happening when you exit the application.  Or just exit with ^C and then it should not happen ?

Mark


From: Wlodek, Jakub <jwlodek@bnl.gov>
Sent: Wednesday, January 23, 2019 1:41 PM
To: Mark Rivers <rivers@cars.uchicago.edu>
Cc: Shen, Guobao <gshen@anl.gov>; tech-talk@aps.anl.gov
Subject: Re: Segmentation Fault when exiting certain Area Detector IOCs


Hi Mark,



I tried retracing the segfault again, and I found that in the driver I had written myself, the segfault occurred at the pPvt->lock() line because I was calling an

additional disconnect from asyn port at the end of my destructor. So it turns out that the two drivers were having separate issues,

I had just assumed they were the same because the seg faults occurred after trying to exit in both.



As for GraphicsMagick, I am using the version compiled in ADSupport,

and my ADSupport version is 1-6 I believe, so that fix should already be applied. When looking at the Magick screen in CSS the

Magick1:DriverVersion_RBV PV says 3.4.0, but I think that is just the ADCore version I am running.



Thanks,

Jakub

________________________________
From: Mark Rivers <rivers@cars.uchicago.edu<mailto:rivers@cars.uchicago.edu>>
Sent: Wednesday, January 23, 2019 1:41:53 PM
To: Wlodek, Jakub
Cc: Shen, Guobao; tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov>
Subject: RE: Segmentation Fault when exiting certain Area Detector IOCs


Aha, so it looks like the problem is actually a signal handler in GraphicsMagick.



What version of GraphicsMagick are you using?  A local install or the version in ADSupport?



ADSupport R1-4 has this entry in RELEASE.md



* Fixed a bug in GraphicsMagickSrc.  We had changed Magick++/lib/Image.cpp to call UnregisterStaticModules in

  a cleanup destructor.  The problem was that this destructor is always called, even if GraphicsMagick has not

  ever been initialized.  That will happen if an application is built with GraphicsMagick but NDFileMagick is

  not loaded in the startup script.  The symptom was an assertion failure when exiting the IOC.  This occurred

  on both Linux and Windows.  Fixed the problem by ensuring that UnregisterStaticModules is not called if

  GraphicsMagick has not been initialized.



This sounds like a very similar problem.  However, that note says it was an assertion failure, while you are seeing an seq fault.



Note that this appears to be completely different than your original message which you said you had traced to a problem in the asynPortDriver destructor calling pPvt->lock().



Mark





From: Wlodek, Jakub <jwlodek@bnl.gov<mailto:jwlodek@bnl.gov>>
Sent: Wednesday, January 23, 2019 12:30 PM
To: Mark Rivers <rivers@cars.uchicago.edu<mailto:rivers@cars.uchicago.edu>>
Cc: Shen, Guobao <gshen@anl.gov<mailto:gshen@anl.gov>>; tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov>
Subject: Re: Segmentation Fault when exiting certain Area Detector IOCs



Hi Mark,



Below is the result of running



gdb prosilicaApp prosilica_exit_core_dump



(gdb) bt

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51

#1  0x00007fbc301ca51a in __GI_abort () at abort.c:118

#2  0x000055de5682da41 in MagickPanicSignalHandler (signo=11) at ../magick.c:840

#3  <signal handler called>

#4  malloc_consolidate (av=av@entry=0x7fbc3052fb00 <main_arena>) at malloc.c:4207

#5  0x00007fbc3020ded8 in _int_free (av=0x7fbc3052fb00 <main_arena>, p=0x55de579b7c00, have_lock=0) at malloc.c:4113

#6  0x00007fbc2fa82971 in ?? () from /usr/lib/x86_64-linux-gnu/libGL.so.1

#7  0x00007fbc298b5486 in ?? () from /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.375.82

#8  0x00007fbc29c236bd in ?? () from /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.375.82

#9  0x00007fbc2fa5101b in ?? () from /usr/lib/x86_64-linux-gnu/libGL.so.1

#10 0x00007fbc2facb2f2 in ?? () from /usr/lib/x86_64-linux-gnu/libGL.so.1

#11 0x00007fbc348fee48 in _dl_fini () at dl-fini.c:240

#12 0x00007fbc301cb940 in __run_exit_handlers (status=status@entry=0, listp=0x7fbc3052f5d8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true,

    run_dtors=run_dtors@entry=true) at exit.c:83

#13 0x00007fbc301cb99a in __GI_exit (status=status@entry=0) at exit.c:105

#14 0x000055de570a2acc in epicsExit (status=0) at ../../src/misc/epicsExit.c:187

#15 0x000055de5636e1dd in main (argc=<optimized out>, argv=<optimized out>) at ../prosilicaAppMain.cpp:21

(gdb)



please let me know if I should run any other commands on the core dump file



Jakub





________________________________

From: Mark Rivers <rivers@cars.uchicago.edu<mailto:rivers@cars.uchicago.edu>>
Sent: Tuesday, January 22, 2019 12:44:36 PM
To: Wlodek, Jakub
Cc: Shen, Guobao; tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov>
Subject: RE: Segmentation Fault when exiting certain Area Detector IOCs



Hi Jakob,



I don?t actually need the core file itself.  I need you to run gdb on that core file yourself and the output.  You do that as follows:



gdb path_to_your prosilica_IOC_application core.xxxxx



where core.xxxxx is the name of the core file.



For example:



gdb /home/epics/support/areaDetector-3-4-0/ADProsilica/iocs/prosilicaIOC/bin/linux-x86_64/prosilicaApp core.12345



When you run that you will get the gdb prompt.  At the prompt type the following command:



bt



Send the output of that command.



Mark





From: Wlodek, Jakub <jwlodek@bnl.gov<mailto:jwlodek@bnl.gov>>
Sent: Tuesday, January 22, 2019 8:40 AM
To: Mark Rivers <rivers@cars.uchicago.edu<mailto:rivers@cars.uchicago.edu>>
Cc: Shen, Guobao <gshen@anl.gov<mailto:gshen@anl.gov>>; tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov>
Subject: Re: Segmentation Fault when exiting certain Area Detector IOCs



Hi all,



I ran the prosilica IOC and generated the following core dump:



prosilica_exit_core_dump<https://brookhavenlab-my.sharepoint.com/:u:/g/personal/jwlodek_bnl_gov/EWtcZWwNxeVDvi73Dysf_R0BsgAjl9H1wtyQigA1mesSOQ?e=BdUfPW>

[Image removed by sender.]<https://brookhavenlab-my.sharepoint.com/:u:/g/personal/jwlodek_bnl_gov/EWtcZWwNxeVDvi73Dysf_R0BsgAjl9H1wtyQigA1mesSOQ?e=BdUfPW>


prosilica_exit_core_dump<https://brookhavenlab-my.sharepoint.com/:u:/g/personal/jwlodek_bnl_gov/EWtcZWwNxeVDvi73Dysf_R0BsgAjl9H1wtyQigA1mesSOQ?e=BdUfPW>

Shared via OneDrive



Thanks for the help,



Jakub



________________________________

From: Mark Rivers <rivers@cars.uchicago.edu<mailto:rivers@cars.uchicago.edu>>
Sent: Saturday, January 19, 2019 3:30:55 PM
To: Wlodek, Jakub
Cc: Shen, Guobao; tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov>
Subject: Re: Segmentation Fault when exiting certain Area Detector IOCs



To get a core dump you need to set the shell limit for core size.

In csh the command is
limit core 1000000

In bash
ulimit -c 1000000

Then you will get a core dump when it crashes.

Mark


Sent from my iPhone

On Jan 19, 2019, at 11:38 AM, Wlodek, Jakub <jwlodek@bnl.gov<mailto:jwlodek@bnl.gov<mailto:jwlodek@bnl.gov%3cmailto:jwlodek@bnl.gov>>> wrote:

Hi all,

Yes that's what I'm running, I'm sorry if I didn't make it clear enough in my original email. I compiled everything with gcc via command line, but when I started getting the Seg Fault I decided to run through the VS Code debugger. Essentially it allows you to point to an executable to run and to give it arguments and then as long as the  source code is in VS Code's path it will allow line by line execution.

I'll be away from the lab until Tuesday for the holiday but I will try capturinv the Seg Fault dump as Guobao suggested.

Thanks everyone for the help,
Jakub
________________________________
From: Shen, Guobao <gshen@anl.gov<mailto:gshen@anl.gov<mailto:gshen@anl.gov%3cmailto:gshen@anl.gov>>>
Sent: Friday, January 18, 2019 4:48:13 PM
To: Rivers, Mark L.; Wlodek, Jakub
Cc: tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov%3cmailto:tech-talk@aps.anl.gov>>
Subject: Re: Segmentation Fault when exiting certain Area Detector IOCs

Hi Mark,
I believe Jakub just uses Visual Studio Code as an IDE environment, and still uses GCC toolchain to compile everything.

Jakub,
To simply the problem, it would be better to run the application from command line without the IDE, in this case, Visual Studio Code.
You can capture the seg fault core dump, and get the stack trace from gdb which Mark asked for.

Thanks,
Guobao

?On 1/18/19, 3:23 PM, "tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov> on behalf of Mark Rivers via Tech-talk<mailto:tech-talk-bounces@aps.anl.gov%3cmailto:tech-talk-bounces@aps.anl.gov%3e%20on%20behalf%20of%20Mark%20Rivers%20via%20Tech-talk>" <tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov%3cmailto:tech-talk-bounces@aps.anl.gov>> on behalf of tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov%3cmailto:tech-talk@aps.anl.gov>>> wrote:

    I don?t know anything about Visual Studio Code, sorry.

    Can I ask why you are not using the gcc compiler and Linux vendor library if you are running on Linux?

    Mark


    Sent from my iPhone

    On Jan 18, 2019, at 10:03 PM, Wlodek, Jakub <jwlodek@bnl.gov<mailto:jwlodek@bnl.gov<mailto:jwlodek@bnl.gov%3cmailto:jwlodek@bnl.gov>><mailto:jwlodek@bnl.gov>> wrote:


    Hi Mark,


    I'm actually running the camera on a Debian 9 machine with Visual Studio Code, which does not have the breadth of debugging features found in standard Visual Studio. I believe Visual Studio is Windows only so I'm not sure if I will be able to get that stack. In VS Code the debugger only allows you to debug a program line by line but doesn't have any output other than a list of active program threads and the terminal to the best of my knowledge.


    Is there another way I could get the call stack?


    Jakub

    ________________________________
    From: Mark Rivers <rivers@cars.uchicago.edu<mailto:rivers@cars.uchicago.edu<mailto:rivers@cars.uchicago.edu%3cmailto:rivers@cars.uchicago.edu>><mailto:rivers@cars.uchicago.edu>>
    Sent: Friday, January 18, 2019 3:27:41 PM
    To: Wlodek, Jakub
    Cc: tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov><mailto:tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov%3cmailto:tech-talk@aps.anl.gov%3e%3cmailto:tech-talk@aps.anl.gov>>
    Subject: Re: Segmentation Fault when exiting certain Area Detector IOCs

    Hi Jakob,

    That is not what I meant by a stack trace. I meant the call stack shown in the Visual Studio debugger, showing which calls led up to the call to pPvt->lock().

    Mark


    Sent from my iPhone

    On Jan 18, 2019, at 7:15 PM, Wlodek, Jakub <jwlodek@bnl.gov<mailto:jwlodek@bnl.gov<mailto:jwlodek@bnl.gov%3cmailto:jwlodek@bnl.gov>><mailto:jwlodek@bnl.gov><mailto:jwlodek@bnl.gov>> wrote:


    Hi Mark,


    Here is the stack trace from the call when exiting from the prosilica camera I have been using (With all debugging masks turned on):

    epics> exit
    Disconnecting camera CAM
    2019/01/18 13:14:14.101 [CAM,-1,0] [../prosilica.cpp:1235] [_main_,0x563326a1f060,0] prosilica:disconnectCamera: disconnecting camera 105850
    2019/01/18 13:14:14.101 [CAM,-1,0] [../../asyn/asynRecord/asynRecord.c:905] [_main_,0x563326a1f060,0] XF:10IDC-BI{GC1380-Cam:1}cam1:AsynIO: exception 0, asynExceptionConnect
    2019/01/18 13:14:14.101 [CAM,-1,0] [../prosilica.cpp:1262] [_main_,0x563326a1f060,0] prosilica:disconnectCamera: Camera disconnected; unique id: 105850
    Uninitializing PvAPI
    prosilicaApp: abort due to signal 11 (SIGSEGV) "Segmentation Fault"...
    Aborted


    When I exit from the driver I've written, I get the following:

    epics> exit
    ./startEPICS.sh: line 3: 30814 Segmentation fault      (core dumped) ../../bin/linux-x86_64/adUVCApp st.cmd


    In both cases the SegFault seems to happen every time I exit from this IOC, and in both cases I can trace it to the pPvt->lock(); line.

    As for the versions,
    ADCore 3.4.0
    ADProsilica 2.5.0
    asyn 4.34

    These should all be the most recent versions, as they were all pulled and compiled from Github on Wednesday, but I have seen this issue with this camera on earlier versions as well.

    Thanks for the help,
    Jakub



    ________________________________
    From: Mark Rivers <rivers@cars.uchicago.edu<mailto:rivers@cars.uchicago.edu<mailto:rivers@cars.uchicago.edu%3cmailto:rivers@cars.uchicago.edu>><mailto:rivers@cars.uchicago.edu><mailto:rivers@cars.uchicago.edu>>
    Sent: Friday, January 18, 2019 12:31:13 PM
    To: Wlodek, Jakub
    Cc: tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov><mailto:tech-talk@aps.anl.gov><mailto:tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov%3cmailto:tech-talk@aps.anl.gov%3e%3cmailto:tech-talk@aps.anl.gov%3e%3cmailto:tech-talk@aps.anl.gov>>
    Subject: Re: Segmentation Fault when exiting certain Area Detector IOCs

    Also please tell me what versions of asyn, ADCore, and ADProsilica you are using.


    Mark



    ________________________________
    From: tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov><mailto:tech-talk-bounces@aps.anl.gov><mailto:tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov%3cmailto:tech-talk-bounces@aps.anl.gov%3e%3cmailto:tech-talk-bounces@aps.anl.gov%3e%3cmailto:tech-talk-bounces@aps.anl.gov>> <tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov%3cmailto:tech-talk-bounces@aps.anl.gov>><mailto:tech-talk-bounces@aps.anl.gov><mailto:tech-talk-bounces@aps.anl.gov>> on behalf of Mark Rivers via Tech-talk <tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov%3cmailto:tech-talk@aps.anl.gov>><mailto:tech-talk@aps.anl.gov><mailto:tech-talk@aps.anl.gov>>
    Sent: Friday, January 18, 2019 11:26 AM
    To: Wlodek, Jakub
    Cc: tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov><mailto:tech-talk@aps.anl.gov><mailto:tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov%3cmailto:tech-talk@aps.anl.gov%3e%3cmailto:tech-talk@aps.anl.gov%3e%3cmailto:tech-talk@aps.anl.gov>>
    Subject: Re: Segmentation Fault when exiting certain Area Detector IOCs

    Hi Jacob,


    > Segmentation Fault: line of st.cmd file that calls ioc binary.


    Does this happen every time you exit the IOC, or only occasionally?


    I just tested a ADProsilica on Linux several times and I don't get this error, I see the following:


    epics> exit

    Disconnecting camera PS1

    Uninitializing PvAPI


    > and it occurs in the line pPvt->lock(), meaning that most likely pPvt is NULL.

    > Is there anything I should check for that could be causing this?


    I suspect that what is happening is that the asynPortDriver destructor has been called, so asynPortDriver::mutexId has been destroyed.   That causes pPvt->lock() to seg fault.  The question is why the connect() method is being called after the destructor has been called.  This seems like things are not shutting down in the correct order.  Can you send a more complete stack trace when the error occurs so we can see where connect() is being called from?


    Mark



    ________________________________
    From: tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov><mailto:tech-talk-bounces@aps.anl.gov><mailto:tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov%3cmailto:tech-talk-bounces@aps.anl.gov%3e%3cmailto:tech-talk-bounces@aps.anl.gov%3e%3cmailto:tech-talk-bounces@aps.anl.gov>> <tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov%3cmailto:tech-talk-bounces@aps.anl.gov>><mailto:tech-talk-bounces@aps.anl.gov><mailto:tech-talk-bounces@aps.anl.gov>> on behalf of Wlodek, Jakub via Tech-talk <tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov%3cmailto:tech-talk@aps.anl.gov>><mailto:tech-talk@aps.anl.gov><mailto:tech-talk@aps.anl.gov>>
    Sent: Friday, January 18, 2019 10:35 AM
    To: tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov><mailto:tech-talk@aps.anl.gov><mailto:tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov%3cmailto:tech-talk@aps.anl.gov%3e%3cmailto:tech-talk@aps.anl.gov%3e%3cmailto:tech-talk@aps.anl.gov>>
    Subject: Segmentation Fault when exiting certain Area Detector IOCs


    Hello EPICS Tech-Talk!


    I have a question regarding a segmentation fault I am getting when exiting certain area detector IOCs. Thus far, I have noticed it on several detectors, including a prosilica IOC and a new driver I have developed for USB Video Class cameras. In the IOC shell, the only information given is:


    Segmentation Fault: line of st.cmd file that calls ioc binary.


    Using the Visual Studio Code debugger I was able to trace the segmentation fault to the following function in the asynPortDriver.cpp file:


    extern "C" {static asynStatus connect(void *drvPvt, asynUser *pasynUser)
    {
        asynPortDriver *pPvt = (asynPortDriver *)drvPvt;
        asynStatus status;

        pPvt->lock();
        status = pPvt->connect(pasynUser);
        pPvt->unlock();
        return(status);
    }}



    and it occurs in the line pPvt->lock(), meaning that most likely pPvt is NULL. Is there anything I should check for that could be causing this?


    Thank you all in advance,


    Jakub Wlodek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.aps.anl.gov/pipermail/tech-talk/attachments/20190123/24bd7d55/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 823 bytes
Desc: image001.jpg
URL: <http://mailman.aps.anl.gov/pipermail/tech-talk/attachments/20190123/24bd7d55/attachment.jpg>

------------------------------

_______________________________________________
Tech-talk mailing list Tech-talk@aps.anl.gov https://mailman.aps.anl.gov/mailman/listinfo/tech-talk


End of Tech-talk Digest, Vol 13, Issue 40
*****************************************

Replies:
RE: Segmentation Fault when exiting certain Area Detector Mark Rivers via Tech-talk

Navigate by Date:
Prev: RE: Segmentation Fault when exiting certain Area Detector IOCs Mark Rivers via Tech-talk
Next: Raspberry Pi 3 B+ Control System Studio Kevin Ramos via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020 
Navigate by Thread:
Prev: Re: Simple alarm monitor Bo Jakobsen via Tech-talk
Next: RE: Segmentation Fault when exiting certain Area Detector Mark Rivers via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020 
ANJ, 24 Jan 2019 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·