RE: Segmentation Fault when exiting certain Area Detector IOCs

[Mark Rivers via Tech-talk <tech-talk@aps.anl.gov>]

Hi Jakob,

 

I don’t actually need the core file itself.  I need you to run gdb on that core file yourself and the output.  You do that as follows:

 

gdb path_to_your prosilica_IOC_application core.xxxxx

 

where core.xxxxx is the name of the core file.

 

For example:

 

gdb /home/epics/support/areaDetector-3-4-0/ADProsilica/iocs/prosilicaIOC/bin/linux-x86_64/prosilicaApp core.12345

 

When you run that you will get the gdb prompt.  At the prompt type the following command:

 

bt

 

Send the output of that command.

 

Mark

 

 

From: Wlodek, Jakub <jwlodek@bnl.gov>
Sent: Tuesday, January 22, 2019 8:40 AM
To: Mark Rivers <rivers@cars.uchicago.edu>
Cc: Shen, Guobao <gshen@anl.gov>; tech-talk@aps.anl.gov
Subject: Re: Segmentation Fault when exiting certain Area Detector IOCs

 

Hi all,

 

I ran the prosilica IOC and generated the following core dump:

 

prosilica_exit_core_dump

prosilica_exit_core_dump Shared via OneDrive

Thanks for the help,

 

Jakub

 

---

From: Mark Rivers <rivers@cars.uchicago.edu>
Sent: Saturday, January 19, 2019 3:30:55 PM
To: Wlodek, Jakub
Cc: Shen, Guobao; tech-talk@aps.anl.gov
Subject: Re: Segmentation Fault when exiting certain Area Detector IOCs

 

To get a core dump you need to set the shell limit for core size.

In csh the command is
limit core 1000000

In bash
ulimit -c 1000000

Then you will get a core dump when it crashes.

Mark


Sent from my iPhone

On Jan 19, 2019, at 11:38 AM, Wlodek, Jakub <jwlodek@bnl.gov<mailto:jwlodek@bnl.gov>> wrote:

Hi all,

Yes that's what I'm running, I'm sorry if I didn't make it clear enough in my original email. I compiled everything with gcc via command line, but when I started getting the Seg Fault I decided to run through the VS Code debugger. Essentially it allows you to point to an executable to run and to give it arguments and then as long as the  source code is in VS Code's path it will allow line by line execution.

I'll be away from the lab until Tuesday for the holiday but I will try capturinv the Seg Fault dump as Guobao suggested.

Thanks everyone for the help,
Jakub
________________________________
From: Shen, Guobao <gshen@anl.gov<mailto:gshen@anl.gov>>
Sent: Friday, January 18, 2019 4:48:13 PM
To: Rivers, Mark L.; Wlodek, Jakub
Cc: tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov>
Subject: Re: Segmentation Fault when exiting certain Area Detector IOCs

Hi Mark,
I believe Jakub just uses Visual Studio Code as an IDE environment, and still uses GCC toolchain to compile everything.

Jakub,
To simply the problem, it would be better to run the application from command line without the IDE, in this case, Visual Studio Code.
You can capture the seg fault core dump, and get the stack trace from gdb which Mark asked for.

Thanks,
Guobao

On 1/18/19, 3:23 PM, "tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov> on behalf of Mark Rivers via Tech-talk" <tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov> on behalf of tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov>> wrote:

    I don’t know anything about Visual Studio Code, sorry.

    Can I ask why you are not using the gcc compiler and Linux vendor library if you are running on Linux?

    Mark


    Sent from my iPhone

    On Jan 18, 2019, at 10:03 PM, Wlodek, Jakub <jwlodek@bnl.gov<mailto:jwlodek@bnl.gov><mailto:jwlodek@bnl.gov>> wrote:


    Hi Mark,


    I'm actually running the camera on a Debian 9 machine with Visual Studio Code, which does not have the breadth of debugging features found in standard Visual Studio. I believe Visual Studio is Windows only so I'm not sure if I will be able to get that stack. In VS Code the debugger only allows you to debug a program line by line but doesn't have any output other than a list of active program threads and the terminal to the best of my knowledge.


    Is there another way I could get the call stack?


    Jakub

    ________________________________
    From: Mark Rivers <rivers@cars.uchicago.edu<mailto:rivers@cars.uchicago.edu><mailto:rivers@cars.uchicago.edu>>
    Sent: Friday, January 18, 2019 3:27:41 PM
    To: Wlodek, Jakub
    Cc: tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov><mailto:tech-talk@aps.anl.gov>
    Subject: Re: Segmentation Fault when exiting certain Area Detector IOCs

    Hi Jakob,

    That is not what I meant by a stack trace. I meant the call stack shown in the Visual Studio debugger, showing which calls led up to the call to pPvt->lock().

    Mark


    Sent from my iPhone

    On Jan 18, 2019, at 7:15 PM, Wlodek, Jakub <jwlodek@bnl.gov<mailto:jwlodek@bnl.gov><mailto:jwlodek@bnl.gov><mailto:jwlodek@bnl.gov>> wrote:


    Hi Mark,


    Here is the stack trace from the call when exiting from the prosilica camera I have been using (With all debugging masks turned on):

    epics> exit
    Disconnecting camera CAM
    2019/01/18 13:14:14.101 [CAM,-1,0] [../prosilica.cpp:1235] [_main_,0x563326a1f060,0] prosilica:disconnectCamera: disconnecting camera 105850
    2019/01/18 13:14:14.101 [CAM,-1,0] [../../asyn/asynRecord/asynRecord.c:905] [_main_,0x563326a1f060,0] XF:10IDC-BI{GC1380-Cam:1}cam1:AsynIO: exception 0, asynExceptionConnect
    2019/01/18 13:14:14.101 [CAM,-1,0] [../prosilica.cpp:1262] [_main_,0x563326a1f060,0] prosilica:disconnectCamera: Camera disconnected; unique id: 105850
    Uninitializing PvAPI
    prosilicaApp: abort due to signal 11 (SIGSEGV) "Segmentation Fault"...
    Aborted


    When I exit from the driver I've written, I get the following:

    epics> exit
    ./startEPICS.sh: line 3: 30814 Segmentation fault      (core dumped) ../../bin/linux-x86_64/adUVCApp st.cmd


    In both cases the SegFault seems to happen every time I exit from this IOC, and in both cases I can trace it to the pPvt->lock(); line.

    As for the versions,
    ADCore 3.4.0
    ADProsilica 2.5.0
    asyn 4.34

    These should all be the most recent versions, as they were all pulled and compiled from Github on Wednesday, but I have seen this issue with this camera on earlier versions as well.

    Thanks for the help,
    Jakub



    ________________________________
    From: Mark Rivers <rivers@cars.uchicago.edu<mailto:rivers@cars.uchicago.edu><mailto:rivers@cars.uchicago.edu><mailto:rivers@cars.uchicago.edu>>
    Sent: Friday, January 18, 2019 12:31:13 PM
    To: Wlodek, Jakub
    Cc: tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov><mailto:tech-talk@aps.anl.gov><mailto:tech-talk@aps.anl.gov>
    Subject: Re: Segmentation Fault when exiting certain Area Detector IOCs

    Also please tell me what versions of asyn, ADCore, and ADProsilica you are using.


    Mark



    ________________________________
    From: tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov><mailto:tech-talk-bounces@aps.anl.gov><mailto:tech-talk-bounces@aps.anl.gov> <tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov><mailto:tech-talk-bounces@aps.anl.gov><mailto:tech-talk-bounces@aps.anl.gov>> on behalf of Mark Rivers via Tech-talk <tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov><mailto:tech-talk@aps.anl.gov><mailto:tech-talk@aps.anl.gov>>
    Sent: Friday, January 18, 2019 11:26 AM
    To: Wlodek, Jakub
    Cc: tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov><mailto:tech-talk@aps.anl.gov><mailto:tech-talk@aps.anl.gov>
    Subject: Re: Segmentation Fault when exiting certain Area Detector IOCs

    Hi Jacob,


    > Segmentation Fault: line of st.cmd file that calls ioc binary.


    Does this happen every time you exit the IOC, or only occasionally?


    I just tested a ADProsilica on Linux several times and I don't get this error, I see the following:


    epics> exit

    Disconnecting camera PS1

    Uninitializing PvAPI


    > and it occurs in the line pPvt->lock(), meaning that most likely pPvt is NULL.

    > Is there anything I should check for that could be causing this?


    I suspect that what is happening is that the asynPortDriver destructor has been called, so asynPortDriver::mutexId has been destroyed.   That causes pPvt->lock() to seg fault.  The question is why the connect() method is being called after the destructor has been called.  This seems like things are not shutting down in the correct order.  Can you send a more complete stack trace when the error occurs so we can see where connect() is being called from?


    Mark



    ________________________________
    From: tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov><mailto:tech-talk-bounces@aps.anl.gov><mailto:tech-talk-bounces@aps.anl.gov> <tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov><mailto:tech-talk-bounces@aps.anl.gov><mailto:tech-talk-bounces@aps.anl.gov>> on behalf of Wlodek, Jakub via Tech-talk <tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov><mailto:tech-talk@aps.anl.gov><mailto:tech-talk@aps.anl.gov>>
    Sent: Friday, January 18, 2019 10:35 AM
    To: tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov><mailto:tech-talk@aps.anl.gov><mailto:tech-talk@aps.anl.gov>
    Subject: Segmentation Fault when exiting certain Area Detector IOCs


    Hello EPICS Tech-Talk!


    I have a question regarding a segmentation fault I am getting when exiting certain area detector IOCs. Thus far, I have noticed it on several detectors, including a prosilica IOC and a new driver I have developed for USB Video Class cameras. In the IOC shell, the only information given is:


    Segmentation Fault: line of st.cmd file that calls ioc binary.


    Using the Visual Studio Code debugger I was able to trace the segmentation fault to the following function in the asynPortDriver.cpp file:


    extern "C" {static asynStatus connect(void *drvPvt, asynUser *pasynUser)
    {
        asynPortDriver *pPvt = (asynPortDriver *)drvPvt;
        asynStatus status;

        pPvt->lock();
        status = pPvt->connect(pasynUser);
        pPvt->unlock();
        return(status);
    }}



    and it occurs in the line pPvt->lock(), meaning that most likely pPvt is NULL. Is there anything I should check for that could be causing this?


    Thank you all in advance,


    Jakub Wlodek