Re: Problem with autosave module making it crash at random basis.

[Mark Rivers <rivers@cars.uchicago.edu>]

Hi Abdalla,


Did you build everything from source (asyn, seq, etc.)

You do first get this warning:

Missing separate debuginfo for /home/control/epics/support/autosave/lib/linux-x86/libautosave.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/9/6c3493510986245273cb0c308a16dc13e0aeee

But you then get this:
Reading symbols from /home/control/epics/support/autosave/lib/linux-x86/libautosave.so
...done.

The traceback now looks reasonable.

So I think it is working.  If you still have the autosave crash file you can run gdb on it as well.

I see that you are running very old versions of other modules as well asyn-4-18 is 6.5 years old.  Is there a reason you don't switch to more recent versions?

Mark


________________________________
From: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo>
Sent: Thursday, April 26, 2018 12:33 AM
To: Mark Rivers; tech-talk@aps.anl.gov
Subject: RE: Problem with autosave module making it crash at random basis.

OK I misunderstood your instructions.

I re-ran gdb as you instructed and I got the attached gdb output (not for the autosave crash). My question is, despite I got the source file where the crash happened, I still got missing debug info warnings for a lot of libraries. Should I try to rebuild everything with debug info integrated or this is sufficient?

Best Regards,
Abdalla.

-----Original Message-----
From: Abdalla Ahmad
Sent: Thursday, April 26, 2018 8:18 AM
To: 'Mark Rivers' <rivers@cars.uchicago.edu>; tech-talk@aps.anl.gov
Subject: RE: Problem with autosave module making it crash at random basis.

Gdb has been started like this

gdb './bin/linux-x86/ioc iocBoot/ioc/st.cmd' crash.xxxxxxx

Unfortunately no ,I did not try more recent version of autosave. The IOC crashed this time in an SNMP driver, so I will have to keep debugging this.

Best Regards,
Abdalla.

-----Original Message-----
From: Mark Rivers [mailto:rivers@cars.uchicago.edu]
Sent: Wednesday, April 25, 2018 2:48 PM
To: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo>; tech-talk@aps.anl.gov
Subject: Re: Problem with autosave module making it crash at random basis.

gdb is printing this message near the beginning:


./bin/linux-x86/ioc iocBoot/ioc/st.cmd: No such file or directory.


It has none of your symbols.  Please send again showing the line where you started gdb.


Have you tried a more recent version of autosave?


Mark



________________________________
From: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo>
Sent: Wednesday, April 25, 2018 12:52 AM
To: Mark Rivers; tech-talk@aps.anl.gov
Subject: RE: Problem with autosave module making it crash at random basis.

Hello Mark

Please find the attached output from gdb when running it with the core dump.

Best Regards,
Abdalla.

-----Original Message-----
From: Abdalla Ahmad
Sent: Wednesday, April 18, 2018 2:58 PM
To: 'Mark Rivers' <rivers@cars.uchicago.edu>; tech-talk@aps.anl.gov
Subject: RE: Problem with autosave module making it crash at random basis.

Unfortunately I deleted the core dump. Once the IOC crash again I will send the complete output.

Abdalla.

-----Original Message-----
From: Mark Rivers [mailto:rivers@cars.uchicago.edu]
Sent: Wednesday, April 18, 2018 2:58 PM
To: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo>; tech-talk@aps.anl.gov
Subject: Re: Problem with autosave module making it crash at random basis.

> The core dump is generated but GDB could not show a backtrace and reported "missing separate debuginfo for the main executable file". Is it the case that the IOC should be built with debug symbols?


I have not seen that when building with the standard non-debug EPICS_HOST_ARCH of linux-x86_64.  It will warn that some system libraries don't have debug symbols, and some variables will be shown as "optimized out", but I have always been able to get a backtrace when it crashed.


Can you send the complete output when you start gdb and try to do a backtrace?


Mark




________________________________
From: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo>
Sent: Wednesday, April 18, 2018 12:46 AM
To: Mark Rivers; tech-talk@aps.anl.gov
Subject: RE: Problem with autosave module making it crash at random basis.

Hi Mark

The core dump is generated but GDB could not show a backtrace and reported "missing separate debuginfo for the main executable file". Is it the case that the IOC should be built with debug symbols?

Best Regards,
Abdalla.

-----Original Message-----
From: Mark Rivers [mailto:rivers@cars.uchicago.edu]
Sent: Sunday, April 08, 2018 4:01 PM
To: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo>; tech-talk@aps.anl.gov
Subject: Re: Problem with autosave module making it crash at random basis.

Hi Abdalla,


The latest autosave Makefile contains these lines:


# To enable autosaveBuild, use dbLoadRecordsHookRegister().  (Requires EPICS # base 3.14.12.5 or later.) USR_CFLAGS += -DDBLOADRECORDSHOOKREGISTER

Since you are running 3.14.12.3 I think you should try commenting out that USR_CFLAGS line.


With your older version of autosave your output showed that valgrind itself had crashed.  You should be able to get a traceback without valgrind, just by enabling core dumps.  Type the following commands:


On bash shell:
ulimit -c 10000000

or on csh shell:
limit core 1000000

Then run the IOC application.  When it crashes it will create a core.xxxxxx file.

Then run the Gnu debugger:
gdb path/to/your/IOC/executable core.xxxxxx

You need to provide the path to your IOC executable and replace core.xxxxxx with the name of the core file it just created.

Once you are in the debugger type:
backtrace

It will show a stack dump to where it crashed.

Mark







________________________________
From: tech-talk-bounces@aps.anl.gov <tech-talk-bounces@aps.anl.gov> on behalf of Abdalla Ahmad <abdalla.ahmad@sesame.org.jo>
Sent: Sunday, April 8, 2018 1:06 AM
To: tech-talk@aps.anl.gov
Subject: Problem with autosave module making it crash at random basis.


Hi



One of our IOCs crashed and it appears to do so from time to time. I used valgrind on the IOC process and the attached file contains valgrind output right when the IOC crashed. The first 2 lines are the IOC shell output before the segmentation fault happened. I could not figure out whether the problem from the module sending something corrupted to the printf functions or the standard C library itself is somehow corrupted. I tried cloning the latest autosave version from github but the build failed. Build output is in the attached too.



OS: Scientific Linux 6.4 32-bit.

EPICS: 3.14.12.3

SynApps 5.6

Autosave 4.8



Best Regards,



Abdalla Ahmad

Control Engineer

SESAME

Allan, Jordan.

Tel: (+962-5) 3511348 , ext. 265

Fax: (+962-5) 3511423

Mob: (+962-7)88183296

www.sesame.org.jo<http://www.sesame.org.jo/>