RE: Problem with autosave module making it crash at random basis.

["Abdalla Ahmad" <abdalla.ahmad@sesame.org.jo>]

OK I misunderstood your instructions.

I re-ran gdb as you instructed and I got the attached gdb output (not for the autosave crash). My question is, despite I got the source file where the crash happened, I still got missing debug info warnings for a lot of libraries. Should I try to rebuild everything with debug info integrated or this is sufficient?

Best Regards,
Abdalla.

-----Original Message-----
From: Abdalla Ahmad 
Sent: Thursday, April 26, 2018 8:18 AM
To: 'Mark Rivers' <rivers@cars.uchicago.edu>; tech-talk@aps.anl.gov
Subject: RE: Problem with autosave module making it crash at random basis.

Gdb has been started like this

gdb './bin/linux-x86/ioc iocBoot/ioc/st.cmd' crash.xxxxxxx

Unfortunately no ,I did not try more recent version of autosave. The IOC crashed this time in an SNMP driver, so I will have to keep debugging this.

Best Regards,
Abdalla.

-----Original Message-----
From: Mark Rivers [mailto:rivers@cars.uchicago.edu] 
Sent: Wednesday, April 25, 2018 2:48 PM
To: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo>; tech-talk@aps.anl.gov
Subject: Re: Problem with autosave module making it crash at random basis.

gdb is printing this message near the beginning:


./bin/linux-x86/ioc iocBoot/ioc/st.cmd: No such file or directory.


It has none of your symbols.  Please send again showing the line where you started gdb.


Have you tried a more recent version of autosave?


Mark



________________________________
From: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo>
Sent: Wednesday, April 25, 2018 12:52 AM
To: Mark Rivers; tech-talk@aps.anl.gov
Subject: RE: Problem with autosave module making it crash at random basis.

Hello Mark

Please find the attached output from gdb when running it with the core dump.

Best Regards,
Abdalla.

-----Original Message-----
From: Abdalla Ahmad
Sent: Wednesday, April 18, 2018 2:58 PM
To: 'Mark Rivers' <rivers@cars.uchicago.edu>; tech-talk@aps.anl.gov
Subject: RE: Problem with autosave module making it crash at random basis.

Unfortunately I deleted the core dump. Once the IOC crash again I will send the complete output.

Abdalla.

-----Original Message-----
From: Mark Rivers [mailto:rivers@cars.uchicago.edu]
Sent: Wednesday, April 18, 2018 2:58 PM
To: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo>; tech-talk@aps.anl.gov
Subject: Re: Problem with autosave module making it crash at random basis.

> The core dump is generated but GDB could not show a backtrace and reported "missing separate debuginfo for the main executable file". Is it the case that the IOC should be built with debug symbols?


I have not seen that when building with the standard non-debug EPICS_HOST_ARCH of linux-x86_64.  It will warn that some system libraries don't have debug symbols, and some variables will be shown as "optimized out", but I have always been able to get a backtrace when it crashed.


Can you send the complete output when you start gdb and try to do a backtrace?


Mark




________________________________
From: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo>
Sent: Wednesday, April 18, 2018 12:46 AM
To: Mark Rivers; tech-talk@aps.anl.gov
Subject: RE: Problem with autosave module making it crash at random basis.

Hi Mark

The core dump is generated but GDB could not show a backtrace and reported "missing separate debuginfo for the main executable file". Is it the case that the IOC should be built with debug symbols?

Best Regards,
Abdalla.

-----Original Message-----
From: Mark Rivers [mailto:rivers@cars.uchicago.edu]
Sent: Sunday, April 08, 2018 4:01 PM
To: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo>; tech-talk@aps.anl.gov
Subject: Re: Problem with autosave module making it crash at random basis.

Hi Abdalla,


The latest autosave Makefile contains these lines:


# To enable autosaveBuild, use dbLoadRecordsHookRegister().  (Requires EPICS # base 3.14.12.5 or later.) USR_CFLAGS += -DDBLOADRECORDSHOOKREGISTER

Since you are running 3.14.12.3 I think you should try commenting out that USR_CFLAGS line.


With your older version of autosave your output showed that valgrind itself had crashed.  You should be able to get a traceback without valgrind, just by enabling core dumps.  Type the following commands:


On bash shell:
ulimit -c 10000000

or on csh shell:
limit core 1000000

Then run the IOC application.  When it crashes it will create a core.xxxxxx file.

Then run the Gnu debugger:
gdb path/to/your/IOC/executable core.xxxxxx

You need to provide the path to your IOC executable and replace core.xxxxxx with the name of the core file it just created.

Once you are in the debugger type:
backtrace

It will show a stack dump to where it crashed.

Mark







________________________________
From: tech-talk-bounces@aps.anl.gov <tech-talk-bounces@aps.anl.gov> on behalf of Abdalla Ahmad <abdalla.ahmad@sesame.org.jo>
Sent: Sunday, April 8, 2018 1:06 AM
To: tech-talk@aps.anl.gov
Subject: Problem with autosave module making it crash at random basis.


Hi



One of our IOCs crashed and it appears to do so from time to time. I used valgrind on the IOC process and the attached file contains valgrind output right when the IOC crashed. The first 2 lines are the IOC shell output before the segmentation fault happened. I could not figure out whether the problem from the module sending something corrupted to the printf functions or the standard C library itself is somehow corrupted. I tried cloning the latest autosave version from github but the build failed. Build output is in the attached too.



OS: Scientific Linux 6.4 32-bit.

EPICS: 3.14.12.3

SynApps 5.6

Autosave 4.8



Best Regards,



Abdalla Ahmad

Control Engineer

SESAME

Allan, Jordan.

Tel: (+962-5) 3511348 , ext. 265

Fax: (+962-5) 3511423

Mob: (+962-7)88183296

www.sesame.org.jo<http://www.sesame.org.jo/>


GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/control/top/bin/linux-x86/ioc...done.
[New Thread 32361]
[New Thread 32418]
[New Thread 32424]
[New Thread 32430]
[New Thread 32436]
[New Thread 32437]
[New Thread 32438]
[New Thread 32439]
[New Thread 32440]
[New Thread 32441]
.
.
.
.
.
[New Thread 32410]                                                         [147/28620]
[New Thread 32388]
[New Thread 32390]
Missing separate debuginfo for /home/control/epics/support/synApps_5_6/support/stream-
2-5-1/lib/linux-x86/libstream.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/4$
/45e0adc47f1ec9892559ad05029a438d1f9155
Missing separate debuginfo for /home/control/epics/support/synApps_5_6/support/asyn-4$
18/lib/linux-x86/libasyn.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/9$
/d0d18a17a5839404f0aeed997fc3a4342eb877
Missing separate debuginfo for /home/control/epics/support/proxy/lib/linux-x86/libpro$
y.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/6$
/000cd38df2a9d885ca775582c51c612f69a718
Missing separate debuginfo for /home/control/epics/support/mbbi2/lib/linux-x86/libmbb$
2.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/5$
/6d42de0bb141f9b5a65d016bf67619c27a49fe
Missing separate debuginfo for /home/control/epics/support/snmp-dls/lib/linux-x86/lib$
evSnmp.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/c$
/f24ed2d3a21511a256ad78c807cb17f71d20fd
Missing separate debuginfo for /home/control/epics/support/synApps_5_6/support/seq-2-$
-3/lib/linux-x86/libseq.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/c$
/a2dd933f90228a6d3c29e36eb9af3051d76568
Missing separate debuginfo for /home/control/epics/support/synApps_5_6/support/seq-2-$
-3/lib/linux-x86/libpv.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/4$
/34c6639f3893226b6909baf51671a1f963da00
Missing separate debuginfo for /home/control/epics/support/autosave/lib/linux-x86/lib$
utosave.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/9$
/6c3493510986245273cb0c308a16dc13e0aeee
Missing separate debuginfo for /home/control/epics/support/synApps_5_6/support/devIoc$
tats-3-1-7/lib/linux-x86/libdevIocStats.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/7$
/9bddab870178007111cf7aaea244f438486d03
Missing separate debuginfo for /home/control/epics/base-3.14.12.3/lib/linux-x86/libre$
Ioc.so.3.14
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/8$
/3ebba14f65ab2dae4b9fe57affaf1cf5cf72d3
Missing separate debuginfo for /home/control/epics/base-3.14.12.3/lib/linux-x86/libso$
tDevIoc.so.3.14
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/3$
/14d64e61e0a8613c50fdee811d4ee38f88e467
Missing separate debuginfo for /home/control/epics/base-3.14.12.3/lib/linux-x86/libmi$
cIoc.so.3.14
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/4$
/0d913debfc45846016200696b3ce2fea9ceac3
Missing separate debuginfo for /home/control/epics/base-3.14.12.3/lib/linux-x86/librs$
vIoc.so.3.14
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/c$
/46c4d243ab1359e6dce2971f9c096b72df4115
Missing separate debuginfo for /home/control/epics/base-3.14.12.3/lib/linux-x86/libdb$
oolsIoc.so.3.14
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.b[90/28620]
/60eba943f4a915423dac6713ab0d79bec29ee7
Missing separate debuginfo for /home/control/epics/base-3.14.12.3/lib/linux-x86/libasI
oc.so.3.14
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/9c
/9977652018ba52482d0efc80fbc86da98aff72
Missing separate debuginfo for /home/control/epics/base-3.14.12.3/lib/linux-x86/libdbI
oc.so.3.14
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/b8
/9ec772de8d86928cfb88f69d4101b31cf0b08e
Missing separate debuginfo for /home/control/epics/base-3.14.12.3/lib/linux-x86/libreg
istryIoc.so.3.14
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/fd
/aafe9ddf83ff9339ac40f7494d902866e7d613
Missing separate debuginfo for /home/control/epics/base-3.14.12.3/lib/linux-x86/libdbS
taticIoc.so.3.14
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/3d
/90fa1ddb8770c7742922788311dc76008b83cf
Missing separate debuginfo for /home/control/epics/base-3.14.12.3/lib/linux-x86/libca.
so.3.14
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/a6
/21171b5fc9d19c9e4c20a6945096719a00117f
Missing separate debuginfo for /home/control/epics/base-3.14.12.3/lib/linux-x86/libCom
.so.3.14
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/0e
/c61080c5a2308cc99cc41f0cf8cabff4c9298e
Missing separate debuginfo for 
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/b3
/8451d0d15ef63514dba9d74da36a4df6f92d97
Reading symbols from /home/control/epics/support/synApps_5_6/support/stream-2-5-1/lib/
linux-x86/libstream.so...done.
Loaded symbols for /home/control/epics/support/synApps_5_6/support/stream-2-5-1/lib/li
nux-x86/libstream.so
Reading symbols from /home/control/epics/support/synApps_5_6/support/asyn-4-18/lib/lin
ux-x86/libasyn.so...done.
Loaded symbols for /home/control/epics/support/synApps_5_6/support/asyn-4-18/lib/linux
-x86/libasyn.so
Reading symbols from /home/control/epics/support/proxy/lib/linux-x86/libproxy.so...don
e.
Loaded symbols for /home/control/epics/support/proxy/lib/linux-x86/libproxy.so
Reading symbols from /home/control/epics/support/mbbi2/lib/linux-x86/libmbbi2.so...don
e.
Loaded symbols for /home/control/epics/support/mbbi2/lib/linux-x86/libmbbi2.so
Reading symbols from /home/control/epics/support/snmp-dls/lib/linux-x86/libdevSnmp.so.
..done.
Loaded symbols for /home/control/epics/support/snmp-dls/lib/linux-x86/libdevSnmp.so
Reading symbols from /home/control/epics/support/synApps_5_6/support/seq-2-1-3/lib/lin
ux-x86/libseq.so...done.
Loaded symbols for /home/control/epics/support/synApps_5_6/support/seq-2-1-3/lib/linux
-x86/libseq.so
Reading symbols from /home/control/epics/support/synApps_5_6/support/seq-2-1-3/lib/lin
ux-x86/libpv.so...(no debugging symbols found)...done.
Loaded symbols for /home/control/epics/support/synApps_5_6/support/seq-2-1-3/lib/linux
-x86/libpv.so
Reading symbols from /home/control/epics/support/autosave/lib/linux-x86/libautosave.so
...done.
Reading symbols from /home/control/epics/support/synApps_5_6/support/devIocStats-3-1-7
/lib/linux-x86/libdevIocStats.so...done.
Loaded symbols for /home/control/epics/support/synApps_5_6/support/devIocStats-3-1-7/l
ib/linux-x86/libdevIocStats.so
Reading symbols from /home/control/epics/base-3.14.12.3/lib/linux-x86/librecIoc.so.3.1
4...done.
Loaded symbols for /home/control/epics/base-3.14.12.3/lib/linux-x86/librecIoc.so.3.14
Reading symbols from /home/control/epics/base-3.14.12.3/lib/linux-x86/libsoftDevIoc.so
.3.14...done.
Loaded symbols for /home/control/epics/base-3.14.12.3/lib/linux-x86/libsoftDevIoc.so.3
.14
Reading symbols from /home/control/epics/base-3.14.12.3/lib/linux-x86/libmiscIoc.so.3.
14...done.
Loaded symbols for /home/control/epics/base-3.14.12.3/lib/linux-x86/libmiscIoc.so.3.14
Reading symbols from /home/control/epics/base-3.14.12.3/lib/linux-x86/librsrvIoc.so.3.
14...done.
Loaded symbols for /home/control/epics/base-3.14.12.3/lib/linux-x86/librsrvIoc.so.3.14
Reading symbols from /home/control/epics/base-3.14.12.3/lib/linux-x86/libdbtoolsIoc.so
.3.14...done.
Loaded symbols for /home/control/epics/base-3.14.12.3/lib/linux-x86/libdbtoolsIoc.so.3
.14
Reading symbols from /home/control/epics/base-3.14.12.3/lib/linux-x86/libasIoc.so.3.14
...done.
Loaded symbols for /home/control/epics/base-3.14.12.3/lib/linux-x86/libasIoc.so.3.14
Reading symbols from /home/control/epics/base-3.14.12.3/lib/linux-x86/libdbIoc.so.3.14
...done.
Loaded symbols for /home/control/epics/base-3.14.12.3/lib/linux-x86/libdbIoc.so.3.14
Reading symbols from /home/control/epics/base-3.14.12.3/lib/linux-x86/libregistryIoc.s
o.3.14...done.
Loaded symbols for /home/control/epics/base-3.14.12.3/lib/linux-x86/libregistryIoc.so.
3.14
Reading symbols from /home/control/epics/base-3.14.12.3/lib/linux-x86/libdbStaticIoc.s
o.3.14...done.
Loaded symbols for /home/control/epics/base-3.14.12.3/lib/linux-x86/libdbStaticIoc.so.
3.14
Reading symbols from /home/control/epics/base-3.14.12.3/lib/linux-x86/libca.so.3.14...
(no debugging symbols found)...done.
Loaded symbols for /home/control/epics/base-3.14.12.3/lib/linux-x86/libca.so.3.14
Reading symbols from /home/control/epics/base-3.14.12.3/lib/linux-x86/libCom.so.3.14..
.done.
Loaded symbols for /home/control/epics/base-3.14.12.3/lib/linux-x86/libCom.so.3.14
Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libreadline.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libreadline.so.6
Reading symbols from /lib/libncurses.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib/libncurses.so.5
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /usr/lib/libnetsnmp.so.20...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libnetsnmp.so.20
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libtinfo.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib/libtinfo.so.5
Reading symbols from /usr/lib/libcrypto.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libcrypto.so.10
Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_dns.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_dns.so.2
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
Core was generated by `./bin/linux-x86/ioc iocBoot/ioc/st.cmd'.
Program terminated with signal 11, Segmentation fault.
#0  0x00399969 in Snmp_Operation (pSnmpAgent=0x8dd2510) at ../drvSnmp.c:345
345                             pVar = respQryPdu->variables;
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.107.el6_4.5.i686 libg
cc-4.4.7-3.el6.i686 libstdc++-4.4.7-3.el6.i686 ncurses-libs-5.7-3.20090208.el6.i686 ne
t-snmp-libs-5.5-44.el6_4.4.i686 openssl-1.0.0-27.el6_4.2.i686 readline-6.0-4.el6.i686 
zlib-1.2.3-29.el6.i686
(gdb) backtrace
#0  0x00399969 in Snmp_Operation (pSnmpAgent=0x8dd2510) at ../drvSnmp.c:345
#1  0x00a697fa in start_routine (arg=0x8d92cf0) at ../../../src/libCom/osi/os/posix/os
dThread.c:385
#2  0x00307a49 in start_thread () from /lib/libpthread.so.0
#3  0x00640aae in clone () from /lib/libc.so.6
(gdb)