EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: RE: Problem with autosave module making it crash at random basis.
From: Mark Rivers <[email protected]>
To: "[email protected]" <[email protected]>
Date: Thu, 26 Apr 2018 12:42:59 +0000
> But one of our colleagues tried the latest version of everything in the synApps package in a motion control IOC, what he found is that versions of individual modules are newer than the already packaged synApps collection. 
> Shouldn't the synApps package already contain the latest version of each module?

synApps has not been released very frequently, the last release (5-8) was over 3 years ago.

However, it is now on Github:
https://github.com/epics-synapps

The support repository there contains a script (assemble_synApps.sh) which will download and assemble all of the synApps modules using more recent versions.

In my experience you don't need to wait for an official synApps release.  You can safely use the latest tagged version of each synApps module and they will work together.  If you find otherwise then report the problem and it will be quickly fixed.

Mark


-----Original Message-----
From: Abdalla Ahmad [mailto:[email protected]] 
Sent: Thursday, April 26, 2018 6:34 AM
To: Mark Rivers <[email protected]>
Subject: RE: Problem with autosave module making it crash at random basis.

Hi Mark

Yes everything from source. The reason for such old versions is that we did not pay enough attention to software upgrades because everything was working fine and we were busy with storage-ring installation. When we decided to upgrade everything (OS, epics, modules, etc. ) from beginning of 2018, an active operation plan popped up and so we have to wait for the next long shutdown to do such an upgrades.

But one of our colleagues tried the latest version of everything in the synApps package in a motion control IOC, what he found is that versions of individual modules are newer than the already packaged synApps collection. Shouldn't the synApps package already contain the latest version of each module?

Best Regards,
Abdalla. 

-----Original Message-----
From: Mark Rivers [mailto:[email protected]] 
Sent: Thursday, April 26, 2018 2:29 PM
To: Abdalla Ahmad <[email protected]>; [email protected]
Subject: Re: Problem with autosave module making it crash at random basis.

Hi Abdalla,


Did you build everything from source (asyn, seq, etc.)

You do first get this warning:

Missing separate debuginfo for /home/control/epics/support/autosave/lib/linux-x86/libautosave.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/9/6c3493510986245273cb0c308a16dc13e0aeee

But you then get this:
Reading symbols from /home/control/epics/support/autosave/lib/linux-x86/libautosave.so
...done.

The traceback now looks reasonable.

So I think it is working.  If you still have the autosave crash file you can run gdb on it as well.

I see that you are running very old versions of other modules as well asyn-4-18 is 6.5 years old.  Is there a reason you don't switch to more recent versions?

Mark


________________________________
From: Abdalla Ahmad <[email protected]>
Sent: Thursday, April 26, 2018 12:33 AM
To: Mark Rivers; [email protected]
Subject: RE: Problem with autosave module making it crash at random basis.

OK I misunderstood your instructions.

I re-ran gdb as you instructed and I got the attached gdb output (not for the autosave crash). My question is, despite I got the source file where the crash happened, I still got missing debug info warnings for a lot of libraries. Should I try to rebuild everything with debug info integrated or this is sufficient?

Best Regards,
Abdalla.

-----Original Message-----
From: Abdalla Ahmad
Sent: Thursday, April 26, 2018 8:18 AM
To: 'Mark Rivers' <[email protected]>; [email protected]
Subject: RE: Problem with autosave module making it crash at random basis.

Gdb has been started like this

gdb './bin/linux-x86/ioc iocBoot/ioc/st.cmd' crash.xxxxxxx

Unfortunately no ,I did not try more recent version of autosave. The IOC crashed this time in an SNMP driver, so I will have to keep debugging this.

Best Regards,
Abdalla.

-----Original Message-----
From: Mark Rivers [mailto:[email protected]]
Sent: Wednesday, April 25, 2018 2:48 PM
To: Abdalla Ahmad <[email protected]>; [email protected]
Subject: Re: Problem with autosave module making it crash at random basis.

gdb is printing this message near the beginning:


./bin/linux-x86/ioc iocBoot/ioc/st.cmd: No such file or directory.


It has none of your symbols.  Please send again showing the line where you started gdb.


Have you tried a more recent version of autosave?


Mark



________________________________
From: Abdalla Ahmad <[email protected]>
Sent: Wednesday, April 25, 2018 12:52 AM
To: Mark Rivers; [email protected]
Subject: RE: Problem with autosave module making it crash at random basis.

Hello Mark

Please find the attached output from gdb when running it with the core dump.

Best Regards,
Abdalla.

-----Original Message-----
From: Abdalla Ahmad
Sent: Wednesday, April 18, 2018 2:58 PM
To: 'Mark Rivers' <[email protected]>; [email protected]
Subject: RE: Problem with autosave module making it crash at random basis.

Unfortunately I deleted the core dump. Once the IOC crash again I will send the complete output.

Abdalla.

-----Original Message-----
From: Mark Rivers [mailto:[email protected]]
Sent: Wednesday, April 18, 2018 2:58 PM
To: Abdalla Ahmad <[email protected]>; [email protected]
Subject: Re: Problem with autosave module making it crash at random basis.

> The core dump is generated but GDB could not show a backtrace and reported "missing separate debuginfo for the main executable file". Is it the case that the IOC should be built with debug symbols?


I have not seen that when building with the standard non-debug EPICS_HOST_ARCH of linux-x86_64.  It will warn that some system libraries don't have debug symbols, and some variables will be shown as "optimized out", but I have always been able to get a backtrace when it crashed.


Can you send the complete output when you start gdb and try to do a backtrace?


Mark




________________________________
From: Abdalla Ahmad <[email protected]>
Sent: Wednesday, April 18, 2018 12:46 AM
To: Mark Rivers; [email protected]
Subject: RE: Problem with autosave module making it crash at random basis.

Hi Mark

The core dump is generated but GDB could not show a backtrace and reported "missing separate debuginfo for the main executable file". Is it the case that the IOC should be built with debug symbols?

Best Regards,
Abdalla.

-----Original Message-----
From: Mark Rivers [mailto:[email protected]]
Sent: Sunday, April 08, 2018 4:01 PM
To: Abdalla Ahmad <[email protected]>; [email protected]
Subject: Re: Problem with autosave module making it crash at random basis.

Hi Abdalla,


The latest autosave Makefile contains these lines:


# To enable autosaveBuild, use dbLoadRecordsHookRegister().  (Requires EPICS # base 3.14.12.5 or later.) USR_CFLAGS += -DDBLOADRECORDSHOOKREGISTER

Since you are running 3.14.12.3 I think you should try commenting out that USR_CFLAGS line.


With your older version of autosave your output showed that valgrind itself had crashed.  You should be able to get a traceback without valgrind, just by enabling core dumps.  Type the following commands:


On bash shell:
ulimit -c 10000000

or on csh shell:
limit core 1000000

Then run the IOC application.  When it crashes it will create a core.xxxxxx file.

Then run the Gnu debugger:
gdb path/to/your/IOC/executable core.xxxxxx

You need to provide the path to your IOC executable and replace core.xxxxxx with the name of the core file it just created.

Once you are in the debugger type:
backtrace

It will show a stack dump to where it crashed.

Mark







________________________________
From: [email protected] <[email protected]> on behalf of Abdalla Ahmad <[email protected]>
Sent: Sunday, April 8, 2018 1:06 AM
To: [email protected]
Subject: Problem with autosave module making it crash at random basis.


Hi



One of our IOCs crashed and it appears to do so from time to time. I used valgrind on the IOC process and the attached file contains valgrind output right when the IOC crashed. The first 2 lines are the IOC shell output before the segmentation fault happened. I could not figure out whether the problem from the module sending something corrupted to the printf functions or the standard C library itself is somehow corrupted. I tried cloning the latest autosave version from github but the build failed. Build output is in the attached too.



OS: Scientific Linux 6.4 32-bit.

EPICS: 3.14.12.3

SynApps 5.6

Autosave 4.8



Best Regards,



Abdalla Ahmad

Control Engineer

SESAME

Allan, Jordan.

Tel: (+962-5) 3511348 , ext. 265

Fax: (+962-5) 3511423

Mob: (+962-7)88183296

www.sesame.org.jo<http://www.sesame.org.jo/>



Navigate by Date:
Prev: Re: Problem with autosave module making it crash at random basis. Mark Rivers
Next: Re: Possible deadlock issue with asyn layer? Mark Davis
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: RE: Problem with autosave module making it crash at random basis. Abdalla Ahmad
Next: Re: Problem with autosave module making it crash at random basis. Konrad, Martin
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020  2021  2022  2023  2024 
ANJ, 26 Apr 2018 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·