Re: areaDetector crashes EPICS IOC and autosave failure

[Jong Woo Kim via Tech-talk <tech-talk at aps.anl.gov>]

Hi Mark,

Thank you so much for letting me know. I could not find "core" in eigerDetectorApp or in any folder around it. Instead, I found systemd-coredump in  "/usr/lib/systemd/systemd-coredump".

The following is the result of running gdb and bt. It seems okay to me. Is this a wrong file?

$ gdb systemd-coredump

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7

Copyright (C) 2013 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-redhat-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>...

Reading symbols from /usr/lib/systemd/systemd-coredump...Reading symbols from /usr/lib/systemd/systemd-coredump...(no debugging symbols found)...done.

(no debugging symbols found)...done.

Missing separate debuginfos, use: debuginfo-install systemd-219-67.el7_7.2.x86_64

(gdb) bt

No stack.

---

From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Friday, January 31, 2020 4:56 PM
To: Jong Woo Kim <Jong.Woo.Kim at asu.edu>
Cc: tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: RE: areaDetector crashes EPICS IOC and autosave failure

 

Hi Jong Woo,

 

Ø  $ gdb /path/to/ADEiger_executable core     --- > coredump doesn't work either

 

I was not clear in my instructions. “/path/to/ADEiger_executable” is not what you should type, you should type the actual path, which is probably something like:

 

EPICS/synApps_6_1/support/areaDetector-R3-7/ADEiger/iocs/eigerIOC/bin/linux-x86_64/eigerDetectorApp.

 

And “core” is the name of the actual core.* file that was created when you got the core dump message when the IOC crashed. It is probably in your iocBoot/iocEiger directory.  But on some systems it can be put in /var or somewhere else.

 

Mark

 

 

 

From: Jong Woo Kim <Jong.Woo.Kim at asu.edu>
Sent: Friday, January 31, 2020 5:45 PM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: tech-talk at aps.anl.gov
Subject: Re: areaDetector crashes EPICS IOC and autosave failure

 

Hi Mark,

 

Thank you so much. I resolved first two issues. But I'm still working on the last one 'eigerDetector crashes EPICS IOC'  ( e.i. Segmentation fault (core dumped) )   

I wasn't able to find where it crashed. The following is the message I've got.

 

Note that the version of areaDetector, ADEiger, ADCore and EPICS base are R3-7, R2-7, R3-8, 3-15-6, respectively.

We're using FileWriter interface to save data files.

____________________________________________________________

 

$ gdb /path/to/ADEiger_executable core     --- > coredump doesn't work either

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7

Copyright (C) 2013 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-redhat-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>...

/path/to/ADEiger_executable: No such file or directory.

.../EPICS/synApps_6_1/support/areaDetector-R3-7/ADEiger/iocs/eigerIOC/iocBoot/iocEiger/autosave/core: No such file or directory.

(gdb)

 

____________________________________________________________

 

I researched for coredump files

____________________________________________________________

 

.../systemd/coredump.conf

.../bin/coredumpctl

.../bin/systemd-coredumpctl

.../lib/systemd/systemd-coredump

.../share/bash-completion/completions/coredumpctl

.../share/man/man1/coredumpctl.1.gz

.../share/man/man5/coredump.conf.5.gz

.../share/man/man5/coredump.conf.d.5.gz

.../share/man/man8/systemd-coredump.8.gz

.../share/zsh/site-functions/_coredumpctl

.../include/config/coredump.h

.../include/config/dev/coredump.h

.../include/config/want/dev/coredump.h

.../include/linux/coredump.h

.../include/linux/devcoredump.h

.../include/config/coredump.h

.../include/config/dev/coredump.h

.../include/config/want/dev/coredump.h

.../include/linux/coredump.h

.../include/linux/devcoredump.h

.../include/config/coredump.h

.../include/config/dev/coredump.h

.../include/config/want/dev/coredump.h

.../include/linux/coredump.h

.../include/linux/devcoredump.h

.../lib/systemd/coredump

____________________________________________________________

 

This is one of examples that results from gdb.

____________________________________________________________

 

$ gdb /var/lib/systemd/coredump

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7

Copyright (C) 2013 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.                                                                                                                                                                                                        

This GDB was configured as "x86_64-redhat-linux-gnu".                                                                                                                                                                                  

For bug reporting instructions, please see:                                                                                                                                                                                            

<http://www.gnu.org/software/gdb/bugs/>...                                                                                                                                                                                              

/var/lib/systemd/coredump: Success.  

 

 

 

 

---

From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Thursday, January 30, 2020 10:35 AM
To: Jong Woo Kim <Jong.Woo.Kim at asu.edu>
Cc: Jong Woo Kim <jongwoo.email at gmail.com>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: RE: areaDetector crashes EPICS IOC and autosave failure

 

Hi Jong Woo,

 

You have not told us what version of ADEiger, ADCore, EPICS base, etc. you are using.

 

Ø  a) It keeps failing to open 'sseq_setting.req' file (Purple in the following message) 

 

Did you uncomment this line from ADCore/iocBoot/EXAMPLE_commonPlugin_settings.req?

#file "sseqRecord_settings.req",     P=$(P),  S=AcquireSequence

 

If so you also need to uncomment this line from ADCore/iocBoot/EXAMPLE_commonPlugins.cmd

#set_requestfile_path("$(CALC)/calcApp/Db")

 

Ø  b) It keeps failing to writing 'auto_settings.sav' file (Blue in the following message) 

Ø  ../save_restore.c(1774): [0xd]=write_it:Permission denied

 

That error seems pretty simple.  It is a file permission problem.  The files in that directory, or the directory itself, must not have write permission for the account that is running ADEiger application.  Fix the file permissions.

 

Ø  c) When the data acquisition is done on eigerDetector, it crashes EPICS IOC (Red in the following message)

Ø  Actually data hasn't been collected
Segmentation fault (core dumped)

 

You have a core dump file.  You need to run gdb on that to find where it crashed.

Run this:

gdb /path/to/ADEiger_executable core.xxx

 

where core.xxx is the name of your core dump file.  Once you get the gdb> prompt type this command

backtrace

 

Mark

 

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Jong Woo Kim via Tech-talk
Sent: Thursday, January 30, 2020 10:33 AM
To: tech-talk at aps.anl.gov
Cc: Jong Woo Kim <Jong.Woo.Kim at asu.edu>; Jong Woo Kim <jongwoo.email at gmail.com>
Subject: areaDetector crashes EPICS IOC and autosave failure

 

Hello,

 

When the data collection is done on 'eigerDetector.adl', EPICS IOC is crashed and actually data hasn't been collected.

In addition, whatever parameters I changed on 'eigerDetector' are not applied in autosave. These are the summary followed by the message.

 

a) It keeps failing to open 'sseq_setting.req' file (Purple in the following message) 

     I made sure that this file has contents and it is in a right folder

          support/calc-R3-7-3/calcApp/Db/sseq_settings.req

          support/calc-R3-7-3/db/sseq_settings.req

 

b) It keeps failing to writing 'auto_settings.sav' file (Blue in the following message) 

     It keeps trying to save settings every 30 second as it is set on the first purple line. But it can't.  

 

c) When the data acquisition is done on eigerDetector, it crashes EPICS IOC (Red in the following message)

     Actually data hasn't been collected

 

 

--------------------------------------------------- Message on monitor--------------------------------------------------------------

 

############################################################################
## EPICS R3.15.6
## EPICS Base built Dec  2 2019
############################################################################
reboot_restore: entry for file 'auto_settings.sav'
reboot_restore: Found filename 'auto_settings.sav' in restoreFileList.
*** restoring from './autosave/auto_settings.sav' at initHookState 6 (before record/device init) ***
save_restore:myFileCopy: Can't open file './autosave/auto_settings.sav_200127-151325'
save_restore: Can't write backup file.
2020/01/27 15:13:25.455 asynPortDriver:drvUserCreate: addr=0, cannot find parameter CLEAR
EIG1:cam1:FWClear devAsynInt32::initCommon drvUserCreate 
Bad init_rec return value  PV: EIG1:cam1:FWClear ao: init_record

reboot_restore: entry for file 'auto_settings.sav'
reboot_restore: Found filename 'auto_settings.sav' in restoreFileList.
*** restoring from './autosave/auto_settings.sav' at initHookState 7 (after record/device init) ***
reboot_restore: done with file 'auto_settings.sav'

iocRun: All initialization complete
# Avoid deluge of messages when debugging
#dbpf $(PREFIX)cam1:PoolUsedMem.SCAN Passive
# save things every thirty seconds
create_monitor_set("auto_settings.req", 30, "P= EIG1:")
save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
epics> save_restore: Can't connect to all status PV(s)
auto_settings.sav: 1330 of 1330 PV's connected
save_restore: Can't connect to status PV(s) for list 'auto_settings.sav'
save_restore:write_it - unable to open file './autosave/auto_settings.sav' [200127-151357]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write save file. [200127-151357]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1774): [0xd]=write_it:Permission denied
save_restore:write_it - unable to open file './autosave/auto_settings.sav' [200127-151427]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write save file. [200127-151427]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
../save_restore.c(1774): [0xd]=write_it:Permission denied
Segmentation fault (core dumped)

[iocEiger]$