RE: areaDetector crashes EPICS IOC and autosave failure

[Mark Rivers via Tech-talk <tech-talk at aps.anl.gov>]

Hi Jong Woo,

 

What model Eiger are you using?  Is it from Dectris?

 

You said you were running R2-7 of ADEiger.  This is not possible R2-7 has not been tagged yet.  Are you running the current master branch, or R2-6?

 

Something is wrong, because your gdb traceback shows that eigerDetector.cpp line 871 is calling eigerParam::get. 

 

In the master branch of eigerDetector.cpp line 871 is the first line below.  In R2-6 line 871 is the second line below.  Neither are calling eigerParam::get().

 

        if(adStatus == ADStatusAcquire || (adStatus == ADStatusAborted && success))

            setIntegerParam(ADStatus, ADStatusIdle);

 

Please verify what version of ADEiger you have.  Are you running a git clone?  If so please type

 

git status

git diff R2-6 eigerDetector.cpp

 

Mark

 

 

From: Jong Woo Kim <Jong.Woo.Kim at asu.edu>
Sent: Wednesday, February 5, 2020 4:41 PM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: Jennings, Guy <jennings at anl.gov>; tech-talk at aps.anl.gov >> EPICS Tech Talk <tech-talk at aps.anl.gov>
Subject: Re: areaDetector crashes EPICS IOC and autosave failure

 

Hi Mark,

 

Thank you so much. I followed exactly what you said and I've got the followings

 

(gdb) backtrace

#0  EigerParam::get (this=0x0, value="") at ../eigerParam.cpp:410

#1  0x000000000050fe8c in eigerDetector::controlTask (this=0x16f8290) at ../eigerDetector.cpp:871

#2  0x0000000000fecc0c in start_routine (arg=0x1709ea0) at ../../../src/libCom/osi/os/posix/osdThread.c:403

#3  0x00007ffff7402ea5 in start_thread () from /lib64/libpthread.so.0

#4  0x00007ffff62ba8cd in clone () from /lib64/libc.so.6

 

Best,

Jong Woo

 

---

From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Wednesday, February 5, 2020 3:04 PM
To: Jong Woo Kim <Jong.Woo.Kim at asu.edu>
Cc: Jennings, Guy <jennings at anl.gov>; tech-talk at aps.anl.gov >> EPICS Tech Talk <tech-talk at aps.anl.gov>
Subject: RE: areaDetector crashes EPICS IOC and autosave failure

 

For me on Centos 7 (very similar to RHEL 7) I get core.xxxxx files in the current working directory.  I am not sure what I set up to get that, nothing that I remember.

 

However, you don’t need to get a core file at all, you can just run your IOC under gdb to start with.

 

Type the following:

 

gdb [path_to_ioc_application]

 

where [path_to_ioc_application] is the command you normally type to start the IOC, but without the second st.cmd argument.

 

Once you get the gdb prompt type

run st.cmd

 

Then when the IOC crashes you can type the command:

 

backtrace

 

Mark

 

 

From: Jong Woo Kim <Jong.Woo.Kim at asu.edu>
Sent: Wednesday, February 5, 2020 2:50 PM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: Jennings, Guy <jennings at anl.gov>; tech-talk at aps.anl.gov >> EPICS Tech Talk <tech-talk at aps.anl.gov>
Subject: Re: areaDetector crashes EPICS IOC and autosave failure

 

Red hat enterprise linux 7.

 

Get Outlook for iOS

---

From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Wednesday, February 5, 2020 1:45:36 PM
To: Jong Woo Kim <Jong.Woo.Kim at asu.edu>
Cc: Jennings, Guy <jennings at anl.gov>; tech-talk at aps.anl.gov >> EPICS Tech Talk <tech-talk at aps.anl.gov>
Subject: Re: areaDetector crashes EPICS IOC and autosave failure

 

What version of Linux are you running?


Sent from my iPhone

On Feb 5, 2020, at 2:16 PM, Jong Woo Kim via Tech-talk <tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>> wrote:


Thank you Mark, Michael and Guy.  But I couldn't find any core dump files. There is folder /var/lib/systemd/coredump. But there is nothing in there. I also used the command 'coredumpctl'. I think something should appear if any.

$ coredumpctl list
No coredumps found.

The message keeps saying like below.

Segmentation fault (core dumped)

What's wrong?

________________________________
From: Jennings, Guy <jennings at anl.gov<mailto:jennings at anl.gov>>
Sent: Wednesday, February 5, 2020 6:39 AM
To: Jong Woo Kim <Jong.Woo.Kim at asu.edu<mailto:Jong.Woo.Kim at asu.edu>>; tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov> >> EPICS Tech Talk <tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>>
Subject: Re: areaDetector crashes EPICS IOC and autosave failure

To get a core dump out of systemd use the coredumpctl command.

See

man coredumpctl

or try

coredumpctl debug



> On Feb 4, 2020, at 4:56 PM, Jong Woo Kim via Tech-talk <tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>> wrote:
>
> Hi Mark,
>
> Thank you so much for letting me know. I could not find "core" in eigerDetectorApp or in any folder around it. Instead, I found systemd-coredump in  "/usr/lib/systemd/systemd-coredump".
> The following is the result of running gdb and bt. It seems okay to me. Is this a wrong file?
>
> $ gdb systemd-coredump
> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <https://urldefense.proofpoint.com/v2/url?u=http-3A__gnu.org_licenses_gpl.html&d=DwIGaQ&c=l45AxH-kUV29SRQusp9vYR0n1GycN4_2jInuKy6zbqQ&r=yfk-0Txa5oue0MIb0KpWAdOXJcpJMv_fkmChV49LPGY&m=DlM9V0xn01k9a4UG8Jo1piG8GCzfUEywJvqtMidZ8j0&s=5SqrqeDD_u5q7YAVAq3G5t22J2QiTSh0ygG4_hwdBFM&e= >
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.gnu.org_software_gdb_bugs_&d=DwIGaQ&c=l45AxH-kUV29SRQusp9vYR0n1GycN4_2jInuKy6zbqQ&r=yfk-0Txa5oue0MIb0KpWAdOXJcpJMv_fkmChV49LPGY&m=DlM9V0xn01k9a4UG8Jo1piG8GCzfUEywJvqtMidZ8j0&s=yiatgN5YlRyxrEDD5VI0j6rOEa6rPgM-oDRq8LFNNNQ&e= >...
> Reading symbols from /usr/lib/systemd/systemd-coredump...Reading symbols from /usr/lib/systemd/systemd-coredump...(no debugging symbols found)...done.
> (no debugging symbols found)...done.
> Missing separate debuginfos, use: debuginfo-install systemd-219-67.el7_7.2.x86_64
> (gdb) bt
> No stack.
>
>
>
>
> From: Mark Rivers <rivers at cars.uchicago.edu<mailto:rivers at cars.uchicago.edu>>
> Sent: Friday, January 31, 2020 4:56 PM
> To: Jong Woo Kim <Jong.Woo.Kim at asu.edu<mailto:Jong.Woo.Kim at asu.edu>>
> Cc: tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov> <tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>>
> Subject: RE: areaDetector crashes EPICS IOC and autosave failure
>
> Hi Jong Woo,
>
> Ø  $ gdb /path/to/ADEiger_executable core     --- > coredump doesn't work either
>
> I was not clear in my instructions. “/path/to/ADEiger_executable” is not what you should type, you should type the actual path, which is probably something like:
>
> EPICS/synApps_6_1/support/areaDetector-R3-7/ADEiger/iocs/eigerIOC/bin/linux-x86_64/eigerDetectorApp.
>
> And “core” is the name of the actual core.* file that was created when you got the core dump message when the IOC crashed. It is probably in your iocBoot/iocEiger directory.  But on some systems it can be put in /var or somewhere else.
>
> Mark
>
>
>
> From: Jong Woo Kim <Jong.Woo.Kim at asu.edu<mailto:Jong.Woo.Kim at asu.edu>>
> Sent: Friday, January 31, 2020 5:45 PM
> To: Mark Rivers <rivers at cars.uchicago.edu<mailto:rivers at cars.uchicago.edu>>
> Cc: tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>
> Subject: Re: areaDetector crashes EPICS IOC and autosave failure
>
> Hi Mark,
>
> Thank you so much. I resolved first two issues. But I'm still working on the last one 'eigerDetector crashes EPICS IOC'  ( e.i. Segmentation fault (core dumped) )
> I wasn't able to find where it crashed. The following is the message I've got.
>
> Note that the version of areaDetector, ADEiger, ADCore and EPICS base are R3-7, R2-7, R3-8, 3-15-6, respectively.
> We're using FileWriter interface to save data files.
> ____________________________________________________________
>
> $ gdb /path/to/ADEiger_executable core     --- > coredump doesn't work either
> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <https://urldefense.proofpoint.com/v2/url?u=http-3A__gnu.org_licenses_gpl.html&d=DwIGaQ&c=l45AxH-kUV29SRQusp9vYR0n1GycN4_2jInuKy6zbqQ&r=yfk-0Txa5oue0MIb0KpWAdOXJcpJMv_fkmChV49LPGY&m=DlM9V0xn01k9a4UG8Jo1piG8GCzfUEywJvqtMidZ8j0&s=5SqrqeDD_u5q7YAVAq3G5t22J2QiTSh0ygG4_hwdBFM&e= >
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.gnu.org_software_gdb_bugs_&d=DwIGaQ&c=l45AxH-kUV29SRQusp9vYR0n1GycN4_2jInuKy6zbqQ&r=yfk-0Txa5oue0MIb0KpWAdOXJcpJMv_fkmChV49LPGY&m=DlM9V0xn01k9a4UG8Jo1piG8GCzfUEywJvqtMidZ8j0&s=yiatgN5YlRyxrEDD5VI0j6rOEa6rPgM-oDRq8LFNNNQ&e= >...
> /path/to/ADEiger_executable: No such file or directory.
> .../EPICS/synApps_6_1/support/areaDetector-R3-7/ADEiger/iocs/eigerIOC/iocBoot/iocEiger/autosave/core: No such file or directory.
> (gdb)
>
> ____________________________________________________________
>
> I researched for coredump files
> ____________________________________________________________
>
> .../systemd/coredump.conf
> .../bin/coredumpctl
> .../bin/systemd-coredumpctl
> .../lib/systemd/systemd-coredump
> .../share/bash-completion/completions/coredumpctl
> .../share/man/man1/coredumpctl.1.gz
> .../share/man/man5/coredump.conf.5.gz
> .../share/man/man5/coredump.conf.d.5.gz
> .../share/man/man8/systemd-coredump.8.gz
> .../share/zsh/site-functions/_coredumpctl
> .../include/config/coredump.h
> .../include/config/dev/coredump.h
> .../include/config/want/dev/coredump.h
> .../include/linux/coredump.h
> .../include/linux/devcoredump.h
> .../include/config/coredump.h
> .../include/config/dev/coredump.h
> .../include/config/want/dev/coredump.h
> .../include/linux/coredump.h
> .../include/linux/devcoredump.h
> .../include/config/coredump.h
> .../include/config/dev/coredump.h
> .../include/config/want/dev/coredump.h
> .../include/linux/coredump.h
> .../include/linux/devcoredump.h
> .../lib/systemd/coredump
> ____________________________________________________________
>
> This is one of examples that results from gdb.
> ____________________________________________________________
>
> $ gdb /var/lib/systemd/coredump
> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <https://urldefense.proofpoint.com/v2/url?u=http-3A__gnu.org_licenses_gpl.html&d=DwIGaQ&c=l45AxH-kUV29SRQusp9vYR0n1GycN4_2jInuKy6zbqQ&r=yfk-0Txa5oue0MIb0KpWAdOXJcpJMv_fkmChV49LPGY&m=DlM9V0xn01k9a4UG8Jo1piG8GCzfUEywJvqtMidZ8j0&s=5SqrqeDD_u5q7YAVAq3G5t22J2QiTSh0ygG4_hwdBFM&e= >
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.gnu.org_software_gdb_bugs_&d=DwIGaQ&c=l45AxH-kUV29SRQusp9vYR0n1GycN4_2jInuKy6zbqQ&r=yfk-0Txa5oue0MIb0KpWAdOXJcpJMv_fkmChV49LPGY&m=DlM9V0xn01k9a4UG8Jo1piG8GCzfUEywJvqtMidZ8j0&s=yiatgN5YlRyxrEDD5VI0j6rOEa6rPgM-oDRq8LFNNNQ&e= >...
> /var/lib/systemd/coredump: Success.
>
>
>
>
> From: Mark Rivers <rivers at cars.uchicago.edu<mailto:rivers at cars.uchicago.edu>>
> Sent: Thursday, January 30, 2020 10:35 AM
> To: Jong Woo Kim <Jong.Woo.Kim at asu.edu<mailto:Jong.Woo.Kim at asu.edu>>
> Cc: Jong Woo Kim <jongwoo.email at gmail.com<mailto:jongwoo.email at gmail.com>>; tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov> <tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>>
> Subject: RE: areaDetector crashes EPICS IOC and autosave failure
>
> Hi Jong Woo,
>
> You have not told us what version of ADEiger, ADCore, EPICS base, etc. you are using.
>
> Ø  a) It keeps failing to open 'sseq_setting.req' file (Purple in the following message)
>
> Did you uncomment this line from ADCore/iocBoot/EXAMPLE_commonPlugin_settings.req?
> #file "sseqRecord_settings.req",     P=$(P),  S=AcquireSequence
>
> If so you also need to uncomment this line from ADCore/iocBoot/EXAMPLE_commonPlugins.cmd
> #set_requestfile_path("$(CALC)/calcApp/Db")
>
> Ø  b) It keeps failing to writing 'auto_settings.sav' file (Blue in the following message)
> Ø  ../save_restore.c(1774): [0xd]=write_it:Permission denied
>
> That error seems pretty simple.  It is a file permission problem.  The files in that directory, or the directory itself, must not have write permission for the account that is running ADEiger application.  Fix the file permissions.
>
> Ø  c) When the data acquisition is done on eigerDetector, it crashes EPICS IOC (Red in the following message)
> Ø  Actually data hasn't been collected
> Segmentation fault (core dumped)
>
> You have a core dump file.  You need to run gdb on that to find where it crashed.
> Run this:
> gdb /path/to/ADEiger_executable core.xxx
>
> where core.xxx is the name of your core dump file.  Once you get the gdb> prompt type this command
> backtrace
>
> Mark
>
>
> From: Tech-talk <tech-talk-bounces at aps.anl.gov<mailto:tech-talk-bounces at aps.anl.gov>> On Behalf Of Jong Woo Kim via Tech-talk
> Sent: Thursday, January 30, 2020 10:33 AM
> To: tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>
> Cc: Jong Woo Kim <Jong.Woo.Kim at asu.edu<mailto:Jong.Woo.Kim at asu.edu>>; Jong Woo Kim <jongwoo.email at gmail.com<mailto:jongwoo.email at gmail.com>>
> Subject: areaDetector crashes EPICS IOC and autosave failure
>
> Hello,
>
> When the data collection is done on 'eigerDetector.adl', EPICS IOC is crashed and actually data hasn't been collected.
> In addition, whatever parameters I changed on 'eigerDetector' are not applied in autosave. These are the summary followed by the message.
>
> a) It keeps failing to open 'sseq_setting.req' file (Purple in the following message)
>      I made sure that this file has contents and it is in a right folder
>           support/calc-R3-7-3/calcApp/Db/sseq_settings.req
>           support/calc-R3-7-3/db/sseq_settings.req
>
> b) It keeps failing to writing 'auto_settings.sav' file (Blue in the following message)
>      It keeps trying to save settings every 30 second as it is set on the first purple line. But it can't.
>
> c) When the data acquisition is done on eigerDetector, it crashes EPICS IOC (Red in the following message)
>      Actually data hasn't been collected
>
>
> --------------------------------------------------- Message on monitor--------------------------------------------------------------
>
> ############################################################################
> ## EPICS R3.15.6
> ## EPICS Base built Dec  2 2019
> ############################################################################
> reboot_restore: entry for file 'auto_settings.sav'
> reboot_restore: Found filename 'auto_settings.sav' in restoreFileList.
> *** restoring from './autosave/auto_settings.sav' at initHookState 6 (before record/device init) ***
> save_restore:myFileCopy: Can't open file './autosave/auto_settings.sav_200127-151325'
> save_restore: Can't write backup file.
> 2020/01/27 15:13:25.455 asynPortDriver:drvUserCreate: addr=0, cannot find parameter CLEAR
> EIG1:cam1:FWClear devAsynInt32::initCommon drvUserCreate
> Bad init_rec return value  PV: EIG1:cam1:FWClear ao: init_record
>
> reboot_restore: entry for file 'auto_settings.sav'
> reboot_restore: Found filename 'auto_settings.sav' in restoreFileList.
> *** restoring from './autosave/auto_settings.sav' at initHookState 7 (after record/device init) ***
> reboot_restore: done with file 'auto_settings.sav'
>
> iocRun: All initialization complete
> # Avoid deluge of messages when debugging
> #dbpf $(PREFIX)cam1:PoolUsedMem.SCAN Passive
> # save things every thirty seconds
> create_monitor_set("auto_settings.req", 30, "P= EIG1:")
> save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
> save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
> save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
> save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
> save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
> save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
> save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
> save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
> save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
> save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
> save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
> save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
> save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
> save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
> save_restore:readReqFile: unable to open file sseq_settings.req. Exiting.
> epics> save_restore: Can't connect to all status PV(s)
> auto_settings.sav: 1330 of 1330 PV's connected
> save_restore: Can't connect to status PV(s) for list 'auto_settings.sav'
> save_restore:write_it - unable to open file './autosave/auto_settings.sav' [200127-151357]
> *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
> save_restore:write_save_file: Can't write save file. [200127-151357]
> *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
> ../save_restore.c(1774): [0xd]=write_it:Permission denied
> save_restore:write_it - unable to open file './autosave/auto_settings.sav' [200127-151427]
> *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
> save_restore:write_save_file: Can't write save file. [200127-151427]
> *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
> ../save_restore.c(1774): [0xd]=write_it:Permission denied
> Segmentation fault (core dumped)
> [iocEiger]$