EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024  Index 2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Beacon handling
From: Timo Korhonen via Core-talk <[email protected]>
To: EPICS Core Talk <[email protected]>
Date: Fri, 8 Nov 2019 16:06:41 +0000

Colleagues,

 

We have recently been investigating reports of IOCs crashing. We are using base 7.0.3 and the problem seems to me somehow to be related to beacon handling in PVA code.

 

I attached a stack trace that I got from our colleagues working on an IOC, trying to integrate a CAEN power supply using StreamDevice.

The crash is probably not related to StreamDevice in any way; it seems only happen when we have PVA support activated.

 

Does the attached stack trace tell you anything? Can you give ideas about what to do to give better troubleshooting information?

 

Thanks,

 

Timo

[root@ipc-detector-01 bm-caen]#
[root@ipc-detector-01 bm-caen]# gdb /epics/base-7.0.3/bin/linux-x86_64/softIocPVA
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /epics/base-7.0.3/bin/linux-x86_64/softIocPVA...done.
(gdb)
(gdb)
(gdb) core core.381813
[New LWP 381836]
[New LWP 381838]
[New LWP 381839]
# (...)
[New LWP 381827]
[New LWP 381817]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/epics/base-7.0.3/bin/linux-x86_64/softIocPVA -D /epics/base-7.0.3/dbd/softIocP'.
Program terminated with signal 6, Aborted.
#0  0x00007fe37a2f5337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
55    return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb)
(gdb)
(gdb) backtrace full
#0  0x00007fe37a2f5337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
        resultvar = 0
        pid = 381813
        selftid = 381836
#1  0x00007fe37a2f6a28 in __GI_abort () at abort.c:90
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x2d, sa_sigaction = 0x2d}, sa_mask = {__val = {45, 140614994131280, 140614984506032, 45, 140613202745480, 140614996503872, 140614987968960, 45,
              1, 45, 140614994131280, 140614987953024, 140614984460034, 0, 140613269851440, 140613202742576}}, sa_flags = 1960852144, sa_restorer = 0x7fe314000d30}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#2  0x00007fe37ac057d5 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95
        terminating = true
        t = <optimized out>
#3  0x00007fe37ac03746 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:38
No locals.
#4  0x00007fe37ac03773 in std::terminate () at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
No locals.
#5  0x00007fe37ac042df in __cxxabiv1::__cxa_pure_virtual () at ../../../../libstdc++-v3/libsupc++/pure.cc:50
No locals.
#6  0x00007fe37b9554d9 in epics::pvData::compare (a=..., b=...) at ../../src/factory/Compare.cpp:86
        nflds = <optimized out>
#7  0x00007fe37b913c4a in epics::pvData::FieldCreate::Helper::cache<epics::pvData::Structure> (create=create@entry=0xce02b0, ent=std::tr1::shared_ptr (count 1, weak 1) 0x7fe310000d30)
    at ../../src/factory/FieldCreateFactory.cpp:63
        cent = 0x7fe314000d30
        centx = <optimized out>
        hash = 2882912583
        G = {mutexPtr = @0xce0300, locked = true}
        itp = {first = {first = 2882912583, second = }, second = {first = 2914433719, second = }}
#8  0x00007fe37b90ceda in epics::pvData::FieldCreate::createStructure (this=this@entry=0xce02b0, fieldNames=std::vector of length 6, capacity 6 = {...},
    fields=std::vector of length 6, capacity 6 = {...}) at ../../src/factory/FieldCreateFactory.cpp:1305
        sp = std::tr1::shared_ptr (count 1, weak 1) 0x7fe310000d30
#9  0x00007fe37b9106d9 in deserializeStructureField (control=0x10ae170, buffer=0x10ae280, fieldCreate=0xce02b0) at ../../src/factory/FieldCreateFactory.cpp:251
        id = ""
        size = 6
        fields = std::vector of length 6, capacity 6 = {std::tr1::shared_ptr (count 72, weak 1) 0xce0af0, std::tr1::shared_ptr (count 18, weak 1) 0xce0e30,
          std::tr1::shared_ptr (count 18, weak 1) 0xce0e30, std::tr1::shared_ptr (count 72, weak 1) 0xce0af0, std::tr1::shared_ptr (count 72, weak 1) 0xce0af0,
          std::tr1::shared_ptr (count 16, weak 1) 0xce19b0}
        fieldNames = std::vector of length 6, capacity 6 = {"connections", "allocatedMemory", "freeMemory", "threads", "deadlocks", "averageSystemLoad"}
#10 epics::pvData::FieldCreate::deserialize (this=0xce02b0, buffer=buffer@entry=0x10ae280, control=control@entry=0x10ae170) at ../../src/factory/FieldCreateFactory.cpp:1468
        typeCode = <optimized out>
        scalarOrArray = <optimized out>
        notArray = <optimized out>
#11 0x00007fe37bcc4229 in (anonymous namespace)::BeaconResponseHandler::handleResponse (this=<optimized out>, responseFrom=0x7fe374e03d90, transport=
    std::tr1::shared_ptr (count 2, weak 1) 0x10ae170, version=<optimized out>, command=<optimized out>, payloadSize=<optimized out>, payloadBuffer=0x10ae280)
    at ../../src/remoteClient/clientContextImpl.cpp:2761
        timestamp = {secondsPastEpoch = 1572541624, nanoseconds = 907433738, userTag = 0}
        field = std::tr1::shared_ptr (empty) 0x0
        guid = {value = "gs\222s\217cD\230\206\346\224", <incomplete sequence \321>}
        serverAddress = {ia = {sin_family = 2, sin_port = 33179, sin_addr = {s_addr = 16914348}, sin_zero = "\000\000\000\000\000\000\000"}, sa = {sa_family = 2,
            sa_data = "\233\201\254\027\002\001\000\000\000\000\000\000\000"}}
        protocol = "tcp"
        data = std::tr1::shared_ptr (empty) 0x0
        context = std::tr1::shared_ptr (count 5, weak 57) 0xfe8e90
        beaconHandler = std::tr1::shared_ptr (count 2, weak 0) 0x7fe310000a20
#12 0x00007fe37bcbecba in (anonymous namespace)::ClientResponseHandler::handleResponse (this=<optimized out>, responseFrom=<optimized out>, transport=..., version=<optimized out>,
    command=<optimized out>, payloadSize=<optimized out>, payloadBuffer=0x10ae280) at ../../src/remoteClient/clientContextImpl.cpp:3008
        payloadSize = <optimized out>
        command = 0 '\000'
        transport = <optimized out>
        this = <optimized out>
        payloadBuffer = 0x10ae280
        version = <optimized out>
        responseFrom = <optimized out>
#13 0x00007fe37bc9b60f in epics::pvAccess::BlockingUDPTransport::processBuffer (this=this@entry=0x10ae170, transport=std::tr1::shared_ptr (count 2, weak 1) 0x10ae170, fromAddress=...,
    receiveBuffer=receiveBuffer@entry=0x10ae280) at ../../src/remote/blockingUDPTransport.cpp:420
        payloadSize = <optimized out>
        nextRequestPosition = 190
#14 0x00007fe37bc9c658 in epics::pvAccess::BlockingUDPTransport::run (this=0x10ae170) at ../../src/remote/blockingUDPTransport.cpp:269
        ignore = false
        bytesRead = <optimized out>
        recvfrom_buffer_start = 0x10ae388 "\312\001\300"
        recvfrom_buffer_len = 65487
        fromAddress = {ia = {sin_family = 2, sin_port = 54291, sin_addr = {s_addr = 16914348}, sin_zero = "\000\000\000\000\000\000\000"}, sa = {sa_family = 2,
            sa_data = "\023\324\254\027\002\001\000\000\000\000\000\000\000"}}
        addrStructSize = 16
        thisTransport = std::tr1::shared_ptr (count 2, weak 1) 0x10ae170
#15 0x00007fe37aee6829 in epicsThreadCallEntryPoint (pPvt=0x10ce480) at ../../src/osi/epicsThread.cpp:83
        pThread = 0x10ce480
        threadDestroyed = false
#16 0x00007fe37aeec7bc in start_routine (arg=0x10ceb90) at ../../src/osi/os/posix/osdThread.c:403
        pthreadInfo = 0x10ceb90
        status = <optimized out>
        blockAllSig = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
#17 0x00007fe37a0aae65 in start_thread (arg=0x7fe374e04700) at pthread_create.c:307
        __res = <optimized out>
        pd = 0x7fe374e04700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140614895159040, 9139648661074023024, 0, 2101248, 0, 140614895159040, -9146696730313420176, -9146676574535609744}, mask_was_saved = 0}}, priv = {
            pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
#18 0x00007fe37a3bd88d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.
(gdb)

Replies:
Re: Beacon handling Michael Davidsaver via Core-talk

Navigate by Date:
Prev: missing: TESTDBD Benjamin Franksen via Core-talk
Next: Re: Beacon handling Michael Davidsaver via Core-talk
Index: 2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Re: missing: TESTDBD Benjamin Franksen via Core-talk
Next: Re: Beacon handling Michael Davidsaver via Core-talk
Index: 2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024 
ANJ, 08 Nov 2019 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·