Experimental Physics and
Industrial Control System

Michael Davidsaver via Tech-talk <tech-talk at aps.anl.gov> · Tue, 11 Oct 2022 16:33:23 -0700

On 10/11/22 13:13, Luchini, Kristi L. wrote:

Hello,

Any news on a fix for this bug?

Not from me.

https://github.com/epics-base/pvAccessCPP/issues/184

Would removing PVA/QSRV from the binary solve the problem in this ioc for now?

It would.  You could also run "stopPVAServer" at the IOC shell
to stop the PVA server.  "startPVAServer" would then restart it
if desired.

A stop and restart should have the effect of clearing the beacon
tracking std::map which is causing this problem, although it will
also disconnect any PVA clients.

 This is an MPS Ioc and rebooting trips beam. So if I can find a temporary solution that would be great until we can get a bug fix.

MpsSupport_DBD += PVAServerRegister.dbd
MpsSupport_DBD += qsrv.dbd
MpsSupportSoft_LIBS += qsrv
USR_LIBS += $(EPICS_BASE_PVA_CORE_LIBS)

-Kristi

-----Original Message-----
From: Luchini, Kristi L.
Sent: Thursday, May 26, 2022 2:57 PM
To: Michael Davidsaver <mdavidsaver at gmail.com>
Cc: Talk EPICS Tech <tech-talk at aps.anl.gov>; Till Straumann <till.straumann at psi.ch>; Joel Sherrill <joel.sherrill at gmail.com>
Subject: Re: Can't create mutex semaphore: too many

Yes, I am using Till’s RTaeME configuration.

Kristi

Sent from my iPhone

On May 26, 2022, at 9:33 AM, Michael Davidsaver <mdavidsaver at gmail.com> wrote:

On 5/25/22 14:15, Michael Davidsaver wrote:

Indeed this appears to be the case.  There is an std::map used to
track which PVA servers have sent beacons.  Entries are added, but never removed.
Each entry has a Mutex (this code does very granular locking...).

I also notice that a temporary Mutex is allocated while processing
each search message.  Not a leak, but also not so efficient.

I've been playing around with "bpftrace" as an alternative to
"systemtap".  (imo. systemtap is great, but decoding its error
messages is no fun)

https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.m
d

$ cat rxmutex.bpf
// Print user stack trace when epicsMutexOsiCreate() called through
processBuffer() // unmangled name...
// epics::pvAccess::BlockingUDPTransport::processBuffer(
//  std::shared_ptr<epics::pvAccess::Transport> const&, osiSockAddr&,
//  osiSockAddr& fromAddress, ByteBuffer* receiveBuffer)
uprobe:"/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-deb
ug/libpvAccess.so.7.1.5":_ZN5epics8pvAccess20BlockingUDPTransport13pr
ocessBufferERKSt10shared_ptrINS0_9TransportEER11osiSockAddrPNS_6pvDat
a10ByteBufferE
{
    @inproc[tid] = 1;
}
uretprobe:"/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-
debug/libpvAccess.so.7.1.5":_ZN5epics8pvAccess20BlockingUDPTransport1
3processBufferERKSt10shared_ptrINS0_9TransportEER11osiSockAddrPNS_6pv
Data10ByteBufferE
{
    delete(@inproc[tid]);
}
uprobe:"/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/
libCom.so.3.21.1":epicsMutexOsiCreate
{
    if(@inproc[tid]) {
        printf("==============\n%s\n", ustack(perf));
    }
}

Why yes, that c++ mangled symbol name is 127 characters long...

$ sudo bpftrace -p `pgrep softIocPVA` rxmutex.bpf Attaching 3
probes...

While running "pvget"

==============
        7f474e0e2d53 epicsMutexOsiCreate+0 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
        7f474dbad4dc epics::pvAccess::ServerChannelFindRequesterImpl::ServerChannelFindRequesterImpl(std::shared_ptr<epics::pvAccess::ServerContextImpl> const&, std::shared_ptr<epics::pvAccess::PeerInfo const> const&, int)+308 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
        7f474dbaced7 epics::pvAccess::ServerSearchHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+1897 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
        7f474dbac094 epics::pvAccess::ServerResponseHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+380 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
        7f474db420b8 epics::pvAccess::BlockingUDPTransport::processBuffer(std::shared_ptr<epics::pvAccess::Transport> const&, osiSockAddr&, epics::pvData::ByteBuffer*)+736 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
        7fffffffe000 0x7fffffffe000 ([unknown])
        7f474e0e1869 epicsThreadCallEntryPoint+97 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
        7f474e0e9d96 start_routine+431 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
        7f474d764ea7 start_thread+215
(/usr/lib/x86_64-linux-gnu/libpthread-2.31.so)

While running another PVA server

==============
        7f474e0e2d53 epicsMutexOsiCreate+0 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
        7f474db49e68 epics::pvAccess::BeaconHandler::BeaconHandler(std::shared_ptr<epics::pvAccess::Context> const&, osiSockAddr const*)+86 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
        7f474db8c2cc (anonymous namespace)::InternalClientContextImpl::getBeaconHandler(osiSockAddr*)+244 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
        7f474db83bfc (anonymous namespace)::BeaconResponseHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+678 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
        7f474db85b8d (anonymous namespace)::ClientResponseHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+349 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
        7f474db420b8 epics::pvAccess::BlockingUDPTransport::processBuffer(std::shared_ptr<epics::pvAccess::Transport> const&, osiSockAddr&, epics::pvData::ByteBuffer*)+736 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
        7fffffffe000 0x7fffffffe000 ([unknown])
        7f474e0e1869 epicsThreadCallEntryPoint+97 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
        7f474e0e9d96 start_routine+431 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
        7f474d764ea7 start_thread+215
(/usr/lib/x86_64-linux-gnu/libpthread-2.31.so)

...

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System