Re: Can't create mutex semaphore: too many

[Michael Davidsaver via Tech-talk <tech-talk at aps.anl.gov>]

On 10/11/22 13:13, Luchini, Kristi L. wrote:
> Hello,
>
> Any news on a fix for this bug?

Not from me.

https://github.com/epics-base/pvAccessCPP/issues/184


> Would removing PVA/QSRV from the binary solve the problem in this ioc for now?

It would.  You could also run "stopPVAServer" at the IOC shell
to stop the PVA server.  "startPVAServer" would then restart it
if desired.

A stop and restart should have the effect of clearing the beacon
tracking std::map which is causing this problem, although it will
also disconnect any PVA clients.


>  This is an MPS Ioc and rebooting trips beam. So if I can find a temporary solution that would be great until we can get a bug fix.
>
> MpsSupport_DBD += PVAServerRegister.dbd
> MpsSupport_DBD += qsrv.dbd
> MpsSupportSoft_LIBS += qsrv
> USR_LIBS += $(EPICS_BASE_PVA_CORE_LIBS)
>
>
> -Kristi
>
> -----Original Message-----
> From: Luchini, Kristi L.
> Sent: Thursday, May 26, 2022 2:57 PM
> To: Michael Davidsaver <mdavidsaver at gmail.com>
> Cc: Talk EPICS Tech <tech-talk at aps.anl.gov>; Till Straumann <till.straumann at psi.ch>; Joel Sherrill <joel.sherrill at gmail.com>
> Subject: Re: Can't create mutex semaphore: too many
>
> Yes, I am using Till’s RTaeME configuration.
>
> Kristi
>
> Sent from my iPhone
>
> > On May 26, 2022, at 9:33 AM, Michael Davidsaver <mdavidsaver at gmail.com> wrote:
> >
> > On 5/25/22 14:15, Michael Davidsaver wrote:
> > > Indeed this appears to be the case.  There is an std::map used to
> > > track which PVA servers have sent beacons.  Entries are added, but never removed.
> > > Each entry has a Mutex (this code does very granular locking...).
> >
> > I also notice that a temporary Mutex is allocated while processing
> > each search message.  Not a leak, but also not so efficient.
> >
> >
> > I've been playing around with "bpftrace" as an alternative to
> > "systemtap".  (imo. systemtap is great, but decoding its error
> > messages is no fun)
> >
> > https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.m
> > d
> >
> > > $ cat rxmutex.bpf
> > > // Print user stack trace when epicsMutexOsiCreate() called through
> > > processBuffer() // unmangled name...
> > > // epics::pvAccess::BlockingUDPTransport::processBuffer(
> > > //  std::shared_ptr<epics::pvAccess::Transport> const&, osiSockAddr&,
> > > //  osiSockAddr& fromAddress, ByteBuffer* receiveBuffer)
> > > uprobe:"/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-deb
> > > ug/libpvAccess.so.7.1.5":_ZN5epics8pvAccess20BlockingUDPTransport13pr
> > > ocessBufferERKSt10shared_ptrINS0_9TransportEER11osiSockAddrPNS_6pvDat
> > > a10ByteBufferE
> > > {
> > >     @inproc[tid] = 1;
> > > }
> > > uretprobe:"/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-
> > > debug/libpvAccess.so.7.1.5":_ZN5epics8pvAccess20BlockingUDPTransport1
> > > 3processBufferERKSt10shared_ptrINS0_9TransportEER11osiSockAddrPNS_6pv
> > > Data10ByteBufferE
> > > {
> > >     delete(@inproc[tid]);
> > > }
> > > uprobe:"/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/
> > > libCom.so.3.21.1":epicsMutexOsiCreate
> > > {
> > >     if(@inproc[tid]) {
> > >         printf("==============\n%s\n", ustack(perf));
> > >     }
> > > }
> >
> > Why yes, that c++ mangled symbol name is 127 characters long...
> >
> >
> > > $ sudo bpftrace -p `pgrep softIocPVA` rxmutex.bpf Attaching 3
> > > probes...
> >
> > While running "pvget"
> >
> > > ==============
> > >         7f474e0e2d53 epicsMutexOsiCreate+0 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
> > >         7f474dbad4dc epics::pvAccess::ServerChannelFindRequesterImpl::ServerChannelFindRequesterImpl(std::shared_ptr<epics::pvAccess::ServerContextImpl> const&, std::shared_ptr<epics::pvAccess::PeerInfo const> const&, int)+308 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
> > >         7f474dbaced7 epics::pvAccess::ServerSearchHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+1897 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
> > >         7f474dbac094 epics::pvAccess::ServerResponseHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+380 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
> > >         7f474db420b8 epics::pvAccess::BlockingUDPTransport::processBuffer(std::shared_ptr<epics::pvAccess::Transport> const&, osiSockAddr&, epics::pvData::ByteBuffer*)+736 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
> > >         7fffffffe000 0x7fffffffe000 ([unknown])
> > >         7f474e0e1869 epicsThreadCallEntryPoint+97 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
> > >         7f474e0e9d96 start_routine+431 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
> > >         7f474d764ea7 start_thread+215
> > > (/usr/lib/x86_64-linux-gnu/libpthread-2.31.so)
> >
> > While running another PVA server
> >
> > > ==============
> > >         7f474e0e2d53 epicsMutexOsiCreate+0 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
> > >         7f474db49e68 epics::pvAccess::BeaconHandler::BeaconHandler(std::shared_ptr<epics::pvAccess::Context> const&, osiSockAddr const*)+86 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
> > >         7f474db8c2cc (anonymous namespace)::InternalClientContextImpl::getBeaconHandler(osiSockAddr*)+244 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
> > >         7f474db83bfc (anonymous namespace)::BeaconResponseHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+678 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
> > >         7f474db85b8d (anonymous namespace)::ClientResponseHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+349 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
> > >         7f474db420b8 epics::pvAccess::BlockingUDPTransport::processBuffer(std::shared_ptr<epics::pvAccess::Transport> const&, osiSockAddr&, epics::pvData::ByteBuffer*)+736 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
> > >         7fffffffe000 0x7fffffffe000 ([unknown])
> > >         7f474e0e1869 epicsThreadCallEntryPoint+97 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
> > >         7f474e0e9d96 start_routine+431 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
> > >         7f474d764ea7 start_thread+215
> > > (/usr/lib/x86_64-linux-gnu/libpthread-2.31.so)
> > ...