RE: Can't create mutex semaphore: too many

["Luchini, Kristi L. via Tech-talk" <tech-talk@aps.anl.gov>]

Hello,

Any news on a fix for this bug?  

Would removing PVA/QSRV from the binary solve the problem in this ioc for now?  This is an MPS Ioc and rebooting trips beam. So if I can find a temporary solution that would be great until we can get a bug fix.

MpsSupport_DBD += PVAServerRegister.dbd
MpsSupport_DBD += qsrv.dbd
MpsSupportSoft_LIBS += qsrv
USR_LIBS += $(EPICS_BASE_PVA_CORE_LIBS)


-Kristi

-----Original Message-----
From: Luchini, Kristi L. 
Sent: Thursday, May 26, 2022 2:57 PM
To: Michael Davidsaver <mdavidsaver@gmail.com>
Cc: Talk EPICS Tech <tech-talk@aps.anl.gov>; Till Straumann <till.straumann@psi.ch>; Joel Sherrill <joel.sherrill@gmail.com>
Subject: Re: Can't create mutex semaphore: too many

Yes, I am using Till’s RTaeME configuration.

Kristi

Sent from my iPhone

> On May 26, 2022, at 9:33 AM, Michael Davidsaver <mdavidsaver@gmail.com> wrote:
> 
> On 5/25/22 14:15, Michael Davidsaver wrote:
>> Indeed this appears to be the case.  There is an std::map used to 
>> track which PVA servers have sent beacons.  Entries are added, but never removed.
>> Each entry has a Mutex (this code does very granular locking...).
> 
> I also notice that a temporary Mutex is allocated while processing 
> each search message.  Not a leak, but also not so efficient.
> 
> 
> I've been playing around with "bpftrace" as an alternative to 
> "systemtap".  (imo. systemtap is great, but decoding its error 
> messages is no fun)
> 
> https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.m
> d
> 
>> $ cat rxmutex.bpf
>> // Print user stack trace when epicsMutexOsiCreate() called through 
>> processBuffer() // unmangled name...
>> // epics::pvAccess::BlockingUDPTransport::processBuffer(
>> //  std::shared_ptr<epics::pvAccess::Transport> const&, osiSockAddr&, 
>> //  osiSockAddr& fromAddress, ByteBuffer* receiveBuffer) 
>> uprobe:"/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-deb
>> ug/libpvAccess.so.7.1.5":_ZN5epics8pvAccess20BlockingUDPTransport13pr
>> ocessBufferERKSt10shared_ptrINS0_9TransportEER11osiSockAddrPNS_6pvDat
>> a10ByteBufferE
>> {
>>    @inproc[tid] = 1;
>> }
>> uretprobe:"/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-
>> debug/libpvAccess.so.7.1.5":_ZN5epics8pvAccess20BlockingUDPTransport1
>> 3processBufferERKSt10shared_ptrINS0_9TransportEER11osiSockAddrPNS_6pv
>> Data10ByteBufferE
>> {
>>    delete(@inproc[tid]);
>> }
>> uprobe:"/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/
>> libCom.so.3.21.1":epicsMutexOsiCreate
>> {
>>    if(@inproc[tid]) {
>>        printf("==============\n%s\n", ustack(perf));
>>    }
>> }
> 
> Why yes, that c++ mangled symbol name is 127 characters long...
> 
> 
>> $ sudo bpftrace -p `pgrep softIocPVA` rxmutex.bpf Attaching 3 
>> probes...
> 
> While running "pvget"
> 
>> ==============
>>        7f474e0e2d53 epicsMutexOsiCreate+0 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>>        7f474dbad4dc epics::pvAccess::ServerChannelFindRequesterImpl::ServerChannelFindRequesterImpl(std::shared_ptr<epics::pvAccess::ServerContextImpl> const&, std::shared_ptr<epics::pvAccess::PeerInfo const> const&, int)+308 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>>        7f474dbaced7 epics::pvAccess::ServerSearchHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+1897 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>>        7f474dbac094 epics::pvAccess::ServerResponseHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+380 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>>        7f474db420b8 epics::pvAccess::BlockingUDPTransport::processBuffer(std::shared_ptr<epics::pvAccess::Transport> const&, osiSockAddr&, epics::pvData::ByteBuffer*)+736 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>>        7fffffffe000 0x7fffffffe000 ([unknown])
>>        7f474e0e1869 epicsThreadCallEntryPoint+97 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>>        7f474e0e9d96 start_routine+431 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>>        7f474d764ea7 start_thread+215 
>> (/usr/lib/x86_64-linux-gnu/libpthread-2.31.so)
> 
> While running another PVA server
> 
>> ==============
>>        7f474e0e2d53 epicsMutexOsiCreate+0 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>>        7f474db49e68 epics::pvAccess::BeaconHandler::BeaconHandler(std::shared_ptr<epics::pvAccess::Context> const&, osiSockAddr const*)+86 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>>        7f474db8c2cc (anonymous namespace)::InternalClientContextImpl::getBeaconHandler(osiSockAddr*)+244 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>>        7f474db83bfc (anonymous namespace)::BeaconResponseHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+678 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>>        7f474db85b8d (anonymous namespace)::ClientResponseHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+349 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>>        7f474db420b8 epics::pvAccess::BlockingUDPTransport::processBuffer(std::shared_ptr<epics::pvAccess::Transport> const&, osiSockAddr&, epics::pvData::ByteBuffer*)+736 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>>        7fffffffe000 0x7fffffffe000 ([unknown])
>>        7f474e0e1869 epicsThreadCallEntryPoint+97 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>>        7f474e0e9d96 start_routine+431 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>>        7f474d764ea7 start_thread+215 
>> (/usr/lib/x86_64-linux-gnu/libpthread-2.31.so)
> ...