Re: Can't create mutex semaphore: too many

["Luchini, Kristi L. via Tech-talk" <tech-talk at aps.anl.gov>]

Yes, i  using till’s configuration

Kristi

Sent from my iPhone

> On May 26, 2022, at 9:33 AM, Michael Davidsaver <mdavidsaver at gmail.com> wrote:
> 
> On 5/25/22 14:15, Michael Davidsaver wrote:
>> Indeed this appears to be the case.  There is an std::map used to track
>> which PVA servers have sent beacons.  Entries are added, but never removed.
>> Each entry has a Mutex (this code does very granular locking...).
> 
> I also notice that a temporary Mutex is allocated
> while processing each search message.  Not a leak,
> but also not so efficient.
> 
> 
> I've been playing around with "bpftrace" as an
> alternative to "systemtap".  (imo. systemtap is
> great, but decoding its error messages is no fun)
> 
> https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md
> 
>> $ cat rxmutex.bpf
>> // Print user stack trace when epicsMutexOsiCreate() called through processBuffer()
>> // unmangled name...
>> // epics::pvAccess::BlockingUDPTransport::processBuffer(
>> //  std::shared_ptr<epics::pvAccess::Transport> const&, osiSockAddr&,
>> //  osiSockAddr& fromAddress, ByteBuffer* receiveBuffer)
>> uprobe:"/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5":_ZN5epics8pvAccess20BlockingUDPTransport13processBufferERKSt10shared_ptrINS0_9TransportEER11osiSockAddrPNS_6pvData10ByteBufferE
>> {
>>    @inproc[tid] = 1;
>> }
>> uretprobe:"/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5":_ZN5epics8pvAccess20BlockingUDPTransport13processBufferERKSt10shared_ptrINS0_9TransportEER11osiSockAddrPNS_6pvData10ByteBufferE
>> {
>>    delete(@inproc[tid]);
>> }
>> uprobe:"/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1":epicsMutexOsiCreate
>> {
>>    if(@inproc[tid]) {
>>        printf("==============\n%s\n", ustack(perf));
>>    }
>> }
> 
> Why yes, that c++ mangled symbol name is 127 characters long...
> 
> 
>> $ sudo bpftrace -p `pgrep softIocPVA` rxmutex.bpf
>> Attaching 3 probes...
> 
> While running "pvget"
> 
>> ==============
>>        7f474e0e2d53 epicsMutexOsiCreate+0 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>>        7f474dbad4dc epics::pvAccess::ServerChannelFindRequesterImpl::ServerChannelFindRequesterImpl(std::shared_ptr<epics::pvAccess::ServerContextImpl> const&, std::shared_ptr<epics::pvAccess::PeerInfo const> const&, int)+308 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>>        7f474dbaced7 epics::pvAccess::ServerSearchHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+1897 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>>        7f474dbac094 epics::pvAccess::ServerResponseHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+380 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>>        7f474db420b8 epics::pvAccess::BlockingUDPTransport::processBuffer(std::shared_ptr<epics::pvAccess::Transport> const&, osiSockAddr&, epics::pvData::ByteBuffer*)+736 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>>        7fffffffe000 0x7fffffffe000 ([unknown])
>>        7f474e0e1869 epicsThreadCallEntryPoint+97 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>>        7f474e0e9d96 start_routine+431 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>>        7f474d764ea7 start_thread+215 (/usr/lib/x86_64-linux-gnu/libpthread-2.31.so)
> 
> While running another PVA server
> 
>> ==============
>>        7f474e0e2d53 epicsMutexOsiCreate+0 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>>        7f474db49e68 epics::pvAccess::BeaconHandler::BeaconHandler(std::shared_ptr<epics::pvAccess::Context> const&, osiSockAddr const*)+86 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>>        7f474db8c2cc (anonymous namespace)::InternalClientContextImpl::getBeaconHandler(osiSockAddr*)+244 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>>        7f474db83bfc (anonymous namespace)::BeaconResponseHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+678 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>>        7f474db85b8d (anonymous namespace)::ClientResponseHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+349 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>>        7f474db420b8 epics::pvAccess::BlockingUDPTransport::processBuffer(std::shared_ptr<epics::pvAccess::Transport> const&, osiSockAddr&, epics::pvData::ByteBuffer*)+736 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>>        7fffffffe000 0x7fffffffe000 ([unknown])
>>        7f474e0e1869 epicsThreadCallEntryPoint+97 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>>        7f474e0e9d96 start_routine+431 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>>        7f474d764ea7 start_thread+215 (/usr/lib/x86_64-linux-gnu/libpthread-2.31.so)
> ...