Re: Can't create mutex semaphore: too many

[Michael Davidsaver via Tech-talk <tech-talk at aps.anl.gov>]

On 5/25/22 14:15, Michael Davidsaver wrote:
> Indeed this appears to be the case.  There is an std::map used to track
> which PVA servers have sent beacons.  Entries are added, but never removed.
> Each entry has a Mutex (this code does very granular locking...).
I also notice that a temporary Mutex is allocated
while processing each search message.  Not a leak,
but also not so efficient.


I've been playing around with "bpftrace" as an
alternative to "systemtap".  (imo. systemtap is
great, but decoding its error messages is no fun)

https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md

> $ cat rxmutex.bpf
> // Print user stack trace when epicsMutexOsiCreate() called through processBuffer()
>
> // unmangled name...
> // epics::pvAccess::BlockingUDPTransport::processBuffer(
> //  std::shared_ptr<epics::pvAccess::Transport> const&, osiSockAddr&,
> //  osiSockAddr& fromAddress, ByteBuffer* receiveBuffer)
>
> uprobe:"/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5":_ZN5epics8pvAccess20BlockingUDPTransport13processBufferERKSt10shared_ptrINS0_9TransportEER11osiSockAddrPNS_6pvData10ByteBufferE
> {
>     @inproc[tid] = 1;
> }
>
> uretprobe:"/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5":_ZN5epics8pvAccess20BlockingUDPTransport13processBufferERKSt10shared_ptrINS0_9TransportEER11osiSockAddrPNS_6pvData10ByteBufferE
> {
>     delete(@inproc[tid]);
> }
>
> uprobe:"/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1":epicsMutexOsiCreate
> {
>     if(@inproc[tid]) {
>         printf("==============\n%s\n", ustack(perf));
>     }
> }
Why yes, that c++ mangled symbol name is 127 characters long...


> $ sudo bpftrace -p `pgrep softIocPVA` rxmutex.bpf
> Attaching 3 probes...
While running "pvget"

> ==============
>
>         7f474e0e2d53 epicsMutexOsiCreate+0 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>         7f474dbad4dc epics::pvAccess::ServerChannelFindRequesterImpl::ServerChannelFindRequesterImpl(std::shared_ptr<epics::pvAccess::ServerContextImpl> const&, std::shared_ptr<epics::pvAccess::PeerInfo const> const&, int)+308 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>         7f474dbaced7 epics::pvAccess::ServerSearchHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+1897 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>         7f474dbac094 epics::pvAccess::ServerResponseHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+380 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>         7f474db420b8 epics::pvAccess::BlockingUDPTransport::processBuffer(std::shared_ptr<epics::pvAccess::Transport> const&, osiSockAddr&, epics::pvData::ByteBuffer*)+736 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>         7fffffffe000 0x7fffffffe000 ([unknown])
>         7f474e0e1869 epicsThreadCallEntryPoint+97 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>         7f474e0e9d96 start_routine+431 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>         7f474d764ea7 start_thread+215 (/usr/lib/x86_64-linux-gnu/libpthread-2.31.so)
While running another PVA server

> ==============
>
>         7f474e0e2d53 epicsMutexOsiCreate+0 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>         7f474db49e68 epics::pvAccess::BeaconHandler::BeaconHandler(std::shared_ptr<epics::pvAccess::Context> const&, osiSockAddr const*)+86 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>         7f474db8c2cc (anonymous namespace)::InternalClientContextImpl::getBeaconHandler(osiSockAddr*)+244 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>         7f474db83bfc (anonymous namespace)::BeaconResponseHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+678 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>         7f474db85b8d (anonymous namespace)::ClientResponseHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+349 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>         7f474db420b8 epics::pvAccess::BlockingUDPTransport::processBuffer(std::shared_ptr<epics::pvAccess::Transport> const&, osiSockAddr&, epics::pvData::ByteBuffer*)+736 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>         7fffffffe000 0x7fffffffe000 ([unknown])
>         7f474e0e1869 epicsThreadCallEntryPoint+97 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>         7f474e0e9d96 start_routine+431 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>         7f474d764ea7 start_thread+215 (/usr/lib/x86_64-linux-gnu/libpthread-2.31.so)
...