Hello,
Any news on a fix for this bug?
Would removing PVA/QSRV from the binary solve the problem in this ioc for now? This is an MPS Ioc and rebooting trips beam. So if I can find a temporary solution that would be great until we can get a bug fix.
MpsSupport_DBD += PVAServerRegister.dbd
MpsSupport_DBD += qsrv.dbd
MpsSupportSoft_LIBS += qsrv
USR_LIBS += $(EPICS_BASE_PVA_CORE_LIBS)
-Kristi
-----Original Message-----
From: Luchini, Kristi L.
Sent: Thursday, May 26, 2022 2:57 PM
To: Michael Davidsaver <mdavidsaver at gmail.com>
Cc: Talk EPICS Tech <tech-talk at aps.anl.gov>; Till Straumann <till.straumann at psi.ch>; Joel Sherrill <joel.sherrill at gmail.com>
Subject: Re: Can't create mutex semaphore: too many
Yes, I am using Till’s RTaeME configuration.
Kristi
Sent from my iPhone
> On May 26, 2022, at 9:33 AM, Michael Davidsaver <mdavidsaver at gmail.com> wrote:
>
> On 5/25/22 14:15, Michael Davidsaver wrote:
>> Indeed this appears to be the case. There is an std::map used to
>> track which PVA servers have sent beacons. Entries are added, but never removed.
>> Each entry has a Mutex (this code does very granular locking...).
>
> I also notice that a temporary Mutex is allocated while processing
> each search message. Not a leak, but also not so efficient.
>
>
> I've been playing around with "bpftrace" as an alternative to
> "systemtap". (imo. systemtap is great, but decoding its error
> messages is no fun)
>
> https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.m
> d
>
>> $ cat rxmutex.bpf
>> // Print user stack trace when epicsMutexOsiCreate() called through
>> processBuffer() // unmangled name...
>> // epics::pvAccess::BlockingUDPTransport::processBuffer(
>> // std::shared_ptr<epics::pvAccess::Transport> const&, osiSockAddr&,
>> // osiSockAddr& fromAddress, ByteBuffer* receiveBuffer)
>> uprobe:"/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-deb
>> ug/libpvAccess.so.7.1.5":_ZN5epics8pvAccess20BlockingUDPTransport13pr
>> ocessBufferERKSt10shared_ptrINS0_9TransportEER11osiSockAddrPNS_6pvDat
>> a10ByteBufferE
>> {
>> @inproc[tid] = 1;
>> }
>> uretprobe:"/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-
>> debug/libpvAccess.so.7.1.5":_ZN5epics8pvAccess20BlockingUDPTransport1
>> 3processBufferERKSt10shared_ptrINS0_9TransportEER11osiSockAddrPNS_6pv
>> Data10ByteBufferE
>> {
>> delete(@inproc[tid]);
>> }
>> uprobe:"/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/
>> libCom.so.3.21.1":epicsMutexOsiCreate
>> {
>> if(@inproc[tid]) {
>> printf("==============\n%s\n", ustack(perf));
>> }
>> }
>
> Why yes, that c++ mangled symbol name is 127 characters long...
>
>
>> $ sudo bpftrace -p `pgrep softIocPVA` rxmutex.bpf Attaching 3
>> probes...
>
> While running "pvget"
>
>> ==============
>> 7f474e0e2d53 epicsMutexOsiCreate+0 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>> 7f474dbad4dc epics::pvAccess::ServerChannelFindRequesterImpl::ServerChannelFindRequesterImpl(std::shared_ptr<epics::pvAccess::ServerContextImpl> const&, std::shared_ptr<epics::pvAccess::PeerInfo const> const&, int)+308 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>> 7f474dbaced7 epics::pvAccess::ServerSearchHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+1897 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>> 7f474dbac094 epics::pvAccess::ServerResponseHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+380 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>> 7f474db420b8 epics::pvAccess::BlockingUDPTransport::processBuffer(std::shared_ptr<epics::pvAccess::Transport> const&, osiSockAddr&, epics::pvData::ByteBuffer*)+736 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>> 7fffffffe000 0x7fffffffe000 ([unknown])
>> 7f474e0e1869 epicsThreadCallEntryPoint+97 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>> 7f474e0e9d96 start_routine+431 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>> 7f474d764ea7 start_thread+215
>> (/usr/lib/x86_64-linux-gnu/libpthread-2.31.so)
>
> While running another PVA server
>
>> ==============
>> 7f474e0e2d53 epicsMutexOsiCreate+0 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>> 7f474db49e68 epics::pvAccess::BeaconHandler::BeaconHandler(std::shared_ptr<epics::pvAccess::Context> const&, osiSockAddr const*)+86 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>> 7f474db8c2cc (anonymous namespace)::InternalClientContextImpl::getBeaconHandler(osiSockAddr*)+244 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>> 7f474db83bfc (anonymous namespace)::BeaconResponseHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+678 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>> 7f474db85b8d (anonymous namespace)::ClientResponseHandler::handleResponse(osiSockAddr*, std::shared_ptr<epics::pvAccess::Transport> const&, signed char, signed char, unsigned long, epics::pvData::ByteBuffer*)+349 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>> 7f474db420b8 epics::pvAccess::BlockingUDPTransport::processBuffer(std::shared_ptr<epics::pvAccess::Transport> const&, osiSockAddr&, epics::pvData::ByteBuffer*)+736 (/home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccess.so.7.1.5)
>> 7fffffffe000 0x7fffffffe000 ([unknown])
>> 7f474e0e1869 epicsThreadCallEntryPoint+97 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>> 7f474e0e9d96 start_routine+431 (/home/mdavidsaver/work/epics/base-git/lib/linux-x86_64-debug/libCom.so.3.21.1)
>> 7f474d764ea7 start_thread+215
>> (/usr/lib/x86_64-linux-gnu/libpthread-2.31.so)
> ...
- Replies:
- Re: Can't create mutex semaphore: too many Michael Davidsaver via Tech-talk
- References:
- Re: Can't create mutex semaphore: too many Michael Davidsaver via Tech-talk
- Re: Can't create mutex semaphore: too many Luchini, Kristi L. via Tech-talk
- Navigate by Date:
- Prev:
RE: Strange memory leak with ADAravis Mark Rivers via Tech-talk
- Next:
Re: Can't create mutex semaphore: too many Michael Davidsaver via Tech-talk
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
<2022>
2023
2024
- Navigate by Thread:
- Prev:
Re: Can't create mutex semaphore: too many Luchini, Kristi L. via Tech-talk
- Next:
Re: Can't create mutex semaphore: too many Michael Davidsaver via Tech-talk
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
<2022>
2023
2024
|
·
Home
·
News
·
About
·
Base
·
Modules
·
Extensions
·
Distributions
·
Download
·