RE: Can't create mutex semaphore: too many

["Luchini, Kristi L. via Tech-talk" <tech-talk at aps.anl.gov>]

Hi Michael,

Yes, I thought the same thing initially. However, I checked the memory, cpu usage over time, MBUFs, File Descriptors over time and it looked ok.  There may be some other system resources that I have not yet looked at.. but 

-Kristi

-----Original Message-----
From: Michael Davidsaver <mdavidsaver at gmail.com> 
Sent: Monday, May 9, 2022 3:45 PM
To: Luchini, Kristi L. <luchini at slac.stanford.edu>
Cc: tech-talk at aps.anl.gov
Subject: Re: Can't create mutex semaphore: too many

On 5/9/22 15:08, Luchini, Kristi L. via Tech-talk wrote:
> Hello,
> 
> I have a MVME6100 ioc running RTEMS that appears to be having a resource issue.  Does anyone recognize these error messages below and happen to know function call that I can add to the st.cmd to increase that resource? Do  I need to increase the number file descriptors?
> 
> If I reboot the ioc the message will go away and the iocs works fine, but after some period of time, a month or so, this message reappears and you can’t access pvs over CA, which I assume is because the iocs seems to be too busy spewing error messages.
> 
> Any help would be appreciated.

Sounds like there is a resource leak somewhere, at least of RTEMS semaphores.
I'm not sure if the mention of "file descriptor" is real, or a case of overloading the meaning of an errno code.

Which drivers and other code is being loaded?

Are any of these creating worker threads after iocInit() ?

Are there any earlier error messages being logged?

You could try running "epicsThreadShowAll()" to see if some driver threads are hanging around when they shouldn't.

Unfortunately, there is no "epicsMutexShowAll" implemented for RTEMS 4.X, and I don't know if your RTEMS builds include the RTEMS shell and its associated diagnostics functions.


> Thanks,
> 
> Kristi
> 
> Message from RTEMS ioc:
> 
> Can't create mutex semaphore: too many
> 
> Can't create mutex semaphore: too many
> 
> Can't create mutex semaphore: too many
> 
> Can't create mutex semaphore: too many
> 
> Cexp@ioc-in10-mp01>
> 
> Cexp@ioc-in10-mp01>Error on UDP RX 172.27.72.43:47591 -> 
> 172.27.75.255:5076 at 46 : Can't create mutex semaphore: too many
> 
> epicsMutex::mutexCreateFailed()
> 
> 0x00 ca024000 27000000 d5b37962 00000000  ..@. '... ..yb ....
> 
> 0x10 c84f9922 00740000 00000000 00000000  .O." .t.. .... ....
> 
> 0x20 0000ffff 00000000 d3130374 6370ff    .... .... ...t cp.
> 
> CAS: Client accept error: Too many open files in system (16)
> 
> Error on UDP RX 172.27.72.99:37311 -> 172.27.75.255:5076 at 46 : Can't 
> create mutex semaphore: too many
> 
> epicsMutex::mutexCreateFailed()
> 
> 0x00 ca024000 27000000 a5a77862 00000000  ..@. '... ..xb ....
> 
> 0x10 ea850c38 004f0000 00000000 00000000  ...8 .O.. .... ....
> 
> 0x20 0000ffff 00000000 d3130374 6370ff    .... .... ...t cp.
> 
>   * Kristi
>