Re: Can't create mutex semaphore: too many

[Joel Sherrill via Tech-talk <tech-talk at aps.anl.gov>]

On Mon, May 9, 2022, 7:41 PM Michael Davidsaver <mdavidsaver at gmail.com> wrote:

On 5/9/22 16:42, Joel Sherrill wrote:
>
>
> On Mon, May 9, 2022, 5:45 PM Michael Davidsaver via Tech-talk <tech-talk at aps.anl.gov <mailto:tech-talk at aps.anl.gov>> wrote:
>
>     On 5/9/22 15:08, Luchini, Kristi L. via Tech-talk wrote:
>      > Hello,
>      >
>      > I have a MVME6100 ioc running RTEMS that appears to be having a resource issue.  Does anyone recognize these error messages below and happen to know function call that I can add to the st.cmd to increase that resource? Do  I need to increase the number file descriptors?
>      >
>      > If I reboot the ioc the message will go away and the iocs works fine, but after some period of time, a month or so, this message reappears and you can’t access pvs over CA, which I assume is because the iocs seems to be too busy spewing error messages.
>      >
>      > Any help would be appreciated.
>
>     Sounds like there is a resource leak somewhere, at least of RTEMS semaphores.
>     I'm not sure if the mention of "file descriptor" is real, or a case of
>     overloading the meaning of an errno code.
>
>     Which drivers and other code is being loaded?
>
>     Are any of these creating worker threads after iocInit() ?
>
>     Are there any earlier error messages being logged?
>
>     You could try running "epicsThreadShowAll()" to see if some driver threads are
>     hanging around when they shouldn't.
>
>     Unfortunately, there is no "epicsMutexShowAll" implemented for RTEMS 4.X, and
>     I don't know if your RTEMS builds include the RTEMS shell and its associated
>     diagnostics functions.
>
>
> I don't know what RTEMS configure options are used but there is an option for unlimited and another for unified workspace. If they both are configured, you have one pool of memory to allocate everything from. This includes RTEMS objects and things you allocate via malloc() and new. In this configuration, any leak would be bad and could lead to this. It would just be a matter of which create or allocate failed. Malloc() will return NULL in this situation and that often isn't error checked.

@joel, I'm 90% certain that Kristi is using Till's GESYS application,
which has a different RTEMS configuration.  So the following isn't
relevant for @SLAC.

Still.  fyi.  The current 7.0 default RTEMS configurations for EPICS applications are:

for RTEMS  <= 4.x which uses separate pools.

https://github.com/epics-base/epics-base/blob/cbae8d37b3da486ac8a68ad6ef9d2028cd98cca0/modules/libcom/RTEMS/score/rtems_config.c#L29-L38

I think that for 4.10, the cpp logic is setting unified workspace. If I remember correctly that's the first version with that feature.

for RTEMS >= 5.x which sets CONFIGURE_UNIFIED_WORK_AREAS

https://github.com/epics-base/epics-base/blob/cbae8d37b3da486ac8a68ad6ef9d2028cd98cca0/modules/libcom/RTEMS/posix/rtems_config.c#L64-L7

This uses 64 as the maximum file descriptors while the previous had a much larger number. But 64 is still a lot. You shouldn't run out. :)

Kristi.. what are the configure settings for maximums/unlimited and unified workspace.

I am pretty sure unified workspace was first an option around 4.10.

--joel

> --joel
>
>
>
>      > Thanks,
>      >
>      > Kristi
>      >
>      > Message from RTEMS ioc:
>      >
>      > Can't create mutex semaphore: too many
>      >
>      > Can't create mutex semaphore: too many
>      >
>      > Can't create mutex semaphore: too many
>      >
>      > Can't create mutex semaphore: too many
>      >
>      > Cexp@ioc-in10-mp01>
>      >
>      > Cexp@ioc-in10-mp01>Error on UDP RX 172.27.72.43:47591 <http://172.27.72.43:47591> -> 172.27.75.255:5076 <http://172.27.75.255:5076> at 46 : Can't create mutex semaphore: too many
>      >
>      > epicsMutex::mutexCreateFailed()
>      >
>      > 0x00 ca024000 27000000 d5b37962 00000000  ..@. '... ..yb ....
>      >
>      > 0x10 c84f9922 00740000 00000000 00000000  .O." .t.. .... ....
>      >
>      > 0x20 0000ffff 00000000 d3130374 6370ff    .... .... ...t cp.
>      >
>      > CAS: Client accept error: Too many open files in system (16)
>      >
>      > Error on UDP RX 172.27.72.99:37311 <http://172.27.72.99:37311> -> 172.27.75.255:5076 <http://172.27.75.255:5076> at 46 : Can't create mutex semaphore: too many
>      >
>      > epicsMutex::mutexCreateFailed()
>      >
>      > 0x00 ca024000 27000000 a5a77862 00000000  ..@. '... ..xb ....
>      >
>      > 0x10 ea850c38 004f0000 00000000 00000000  ...8 .O.. .... ....
>      >
>      > 0x20 0000ffff 00000000 d3130374 6370ff    .... .... ...t cp.
>      >
>      >   * Kristi
>      >
>