Re: [EXTERNAL] Re: Channel Access performance with RTEMS on MVME5500

[Joel Sherrill via Tech-talk <tech-talk at aps.anl.gov>]

Just an FYI that a mutex may be tracking who locked it to ensure it is unlocked by the same thread. This is required to ensure priority inversion avoidance protocols can implement their tracking. 

--joel

On Thu, Mar 20, 2025 at 10:52 AM Michael Davidsaver via Tech-talk <tech-talk at aps.anl.gov> wrote:

On 3/20/25 08:42, Johnson, Andrew N. wrote:
> Hi Alex,
>
> I will third these recommendations, they probably explain why your IOC was hanging.
>
> I don’t think a spin-lock is likely to be useful on an MVME5500 since that isn’t SMP, so while it’s spinning inside the ISR there’s no other CPU available to finish off whatever currently has the lock held (unless that would be released by a higher priority interrupt, which seems very unlikely).

I default to recommending epicsSpin for portability.

On UP RTEMS/vxWorks epicsSpin is a portable way to disables interrupts.

On other targets it is a non-recursive PI Mutex.


> - Andrew
>
> --
>
> Complexity comes for free, Simplicity you have to work for.
>
> On 3/20/25, 10:22 AM, "Tech-talk" <tech-talk-bounces at aps.anl.gov> wrote:
>
> Hi Alex,
>
> On 3/20/25 06:39, Kasemir, Kay via Tech-talk wrote:
>
>     > EPICS Mutexes in an interrupt context … crashes
>
>     No idea about RTEMS in particular, but you can generally not _/take/_ a mutex in an interrupt context.
>
> Regardless of USE_POSIX_THREAD_PRIORITY_SCHEDULING, locking a contended mutex from interrupt context WILL DEADLOCK.  Disabling priority inheritance may make contention less likely in some situations, but this does not change the fundamentals.  Any use of epicsMutex outside of regular/thread context is Undefined Behavior.
>
>     You may be able to _/give/_ a mutex to wake a thread that’s waiting on that mutex, but details will depend on the OS.
>
>     You can use epicsEvent from within an interrupt service routine to signal a thread that it should start doing some work.
>
>     If the interrupt context currently uses mutexes, it might be best to rewrite it to use epicsEvent and maybe move work out of the interrupt service routine into a thread that’s woken by the event.
>
> I will second this recommendation.  I also find that using epicsEvent as a tool to hand work to a thread is often the best design.
>
> Alternately, if you are looking of a more direct search/replace fix, then I would recommend looking at epicsSpin (aka. spin-lock) which is a mutual exclusion primitive intended to be IRQ safe.
>
>     On 2025-03-20, 9:32 AM, "Tech-talk" <tech-talk-bounces at aps.anl.gov> <mailto:tech-talk-bounces at aps.anl.gov> wrote:
>
>     Hello Alex,
>
>     the ‘USE_POSIX_THREAD_PRIORITY_SCHEDULING = NO’' is an EPICS thing.
>
>     I don't know exactly what this does as I have never used it with RTEMS.
>
>     I think that cancels out the whole point of a priority-based system ;-)
>
>     All your jobs have a very low priority 253 (in RTEMS) and priority 2 in Posix.
>
>     Based on my gut feeling, I would recommend setting back to USE_POSIX_THREAD_PRIORITY_SCHEDULING=YES and then
>
>     in rtems_init for POSIX_Init() to set the priority high. You can use the code element I sent to you in the last tech-talk-reply.
>
>     Then all the interrupt handlers will also get a high priority and I assume that the crashes will also go away.
>
>     Would you like to try that?
>
>     Good luck,
>
>     Heinz
>
>      > On 20. Mar 2025, at 12:50, Wells, Alex (DLSLtd,RAL,LSCI) <alex.wells at diamond.ac.uk <mailto:alex.wells at diamond.ac.uk>> wrote:
>
>      >
>
>      > Hi Heinz,
>
>      >
>
>      > Thank you for the information. There's definitely something fishy going on with the thread priorities somewhere; it seems that everything is being started with the same priority.
>
>      >
>
>      > I will also note that we are building RTEMS with "USE_POSIX_THREAD_PRIORITY_SCHEDULING = NO". This appears to be required, as we use EPICS Mutexes in an interrupt context, and if we leave this set to "YES" we see a crashes. Could this be related to the oddities below?
>