Experimental Physics and
Industrial Control System

To provide a bit of background, the POSIX standard has no concept
of hardware interrupts. On top of that, mutexes are designed to allow
a thread to lock a resource -- they are not (in Classic API terms), simple
binary semaphores which can be used for condition signaling. They always
have a "holder" and only the holder thread can release the mutex. This means
that mutexes must be acquired by threads and released by the same thread
that acquired that mutex (e.g. it is the holder). On top of that,
priority inversion
avoidance protocols like priority inheritance and priority ceiling protocols
implicitly assume that a thread holds the mutex since it needs to change
the priority.

Even in the Classic API, if you configure a semaphore as a true mutex
(e.g. Binary Semaphore in Classic API terms), it would have to be used
under the same restrictions.

Having debugged systems which violated the rules and released mutexes
from ISR handlers before this fatal error case was present, we should
really appreciate it being caught. It is very hard to track down and debug
otherwise.

I'm glad this already has a code fix but understanding the restriction is
always good.

--joel

On Wed, Jan 12, 2022 at 9:18 AM Heinz Junkes via Tech-talk
<tech-talk at aps.anl.gov> wrote:
>
> Sorry send the wrong link.
> This one might fitbetter : https://github.com/epics-base/epics-base/issues/202
>
>
> Viele Grüße
> Heinz Junkes
> --
> Experience directly varies with equipment ruined.
>
>
>
> > On 12. Jan 2022, at 15:24, Francisco Ramos via Tech-talk <tech-talk at aps.anl.gov> wrote:
> >
> > Dear community,
> >
> >     I'm writing to see if you can help me to solve a problem I'm doing a migrating the "PMAC-VME Device Driver" to EPCIS version 7.0.6.1. with rtems5. The driver is working for EPCIS version 3.14.8 and RTEMS 4.10.
> >     The code compiles perfectly for Epics 7 and rtems5, but when I deploy it on the MVM2700 LCU, I get the following error: Error code 30 (INTERNAL_ERROR_BAD_THREAD_DISPATCH_DISABLE_LEVEL). The RTEMS documentation describes the error as follows:
> >         INTERNAL_ERROR_BAD_THREAD_DISPATCH_DISABLE_LEVEL (30)
> >              It is illegal to call blocking operating system services with thread dispatching disabled, for example in interrupt context.
> >
> >     I attach the Stack Trace at the end of the sms.
> >
> >     I have run addr2line to get more information and you can give me suggestions on how I might fix the problem. Thanks in advance for your help.
> >
> >     In the following table, I have put the first group of "Stack Trace" for easier reading.  I copy below the code fragment of the "PMAC-VME Device Driver" referring to the line marked in bold in the table.
> >
> > static int mbxRcptISRcnt = 0;
> > PMAC_LOCAL void pmacMbxReceiptISR(void *p)
> > {
> >    PMAC_CTLR   *pPmacCtlr = (PMAC_CTLR *)p;
> >
> >    cntRxISR++;
> >    epicsEventSignal (pPmacCtlr->ioMbxReceiptSem);
> >    mbxRcptISRcnt++;    // line 130 of the drvPmacVme.c file.
> >    return;
> > }
> >
> >     After the table, I attach the Stack Trace output and the ADDR2LINE output.
> >
> > ADDR  function        File And Line
> > 0x00254f84    bsp_reset       /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/lib/libbsp/powerpc/motorola_powerpc/../../../../../../../../rtems/c/src/lib/libbsp/powerpc/motorola_powerpc/../../../../../../bsps/powerpc/motorola_powerpc/start/bspreset.c:16
> > 0x00254ec4    bsp_fatal_extension     /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/lib/libbsp/powerpc/motorola_powerpc/../../../../../../../../rtems/c/src/lib/libbsp/powerpc/motorola_powerpc/../../../../../../bsps/shared/start/bspfatal-default.c:99
> > 0x00135648    _User_extensions_Iterate        /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/cpukit/../../../../../rtems/c/src/../../cpukit/score/src/userextiterate.c:172
> > 0x00135854    _Terminate      /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/cpukit/../../../../../rtems/c/src/../../cpukit/score/src/interr.c:38
> > 0x001358ac    _Workspace_Handler_initialization       /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/cpukit/../../../../../rtems/c/src/../../cpukit/score/src/wkspace.c:54
> > 0x001321f4    _Thread_Dispatch_enable /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/cpukit/../../../../../rtems/c/src/../../cpukit/score/src/threaddispatch.c:364
> > 0x0012aa4c    _POSIX_Get_error        /gem_base/targetOS/RTEMS/rtems/kernel/rtems/cpukit/include/rtems/posix/posixapi.h:70
> > 0x0010b08c    epicsEventWait  /builddir/build/BUILD/epics-base-git-13.a417dec/modules/libcom/src/O.RTEMS-mvme2700/../osi/os/posix/osdEvent.c:105
> > 0x001039e8    epicsEventMustWait      /builddir/build/BUILD/epics-base-git-13.a417dec/modules/libcom/src/O.RTEMS-mvme2700/../osi/epicsEvent.cpp:127
> > 0x000f502c    errlogFlush     /builddir/build/BUILD/epics-base-git-13.a417dec/modules/libcom/src/O.RTEMS-mvme2700/../error/errlog.c:551
> > 0x000fd6cc    cantProceed     /builddir/build/BUILD/epics-base-git-13.a417dec/modules/libcom/src/O.RTEMS-mvme2700/../misc/cantProceed.c:69
> > 0x0001360c    pmacMbxReceiptISR       /home/framos/work/pmaclib.git/pmaclibApp/src/O.RTEMS-mvme2700/../drvPmacVme.c:130
> > 0x0025f628    universeVMEISR  /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/lib/libbsp/powerpc/motorola_powerpc/../../../../../../../../rtems/c/src/lib/libbsp/powerpc/motorola_powerpc/../../../../../../bsps/powerpc/shared/vme/vmeUniverse.c:1978
> > 0x00262f64    bsp_interrupt_handler_dispatch  /gem_base/targetOS/RTEMS/rtems/kernel/rtems/bsps/include/bsp/irq-generic.h:274
> > 0x0025dd08    wrap_change_msr_done_std        /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/lib/libbsp/powerpc/motorola_powerpc/../../../../../../../../rtems/c/src/lib/libbsp/powerpc/motorola_powerpc/../../../../../../bsps/powerpc/shared/exceptions/ppc_exc.S:129
> >
> >
> > Stack Trace
> > fatal source: 0 (INTERNAL_ERROR_CORE)
> > fatal code: 30 (INTERNAL_ERROR_BAD_THREAD_DISPATCH_DISABLE_LEVEL)
> > RTEMS version: 5.0.0.645dbc5fccd8b73e12ca17ab35962fe751854913
> > RTEMS tools: 7.5.0 20191114 (RTEMS 5, RSB 5 (803d42cda7b3), Newlib 7947581)
> > executing thread ID: 0x08b010010
> > executing thread name:
> > Printing a stack trace for your convenience :-)
> >
> > *** FATAL ***
> > fatal source: 9 (RTEMS_FATAL_SOURCE_EXCEPTION)
> > exception vector 3 (0x3)
> >   next PC or address of fault = 0x0025cb8c
> >   saved MSR = 0x00001032
> >   context = interrupt, ISR nest level = 1
> >   thread dispatch disable level = 2
> >   R0  = 0x00254f84 R1  = 0x00494758 R2  = 0x00000000 R3  = 0x00000030
> >   R4  = 0x00000000 R5  = 0x002bca1c R6  = 0x004947b0 R7  = 0x0025dd08
> >   R8  = 0x004947a0 R9  = 0xa5a5a5a5 R10 = 0x004947a0 R11 = 0x00000000
> >   R12 = 0x40842402 R13 = 0x003006c0 R14 = 0x00ef1640 R15 = 0x00000000
> >   R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
> >   R20 = 0x00000000 R21 = 0x00000000 R22 = 0x00000000 R23 = 0x00000000
> >   R24 = 0x00336560 R25 = 0x00000000 R26 = 0x002a41b0 R27 = 0x00135580
> >   R28 = 0x004948a8 R29 = 0x00494760 R30 = 0x0000001e R31 = 0x003174e8
> >   CR  = 0x40842408
> >   CTR = 0x00000017
> >   XER = 0x20000000
> >   LR  = 0x00254f88
> >   DAR = 0xa5a5a5a9
> >   executing thread ID = 0x0b010010, name =
> > Stack Trace:
> >   IP: 0x0025cb8c, LR: 0x00254f88
> > --^ 0x00254f84--^ 0x00254ec4--^ 0x00135648--^ 0x00135854--^ 0x001358ac
> > --^ 0x001321f4--^ 0x0012aa4c--^ 0x0010b08c--^ 0x001039e8--^ 0x000f502c
> > --^ 0x000fd6cc--^ 0x0001360c--^ 0x0025f628--^ 0x00262f64--^ 0x0025dd08
> >
> > *** FATAL ***
> > fatal source: 9 (RTEMS_FATAL_SOURCE_EXCEPTION)
> > exception vector 3 (0x3)
> >   next PC or address of fault = 0x0025d240
> >   saved MSR = 0x00001032
> >   context = interrupt, ISR nest level = 1
> >   thread dispatch disable level = 2
> >   R0  = 0x0025d270 R1  = 0x004945e8 R2  = 0x00000000 R3  = 0x002bdd8c
> >   R4  = 0x00000000 R5  = 0x002784e8 R6  = 0x00494578 R7  = 0x002784ea
> >   R8  = 0x002784e9 R9  = 0xa5a5a5a5 R10 = 0x800003fd R11 = 0x00000000
> >   R12 = 0x40442202 R13 = 0x003006c0 R14 = 0x004946a8 R15 = 0x00000000
> >   R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
> >   R20 = 0x00000000 R21 = 0x00000000 R22 = 0x00000000 R23 = 0x00000000
> >   R24 = 0x00336560 R25 = 0x00000000 R26 = 0x002a41b0 R27 = 0x002784e8
> >   R28 = 0xcccccccd R29 = 0x002bdd8c R30 = 0x00494a10 R31 = 0x00000010
> >   CR  = 0x40442808
> >   CTR = 0x00254fe8
> >   XER = 0x20000000
> >   LR  = 0x0025d270
> >   DAR = 0xa5a5a5a9
> >   executing thread ID = 0x0b010010, name =
> > Stack Trace:
> >   IP: 0x0025d240, LR: 0x0025d270
> > --^ 0x00254f64--^ 0x00135648--^ 0x00135854--^ 0x002662f0--^ 0x0025de68
> > --^ 0x40842402--^ 0x00254f84--^ 0x00254ec4--^ 0x00135648--^ 0x00135854
> > --^ 0x001358ac--^ 0x001321f4--^ 0x0012aa4c--^ 0x0010b08c--^ 0x001039e8
> > --^ 0x000f502c--^ 0x000fd6cc--^ 0x0001360c--^ 0x0025f628--^ 0x00262f64
> > --^ 0x0025dd08
> >
> > ADDR2LINE
> >
> > line to Analyzed  0x00254f84 0x00254ec4 0x00135648 0x00135854 0x001358ac
> > ADDR: 0x00254f84 -> bsp_reset
> > /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/lib/libbsp/powerpc/motorola_powerpc/../../../../../../../../rtems/c/src/lib/libbsp/powerpc/motorola_powerpc/../../../../../../bsps/powerpc/motorola_powerpc/start/bspreset.c:16
> > ADDR: 0x00254ec4 -> bsp_fatal_extension
> > /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/lib/libbsp/powerpc/motorola_powerpc/../../../../../../../../rtems/c/src/lib/libbsp/powerpc/motorola_powerpc/../../../../../../bsps/shared/start/bspfatal-default.c:99
> > ADDR: 0x00135648 -> _User_extensions_Iterate
> > /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/cpukit/../../../../../rtems/c/src/../../cpukit/score/src/userextiterate.c:172
> > ADDR: 0x00135854 -> _Terminate
> > /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/cpukit/../../../../../rtems/c/src/../../cpukit/score/src/interr.c:38
> > ADDR: 0x001358ac -> _Workspace_Handler_initialization
> > /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/cpukit/../../../../../rtems/c/src/../../cpukit/score/src/wkspace.c:54
> > line to Analyzed  0x001321f4 0x0012aa4c 0x0010b08c 0x001039e8 0x000f502c
> > ADDR: 0x001321f4 -> _Thread_Dispatch_enable
> > /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/cpukit/../../../../../rtems/c/src/../../cpukit/score/src/threaddispatch.c:364
> > ADDR: 0x0012aa4c -> _POSIX_Get_error
> > /gem_base/targetOS/RTEMS/rtems/kernel/rtems/cpukit/include/rtems/posix/posixapi.h:70
> > ADDR: 0x0010b08c -> epicsEventWait
> > /builddir/build/BUILD/epics-base-git-13.a417dec/modules/libcom/src/O.RTEMS-mvme2700/../osi/os/posix/osdEvent.c:105
> > ADDR: 0x001039e8 -> epicsEventMustWait
> > /builddir/build/BUILD/epics-base-git-13.a417dec/modules/libcom/src/O.RTEMS-mvme2700/../osi/epicsEvent.cpp:127
> > ADDR: 0x000f502c -> errlogFlush
> > /builddir/build/BUILD/epics-base-git-13.a417dec/modules/libcom/src/O.RTEMS-mvme2700/../error/errlog.c:551
> > line to Analyzed  0x000fd6cc 0x0001360c 0x0025f628 0x00262f64 0x0025dd08
> > ADDR: 0x000fd6cc -> cantProceed
> > /builddir/build/BUILD/epics-base-git-13.a417dec/modules/libcom/src/O.RTEMS-mvme2700/../misc/cantProceed.c:69
> > ADDR: 0x0001360c -> pmacMbxReceiptISR
> > /home/framos/work/pmaclib.git/pmaclibApp/src/O.RTEMS-mvme2700/../drvPmacVme.c:130
> > ADDR: 0x0025f628 -> universeVMEISR
> > /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/lib/libbsp/powerpc/motorola_powerpc/../../../../../../../../rtems/c/src/lib/libbsp/powerpc/motorola_powerpc/../../../../../../bsps/powerpc/shared/vme/vmeUniverse.c:1978
> > ADDR: 0x00262f64 -> bsp_interrupt_handler_dispatch
> > /gem_base/targetOS/RTEMS/rtems/kernel/rtems/bsps/include/bsp/irq-generic.h:274
> > ADDR: 0x0025dd08 -> wrap_change_msr_done_std
> > /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/lib/libbsp/powerpc/motorola_powerpc/../../../../../../../../rtems/c/src/lib/libbsp/powerpc/motorola_powerpc/../../../../../../bsps/powerpc/shared/exceptions/ppc_exc.S:129
> > line to Analyzed
> > line to Analyzed  0x00254f64 0x00135648 0x00135854 0x002662f0 0x0025de68
> > ADDR: 0x00254f64 -> rtems_exception_frame_print
> > /gem_base/targetOS/RTEMS/rtems/kernel/rtems/cpukit/include/rtems/fatal.h:60
> > ADDR: 0x00135648 -> _User_extensions_Iterate
> > /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/cpukit/../../../../../rtems/c/src/../../cpukit/score/src/userextiterate.c:172
> > ADDR: 0x00135854 -> _Terminate
> > /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/cpukit/../../../../../rtems/c/src/../../cpukit/score/src/interr.c:38
> > ADDR: 0x001358ac -> _Workspace_Handler_initialization
> > /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/cpukit/../../../../../rtems/c/src/../../cpukit/score/src/wkspace.c:54
> > line to Analyzed  0x001321f4 0x0012aa4c 0x0010b08c 0x001039e8 0x000f502c
> > ADDR: 0x001321f4 -> _Thread_Dispatch_enable
> > /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/cpukit/../../../../../rtems/c/src/../../cpukit/score/src/threaddispatch.c:364
> > ADDR: 0x0012aa4c -> _POSIX_Get_error
> > /gem_base/targetOS/RTEMS/rtems/kernel/rtems/cpukit/include/rtems/posix/posixapi.h:70
> > ADDR: 0x0010b08c -> epicsEventWait
> > /builddir/build/BUILD/epics-base-git-13.a417dec/modules/libcom/src/O.RTEMS-mvme2700/../osi/os/posix/osdEvent.c:105
> > ADDR: 0x001039e8 -> epicsEventMustWait
> > /builddir/build/BUILD/epics-base-git-13.a417dec/modules/libcom/src/O.RTEMS-mvme2700/../osi/epicsEvent.cpp:127
> > ADDR: 0x000f502c -> errlogFlush
> > /builddir/build/BUILD/epics-base-git-13.a417dec/modules/libcom/src/O.RTEMS-mvme2700/../error/errlog.c:551
> > line to Analyzed  0x000fd6cc 0x0001360c 0x0025f628 0x00262f64 0x0025dd08
> > ADDR: 0x000fd6cc -> cantProceed
> > /builddir/build/BUILD/epics-base-git-13.a417dec/modules/libcom/src/O.RTEMS-mvme2700/../misc/cantProceed.c:69
> > ADDR: 0x0001360c -> pmacMbxReceiptISR
> > /home/framos/work/pmaclib.git/pmaclibApp/src/O.RTEMS-mvme2700/../drvPmacVme.c:130
> > ADDR: 0x0025f628 -> universeVMEISR
> > /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/lib/libbsp/powerpc/motorola_powerpc/../../../../../../../../rtems/c/src/lib/libbsp/powerpc/motorola_powerpc/../../../../../../bsps/powerpc/shared/vme/vmeUniverse.c:1978
> > ADDR: 0x00262f64 -> bsp_interrupt_handler_dispatch
> > /gem_base/targetOS/RTEMS/rtems/kernel/rtems/bsps/include/bsp/irq-generic.h:274
> > ADDR: 0x0025dd08 -> wrap_change_msr_done_std
> > /gem_base/targetOS/RTEMS/rtems/kernel/mvme2307/powerpc-rtems5/c/mvme2307/lib/libbsp/powerpc/motorola_powerpc/../../../../../../../../rtems/c/src/lib/libbsp/powerpc/motorola_powerpc/../../../../../../bsps/powerpc/shared/exceptions/ppc_exc.S:129
> >
> > Regards,
> >
> >
> >
>