EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  <20212022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  <20212022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Re: Beatnik RTEMS4.10.2 vmeTsi148 ISR: ERROR: no handler registered
From: Matt Rippa via Tech-talk <tech-talk at aps.anl.gov>
To: "Johnson, Andrew N." <anj at anl.gov>
Cc: "rtems-users at rtems.org" <users at rtems.org>, rt-swg <rt-swg at gemini.edu>, Talk EPICS Tech <tech-talk at aps.anl.gov>
Date: Fri, 12 Mar 2021 07:29:38 -1000
Hi Andrew,

Thanks for your email. It helped me learn a little more. We have a bancomm635 timing board installed, physically.
However we've stopped initializing it some time ago in our startup. So the interrupt handler isn't configured as you pointed out. I know what to do to prevent this from happening again. I think the consequence of this is extreme (our system stopped working).

I haven't tracked down vector 0x45 yet. But we also have 2 GreenSprings IPAC carriers for our canbus cards that can generate interrupts. These isr's should all be configured. This uses the drvTip810 driver. https://github.com/epics-modules/ipac/tree/2.14

We can run the t810Report 3 if we see this happen again.

-Matt

$ egrep -rn --include="*.c" BC635VEC bancomm              
bancomm/bancommApp/src/drvBc635.c:121:#define BC635VEC    0x40
bancomm/bancommApp/src/drvBc635.c:702:        if( (status = devConnectInterruptVME(BC635VEC, isr_bc635, NULL) ) != OK )
bancomm/bancommApp/src/drvBc635.c:707:        pbc635->vector = BC635VEC;      /* Interrupt vector */



On Thu, Mar 11, 2021 at 5:59 PM Johnson, Andrew N. <anj at anl.gov> wrote:
Hi Matt,

You appear to have had 2 separate unconfigured interrupts, which triggered about 5 minutes apart. Editing the log slightly to make that clearer:

> <8/3/2021 8:00:34 hst>vmeTsi148 ISR: ERROR: no handler registered (level 2) IACK 0x00000045 -- DISABLING level 2
> <8/3/2021 8:05:14 hst>vmeTsi148 ISR: ERROR: no handler registered (level 4) IACK 0x00000040 -- DISABLING level 4


I would guess that those IACK numbers 0x45 and 0x40 are VME interrupt vectors, so they probably came from one or more VME cards in the crate. My VxWorks BSPs have routines that let me see the complete interrupt vector table, but I don’t know if the Beatnik BSP has anything similar. You should have a record of what vectors are used for each VME card that can generate interrupts, so check whether those interrupts should have registered handlers or not.

It’s also possible that these interrupts might have been generated by the Tsi148 chip itself. In many VxWorks VME BSPs the on-board drivers route their interrupts through the same vector table even though there’s no specific need to do it that way. There are VMEbus SYSFail and PowerFail interrupts for example that the Tempe (Tsi148) chip can generate, although none of the chip interrupts should have been enabled without registering a handler for them first. SYSfail can be asserted by another VME board so that wouldn’t give you a definitive answer if it turned out to be that.

Hope this gives you some clues,

- Andrew



On Mar 11, 2021, at 9:35 PM, Matt Rippa via Tech-talk <tech-talk at aps.anl.gov> wrote:
>
> It looks like the IRQEntry vector triggering this interrupt was null. I'm not sure what condition would arise leading to this.
> Maybe memory corruption or a bus error.
>
> https://git.rtems.org/rtems/tree/c/src/lib/libbsp/shared/vmeUniverse/vmeTsi148.c?h=4.10.2#n1562
>
> On Thu, Mar 11, 2021 at 2:42 PM Matt Rippa <mrippa at gemini.edu> wrote:
> Hello,
>
> Our Primary Mirror control system has a beatnik bsp running on a mvme6100 with RTEMS4.10.2 and EPICS 3.14.12.8 (which understandably is not actively supported). We've experienced a single event that we've never seen before. vmeTsi148 ISR: ERROR: no handler
>
> After this it appears all interrupts were disabled and the VME bus communications are down. This system is normally well behaved with uptimes measured in months. In 2020 we had an uptime of 301 days!
>
> Has anyone seen an error like this on the beatnik bsp?
>
> Subsequent to the error which corresponds closely with a casr command, we see dbScan warnings in our logs. These persisted several times per minute until the system was reboot.
>
> Thanks for any insight.
>
> -Matt Rippa
>
>
> <8/3/2021 8:00:32 hst>pcs-mk-ioc> casr
> <8/3/2021 8:00:33 hst>Channel Access Server V4.13
> <8/3/2021 8:00:33 hst>Connected circuits:
> <8/3/2021 8:00:33 hst>TCP xxxxxxxxxxxxxxxxxxxx, V4.13, 6 Channels, Priority=0
> <8/3/2021 8:00:33 hst>TCP xxxxxxxxxxxxxxxxxxxx, V4.13, 134 Channels, Priority=80
> <8/3/2021 8:00:33 hst>TCP , V4.13, 1 Channels, Priority=0
> <8/3/2021 8:00:33 hst>TCP , V4.13, 5 Channels, Priority=0
> <8/3/2021 8:00:33 hst>TCP , V4.13, 8 Channels, Priority=0
> <8/3/2021 8:00:33 hst>TCP , V4.13, 60 Channels, Priority=0
> <8/3/2021 8:00:33 hst>TCP , V4.13, 9 Channels, Priority=0
> <8/3/2021 8:00:34 hst>TCP , V4.13, 9 Channels, Priority=0
> <8/3/2021 8:00:34 hst>TCP , V4.13, 71 Channels, Priority=0
> <8/3/2021 8:00:34 hst>TCP , V4.13, 5 Channels, Priority=0
> <8/3/2021 8:00:34 hst>TCP , V4.13, 359 Channels, Priority=20
> <8/3/2021 8:00:34 hst>TCP , 741 Channels, Priority=20
> <8/3/2021 8:00:34 hst>TCP , V4.13, 5 Channels, Priority=20
> <8/3/2021 8:00:34 hst>pcs-mk-ioc> vmeTsi148 ISR: ERROR: no handler registered (level 2) IACK 0x00000045 -- DISABLING level 2
> <8/3/2021 8:05:14 hst>vmeTsi148 ISR: ERROR: no handler registered (level 4) IACK 0x00000040 -- DISABLING level 4
> <8/3/2021 8:34:54 hst>dbScan warning from '.2 second' scan thread:
> <8/3/2021 8:34:54 hst>        Scan processing averages 0.300 seconds (0.300 .. 0.300).
> <8/3/2021 8:34:54 hst>        Over-runs have now happened 10 times in a row.
> <8/3/2021 8:34:54 hst>        To fix this, move some records to a slower scan rate.
> <8/3/2021 8:34:54 hst>
> <8/3/2021 8:35:09 hst>dbScan warning from '.2 second' scan thread:
> <8/3/2021 8:35:09 hst>        Scan processing averages 0.300 seconds (0.300 .. 0.300).
> <8/3/2021 8:35:09 hst>        Over-runs have now happened 10 times in a row.
> <8/3/2021 8:35:09 hst>        To fix this, move some records to a slower scan rate.
> <8/3/2021 8:35:09 hst>
> ...
>

--
Complexity comes for free, simplicity you have to work for.


References:
Beatnik RTEMS4.10.2 vmeTsi148 ISR: ERROR: no handler registered Matt Rippa via Tech-talk
Re: Beatnik RTEMS4.10.2 vmeTsi148 ISR: ERROR: no handler registered Matt Rippa via Tech-talk
Re: Beatnik RTEMS4.10.2 vmeTsi148 ISR: ERROR: no handler registered Johnson, Andrew N. via Tech-talk

Navigate by Date:
Prev: Re: New to EPICS - v4 questions Mark Rivers via Tech-talk
Next: RE: Accessing Record fields from asyn driver Mark Rivers via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  <20212022  2023  2024 
Navigate by Thread:
Prev: Re: Beatnik RTEMS4.10.2 vmeTsi148 ISR: ERROR: no handler registered Johnson, Andrew N. via Tech-talk
Next: SEQ Record Question Manoussakis, Adamandios via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  <20212022  2023  2024 
ANJ, 12 Mar 2021 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·