Re: Bus errors accessing VME with base 7.0.6.1 and latest synApps modules

[Michael Davidsaver via Tech-talk <tech-talk at aps.anl.gov>]

On 5/21/22 15:34, Mark Rivers wrote:
> ØOk, so all powerpc.  PPC Machine Check exception is asynchronous.
>
> ØSo I'm more confident in claiming the mention of "CAS-event" is false.
>
> ØThe faulting instruction probably originates on some other scan/driver thread, then there is a context switch to "CAS-event" because of a call to db_post_events().
>
> I’m not sure I understand the logic.  The other scan/driver threads are always running.  The IP330 is always interrupting at 2 kHz, and doing callbacks to device support.  It runs with no VME bus errors at all until I open an medm screen or run camonitor.  So it seems that the problem must be caused by having the CAS-event task do CA monitors, and it is not just that the CAS-event task is being incorrectly blamed for the problem?
Oh, what I say is far from a concrete explanation.  There is clearly
something more going on here.

I think it could only be true if the faulting operation were a "posted
write".  Meaning the VME bridge buffers the operation, letting the CPU
proceed before the VME bus cycle has actually completed.

This is why you may sometimes finds drivers with a "dummy" load after
an important store (eg. interrupt acknowledge).  Waiting for the load
instruction will also wait for the preceding store to complete.

eg. an admonishment by Till from the RTEMS universe2 bridge driver.

https://github.com/RTEMS/rtems/blob/a316a9ddaeaa8f6316b2a2d29ca82b3ad40d2d22/bsps/powerpc/shared/vme/vmeUniverse.c#L2187-L2189

"posted writes" are a configuration option for each address window.
Disabling this may give a more accurate address with the exception,
at the expense of some slow down.


Of course, none of this would explain why these particular addresses
are being accessed, nor why they fault.


On 5/21/22 16:45, Mark Rivers wrote:
> ØI will also try to make a thin vxWorks IOC application with basically just Industry Pack module support in case it is some strange interaction with another module.
>
> A thin IOC with only seq, asyn, iocStats, ipac,  ip330, dac128V, and ipUnidig does not fail with base 7.0.6.1.
>
> I will add things back in one at a time and see what is actually causing the problem.
>
> Luckily it is a gray and rainy day in Chicago. J
Some people have all the luck.  Today was terribly sunny here in CA :)

(I've been here 4.5 years, and I still can't get over the weather!)


> I will try 7.0.6.
>
> I will also try to make a thin vxWorks IOC application with basically just Industry Pack module support in case it is some strange interaction with another module.
>
> Mark
>
> -----Original Message-----
> From: Michael Davidsaver <mdavidsaver at gmail.com>
> Sent: Saturday, May 21, 2022 4:31 PM
> To: Mark Rivers <rivers at cars.uchicago.edu>
> Cc: tech-talk at aps.anl.gov
> Subject: Re: Bus errors accessing VME with base 7.0.6.1 and latest synApps modules
>
> On 5/21/22 11:05, Mark Rivers wrote:
>
>  > ØWhat specific board is involved?  (eg. mvme3100?)
>
>  >
>
>  > The test crate is an MVME5100.  But the production crates that were also failing include several MVME2700 boards as well as some MVME5100.
>
> Ok, so all powerpc.  PPC Machine Check exception is asynchronous.
>
> So I'm more confident in claiming the mention of "CAS-event"
>
> is false.  The faulting instruction probably originates on some other scan/driver thread, then there is a context switch to "CAS-event" because of a call to db_post_events().
>
> It still seems odd to me that the CPU could get all the way into db_post_events() and wake up "CAS-event" before a VME cycle completes.  (maybe there are VME timeout happening?)
>
> In addition to Base 7.0.5 and and 7.0.6.1, could you test with 7.0.6 ?
>
> This might narrow things down a little.
>
> Since you already have a version range to suspect, you could try to narrow down further with git-bisect.
>
> (although I can't honestly recommend this as a good way to pass what for me is a nice Saturday afternoon.)
>
> https://git-scm.com/docs/git-bisect <https://git-scm.com/docs/git-bisect>
>
>  > git bisect start R7.0.5 R7.0.6.1