Experimental Physics and
Industrial Control System

"Davidsaver, Michael" <[email protected]> · Mon, 12 Jul 2010 10:54:28 -0400

All, 

Has anyone seen this behavior before in a MAXv (or elsewhere)?

We have been having some sporadic crashes with a VME crate with 5 MAXv
motor controllers in it (detail below).  The crate in its present
configuration has been in operation for ~10 months.  During that time we
have seen the following message occur randomly (~20 sec period) when an
axis is moving.

> <179>syslog:
> drvMAXv.cc:motorIsr: command error - card 03

This happens for all five cards.  All cards were recently returned to
OMS and updated to firmware 1.33, but the problem remains.

As it didn't seem to have any effect on the operation of the device we
ignored it, assuming that the crashes were the result of the known
watchdog timer problem (which we were also seeing).  However it now
seems that this is a symptom of a different, and fairly nasty problem.
During a recent attempt at debugging the problem we added a printk to
the interrupt handler to print the interrupt status register when a
command error is indicated.

>     if (status1_flag.Bits.cmndError)
>     {
> #ifdef __rtems__
>         printk("at cmd err Status %08x\n", status1_flag.All);
> #endif

Note that according to the MAXv documentation the top 5 bits of this
register are unused and should always be zeros.

What we now see is:

> <179>syslog:
> drvMAXv.cc:motorIsr: command error - card 01 at cmd err Status 
> 03000004
> <179>syslog:
> drvMAXv.cc:motorIsr: command error - card 00 at cmd err Status 
> 7FFF0010
> <179>syslog:
> drvMAXv.cc:motorIsr: command error - card 02 at cmd err Status 
> 3FDF0002
> <179>syslog:
> drvMAXv.cc:motorIsr: command error - card 03 at cmd err Status 
> 03810004
> <179>syslog:
> drvMAXv.cc:motorIsr: command error - card 00 at cmd err Status 
> 01810020
> <179>syslog:
> drvMAXv.cc:motorIsr: command error - card 04 at cmd err Status 
> 3FCF0010

So it looks like the top 16-bits are occasionally set when they should
be.  We haven't had a chance to investigate with a logic analyzer, but
it looks like a classic VME timing problem (top 2 bytes driven too
late).

The crate controller is a mvme3100 running RTEMS 4.9.3 with EPICS
3.14.10 and synapps 5.5 w/ the motor module updated to 6.5.1.  The crate
also contains a Joerger VSC16 scaler and a Precision Analog Systems DAC.
The scaler is 32-bit and has been showing no signs of this problem.

Michael Davidsaver
NSLSII Controls Group
Brookhaven National Lab
(631) 344-3698

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System