EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  <20142015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  <20142015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Re: MAXv model-3 - error reading axis position after 5 attempts
From: Jens Eden <[email protected]>
To: "D'Ewart, J. Mitch" <[email protected]>
Cc: [email protected]
Date: Mon, 10 Mar 2014 16:04:04 +0100

Mitch,

> It looks like the timeout may not be long enough so our messages are getting out of sync.
Do you get timeout error messages, if setting "motorOMSBASEdebug = 1"?

> 1. increase the timeout  time ...

Currently timeout is set to: timeout = epicsThreadSleepQuantum() + 0.001;

Here this sums up to 21 msecs (epicsThreadSleepQuantum() = 20 msecs) and I
never got a timeout here.

Since epicsThreadSleepQuantum may vary, it shouldn't be used to set the timeout.
Could you try to set timeout to 21 msecs (timeout = 0.021) and see if that
is sufficient?


2. multiple responses in sendOnly().
I did this quicj test to reproduce this:

 1. send "AY LT?", read response, send "WY" read response
caput OMSMv:io3401omsSendRecv "AY LT?" ; caput OMSMv:io3401omsSendRecv "WY"
omsMAXv::send: sending: AY LT?
omsMAXv::sendReceive: received =l
omsMAXv::send: sending: WY
omsMAXv::sendReceive: received MAXv-8000 ver:1.45 ...

 2. send "AY LT?", don't read response, send "WY" read response
caput OMSMv:io3401omsSend "AY LT?" ; caput OMSMv:io3401omsSendRecv "WY"
omsMAXv::send: sending: AY LT?
omsMAXv::send: sending: WY
omsMAXvAsyn:sendOnly:MAXv1: flushing 3 characters        <= flush unread response
omsMAXv::sendReceive: received MAXv-8000 ver:1.45 ...

 3. send "AY LT? AX LT?", don't read 2 responses, send "WY" read response
 caput OMSMv:io3401omsSend "AY LT? AX LT?" ; caput OMSMv:io3401omsSendRecv "WY"
omsMAXv::send: sending: AY LT? AX LT?
omsMAXv::send: sending: WY
omsMAXvAsyn:sendOnly:MAXv1: flushing 3 characters     <= flush first unread response
omsMAXv::sendReceive: received =l                     <= 2nd unread response, should be firmware string

 4. send more than 2 requests: MAXv remains unresponsive until power cycle:
 caput OMSMv:io3401omsSend "AY LT?;AX LT?;AZ LT?;" ; caput OMSMv:io3401omsSendRecv "WY"

> 2) Mitch, changed the function sendOnly() to flush multiple responses.
Could you send me your modifications?

Thanks
Jens


Am 07.03.2014 18:57, schrieb D'Ewart, J. Mitch:
The problem we are seeing

omsBaseDriver:getAxesArray:MAXv-01: array string conversion error, count: 0, axes: 8, input: >MNNN,PNNN,MNNN,PNLN,PNLN,PNLN,PNLN,PNLN<^M

This seems to be a request and response not matching up.  In this case the response to getAxesArray() should be numerical, for example

"AM RP?"
responds with
"0,0,0,0,0,0,0"
but is getting
"MNNN,PNNN,MNNN,PNLN,PNLN,PNLN,PNLN,PNLN"

In this case sendOnly() is called from sendReceive().  sendOnly() flushes the response queue

     /* see if junk at input port - should not be any data available */
     if ((epicsUInt16) pmotor->inGetIndex != (epicsUInt16) pmotor->inPutIndex)
     {
         // flush cards response Buffer
#ifdef DEBUG
         int deltaIndex = ((epicsUInt16)pmotor->inPutIndex) - ((epicsUInt16)pmotor->inGetIndex);
         Debug(32, "%s:%s:%s: flushing %d characters\n",
                 driverName, functionName, portName, (((deltaIndex < 0) ? BUFFER_SIZE +
                         deltaIndex : deltaIndex)));
#endif
         putIndex = (epicsUInt16) pmotor->inPutIndex;
         pmotor->inGetIndex = putIndex;
         pmotor->status1_flag.Bits.text_response = 0;
         flag1.All = pmotor->status1_flag.All;
         pmotor->status1_flag.All = flag1.All;
         pmotor->msg_semaphore=0;

     }

  and then sends the new request.  sendReceive() then waits for a response flag to be set or timeout.

     Debug(64, "omsMAXv::sendReceive: receiving\n");
     itera = 0;
     double time = 0.0;
     double timeout = epicsThreadSleepQuantum() + 0.001;
     while ((pmotor->status1_flag.Bits.text_response == 0) && (time < timeout)){
         Debug(32, "%s:%s:%s: Waiting for reponse, itera:%d\n",
                 driverName, functionName, portName, itera);
         //  busy-waiting but not more than 2000 times
         if (itera > 2000){
             time += epicsThreadSleepQuantum();
             epicsThreadSleep(epicsThreadSleepQuantum());
         }
         itera++;
     }

It looks like the timeout may not be long enough so our messages are getting out of sync.  The two proposed solutions are to 1. increase the timeout  time and 2. flush multiple responses in sendOnly().

Mitch D'Ewart



________________________________________
From: [email protected] [[email protected]] On Behalf Of Mark Rivers [[email protected]]
Sent: Thursday, March 06, 2014 3:33 PM
To: Luchini, Kristi L.; [email protected]
Subject: RE: MAXv model-3 -  error reading axis position after 5 attempts

When sendOnly() is called does the MaxV actually send a response to that message?  If so then sendOnly should wait for it, even if it will discard the response.  If it does not then the next time a thread does a write/read it cannot reliably know if a response it got was from its query or a previous sendOnly response.

Mark


-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Luchini, Kristi L.
Sent: Thursday, March 06, 2014 5:29 PM
To: [email protected]
Subject: MAXv model-3 - error reading axis position after 5 attempts

Hello,

Has anyone seen this error on the new model-3 version of the MAXv driver (omsAsyn) ?

    8544 omsBaseDriver:BasePoller:MAXv-01: error reading axis position after 5 attempts^M
    8545 omsBaseDriver:BasePoller:MAXv-01: error reading axis position after 5 attempts^M
    8546 omsBaseDriver:getAxesArray:MAXv-01: array string conversion error, count: 0, axes: 8, input: >MNNN,PNNN,MNNN,PNLN,PNLN,PNLN,PNLN,PNLN<^M
    8547 2014/02/05 12:03:30.540 omsBaseDriver:readInt32:MAXv-01 wrong AUX state N^M
    8548 2014/02/05 12:03:30.670omsBaseDriver:getAxesArray:MAXv-01: array string conversion error, count: 0, axes: 8, input: >MNNN,PNNN,MNNN,PNLN,PNLN,PNLN,PNLN,PNLN<^M

We have seen this error in  a few motion control applications with MAXv controllers.  All of the IOCs were
  running motor-R6-8 asyn MAXv driver.  The problem seems to occur when we're using a number of axis, and in
some cases using virtual motors where multiple axis are moved at once.

1) MAXv  controller, with 2 steppers and 1 servo motors. This  MAXv has FW 1.45
2) MAXv controller , with 8 steppers. This MAXv has FW 1.44 (vitual motors moving 4-axis at once, and 2-axis at once using transform record)
3) MAXv controller, with 7 steppers.  This  MAXv has FW 1.31  (virtual motors, moves 4 axis at once, and 2-axis using transform record)


Mitch D'Ewart found that  "The omsMAXv::sendOnly function checks the input buffer and flushes one response if there is anything there but we seem to have come on a case where there are multiple responses backed up.  These responses are controlled by a message semaphore.  It seems that once sendOnly flushes the input buffer and clears the semaphore then the MAXv sends another backed up message response.  Finally the receive part of the calling function gets the wrong message.

There are two solutions that have been implemented recently at SLAC. Both will be booted on Monday 3/10/2014, in production.
So we don't know if the problem is fixed yet.
1) Ziga added eieio in omsMAXv.cpp by including basicIoOps.h from Till Strauman.

Example:
static void writeReg8(volatile epicsUInt8 *a8, epicsUInt8 value)
{
#ifdef HAS_IOOPS_H
     out_8(a8, value);
#else
     *a8 = value;
#endif
}

2) Mitch, changed the function sendOnly() to flush multiple responses.

Details:
Motor-R6-8
Asyn-R4-12
EPICS R3.14.12
RTEMS-4.9.4
MVME6100 CPU.
MAXV-8000

QUESTION:
How shold asyn support implementation  respond in this sort of case, where the response messages get out of sync with the queries?
Do you simply clear you the response buffer, buffer and start again, or is there another solution?  Has anyone else implemented a fix to this
Particular MAxv but or to another asyn support driver?

Regards,
   Kristi







Replies:
RE: MAXv model-3 - error reading axis position after 5 attempts D'Ewart, J. Mitch
References:
MAXv model-3 - error reading axis position after 5 attempts Luchini, Kristi L.
RE: MAXv model-3 - error reading axis position after 5 attempts Mark Rivers
RE: MAXv model-3 - error reading axis position after 5 attempts D'Ewart, J. Mitch

Navigate by Date:
Prev: SIS3820 firmware enhancement proposal Mark Rivers
Next: Using a Waveform record (char type representing a String) as input to an aSub record David Michel
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  <20142015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: RE: MAXv model-3 - error reading axis position after 5 attempts D'Ewart, J. Mitch
Next: RE: MAXv model-3 - error reading axis position after 5 attempts D'Ewart, J. Mitch
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  <20142015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 17 Dec 2015 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·