EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  <20142015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  <20142015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: RE: MAXv model-3 - error reading axis position after 5 attempts
From: "D'Ewart, J. Mitch" <[email protected]>
To: Jens Eden <[email protected]>
Cc: "[email protected]" <[email protected]>
Date: Tue, 11 Mar 2014 09:42:05 -0700
Hi Jens,

Thanks for looking at this.  

 > Currently timeout is set to: timeout = epicsThreadSleepQuantum() + 0.001; 
 > 
 > Here this sums up to 21 msecs (epicsThreadSleepQuantum() = 20 msecs) and I
 > never got a timeout here.

In our case (mvme 6100 and RTEMS) epicsThreadSleepQuantum() = 0.001, so we were only waiting 2 ms before timeout.  I have increased it to 21ms.

 > Could you send me your modifications?
Here is the patch - it does a wait after clearing the message semaphore and then rechecks if any more messages have been sent.  This also has the hardcoded timeout of 0.021s.


--- motorApp/OmsAsynSrc/omsMAXv.cpp     2014-01-20 04:58:11.000000002 -0800
+++ motorApp/OmsAsynSrc/omsMAXv.cpp     2014-03-11 08:24:22.000000002 -0700
@@ -296,7 +296,8 @@
     }

     /* see if junk at input port - should not be any data available */
-    if ((epicsUInt16) pmotor->inGetIndex != (epicsUInt16) pmotor->inPutIndex)
+    int flushTime = 0;
+    while (((epicsUInt16) pmotor->inGetIndex != (epicsUInt16) pmotor->inPutIndex) && (flushTime < 5))
     {
         // flush cards response Buffer
 #ifdef DEBUG
@@ -311,7 +312,8 @@
         flag1.All = pmotor->status1_flag.All;
         pmotor->status1_flag.All = flag1.All;
         pmotor->msg_semaphore=0;
-
+        epicsThreadSleep(epicsThreadSleepQuantum());
+        flushTime++;
     }

     putIndex = (epicsUInt16) pmotor->outPutIndex;
@@ -383,7 +385,7 @@

     itera = 0;
     double time = 0.0;
-    double timeout = epicsThreadSleepQuantum() + 0.001;
+    double timeout = 0.021;
     while ((pmotor->status1_flag.Bits.text_response == 0) && (time < timeout)){
         Debug(32, "%s:%s:%s: Waiting for reponse, itera:%d\n",
                 driverName, functionName, portName, itera);




________________________________________
From: Jens Eden [[email protected]]
Sent: Monday, March 10, 2014 8:04 AM
To: D'Ewart, J. Mitch
Cc: [email protected]
Subject: Re: MAXv model-3 -  error reading axis position after 5 attempts

Mitch,

 > It looks like the timeout may not be long enough so our messages are getting out of sync.
Do you get timeout error messages, if setting "motorOMSBASEdebug = 1"?

 > 1. increase the timeout  time ...

Currently timeout is set to: timeout = epicsThreadSleepQuantum() + 0.001;

Here this sums up to 21 msecs (epicsThreadSleepQuantum() = 20 msecs) and I
never got a timeout here.

Since epicsThreadSleepQuantum may vary, it shouldn't be used to set the timeout.
Could you try to set timeout to 21 msecs (timeout = 0.021) and see if that
is sufficient?


2. multiple responses in sendOnly().
I did this quicj test to reproduce this:

  1. send "AY LT?", read response, send "WY" read response
caput OMSMv:io3401omsSendRecv "AY LT?" ; caput OMSMv:io3401omsSendRecv "WY"
omsMAXv::send: sending: AY LT?
omsMAXv::sendReceive: received =l
omsMAXv::send: sending: WY
omsMAXv::sendReceive: received MAXv-8000 ver:1.45 ...

  2. send "AY LT?", don't read response, send "WY" read response
caput OMSMv:io3401omsSend "AY LT?" ; caput OMSMv:io3401omsSendRecv "WY"
omsMAXv::send: sending: AY LT?
omsMAXv::send: sending: WY
omsMAXvAsyn:sendOnly:MAXv1: flushing 3 characters        <= flush unread response
omsMAXv::sendReceive: received MAXv-8000 ver:1.45 ...

  3. send "AY LT? AX LT?", don't read 2 responses, send "WY" read response
  caput OMSMv:io3401omsSend "AY LT? AX LT?" ; caput OMSMv:io3401omsSendRecv "WY"
omsMAXv::send: sending: AY LT? AX LT?
omsMAXv::send: sending: WY
omsMAXvAsyn:sendOnly:MAXv1: flushing 3 characters     <= flush first unread response
omsMAXv::sendReceive: received =l                     <= 2nd unread response, should be firmware string

  4. send more than 2 requests: MAXv remains unresponsive until power cycle:
  caput OMSMv:io3401omsSend "AY LT?;AX LT?;AZ LT?;" ; caput OMSMv:io3401omsSendRecv "WY"

 > 2) Mitch, changed the function sendOnly() to flush multiple responses.
Could you send me your modifications?

Thanks
Jens


Am 07.03.2014 18:57, schrieb D'Ewart, J. Mitch:
> The problem we are seeing
>
> omsBaseDriver:getAxesArray:MAXv-01: array string conversion error, count: 0, axes: 8, input: >MNNN,PNNN,MNNN,PNLN,PNLN,PNLN,PNLN,PNLN<^M
>
> This seems to be a request and response not matching up.  In this case the response to getAxesArray() should be numerical, for example
>
> "AM RP?"
> responds with
> "0,0,0,0,0,0,0"
> but is getting
> "MNNN,PNNN,MNNN,PNLN,PNLN,PNLN,PNLN,PNLN"
>
> In this case sendOnly() is called from sendReceive().  sendOnly() flushes the response queue
>
>      /* see if junk at input port - should not be any data available */
>      if ((epicsUInt16) pmotor->inGetIndex != (epicsUInt16) pmotor->inPutIndex)
>      {
>          // flush cards response Buffer
> #ifdef DEBUG
>          int deltaIndex = ((epicsUInt16)pmotor->inPutIndex) - ((epicsUInt16)pmotor->inGetIndex);
>          Debug(32, "%s:%s:%s: flushing %d characters\n",
>                  driverName, functionName, portName, (((deltaIndex < 0) ? BUFFER_SIZE +
>                          deltaIndex : deltaIndex)));
> #endif
>          putIndex = (epicsUInt16) pmotor->inPutIndex;
>          pmotor->inGetIndex = putIndex;
>          pmotor->status1_flag.Bits.text_response = 0;
>          flag1.All = pmotor->status1_flag.All;
>          pmotor->status1_flag.All = flag1.All;
>          pmotor->msg_semaphore=0;
>
>      }
>
>   and then sends the new request.  sendReceive() then waits for a response flag to be set or timeout.
>
>      Debug(64, "omsMAXv::sendReceive: receiving\n");
>      itera = 0;
>      double time = 0.0;
>      double timeout = epicsThreadSleepQuantum() + 0.001;
>      while ((pmotor->status1_flag.Bits.text_response == 0) && (time < timeout)){
>          Debug(32, "%s:%s:%s: Waiting for reponse, itera:%d\n",
>                  driverName, functionName, portName, itera);
>          //  busy-waiting but not more than 2000 times
>          if (itera > 2000){
>              time += epicsThreadSleepQuantum();
>              epicsThreadSleep(epicsThreadSleepQuantum());
>          }
>          itera++;
>      }
>
> It looks like the timeout may not be long enough so our messages are getting out of sync.  The two proposed solutions are to 1. increase the timeout  time and 2. flush multiple responses in sendOnly().
>
> Mitch D'Ewart
>
>
>
> ________________________________________
> From: [email protected] [[email protected]] On Behalf Of Mark Rivers [[email protected]]
> Sent: Thursday, March 06, 2014 3:33 PM
> To: Luchini, Kristi L.; [email protected]
> Subject: RE: MAXv model-3 -  error reading axis position after 5 attempts
>
> When sendOnly() is called does the MaxV actually send a response to that message?  If so then sendOnly should wait for it, even if it will discard the response.  If it does not then the next time a thread does a write/read it cannot reliably know if a response it got was from its query or a previous sendOnly response.
>
> Mark
>
>
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On Behalf Of Luchini, Kristi L.
> Sent: Thursday, March 06, 2014 5:29 PM
> To: [email protected]
> Subject: MAXv model-3 - error reading axis position after 5 attempts
>
> Hello,
>
> Has anyone seen this error on the new model-3 version of the MAXv driver (omsAsyn) ?
>
>     8544 omsBaseDriver:BasePoller:MAXv-01: error reading axis position after 5 attempts^M
>     8545 omsBaseDriver:BasePoller:MAXv-01: error reading axis position after 5 attempts^M
>     8546 omsBaseDriver:getAxesArray:MAXv-01: array string conversion error, count: 0, axes: 8, input: >MNNN,PNNN,MNNN,PNLN,PNLN,PNLN,PNLN,PNLN<^M
>     8547 2014/02/05 12:03:30.540 omsBaseDriver:readInt32:MAXv-01 wrong AUX state N^M
>     8548 2014/02/05 12:03:30.670omsBaseDriver:getAxesArray:MAXv-01: array string conversion error, count: 0, axes: 8, input: >MNNN,PNNN,MNNN,PNLN,PNLN,PNLN,PNLN,PNLN<^M
>
> We have seen this error in  a few motion control applications with MAXv controllers.  All of the IOCs were
>   running motor-R6-8 asyn MAXv driver.  The problem seems to occur when we're using a number of axis, and in
> some cases using virtual motors where multiple axis are moved at once.
>
> 1) MAXv  controller, with 2 steppers and 1 servo motors. This  MAXv has FW 1.45
> 2) MAXv controller , with 8 steppers. This MAXv has FW 1.44 (vitual motors moving 4-axis at once, and 2-axis at once using transform record)
> 3) MAXv controller, with 7 steppers.  This  MAXv has FW 1.31  (virtual motors, moves 4 axis at once, and 2-axis using transform record)
>
>
> Mitch D'Ewart found that  "The omsMAXv::sendOnly function checks the input buffer and flushes one response if there is anything there but we seem to have come on a case where there are multiple responses backed up.  These responses are controlled by a message semaphore.  It seems that once sendOnly flushes the input buffer and clears the semaphore then the MAXv sends another backed up message response.  Finally the receive part of the calling function gets the wrong message.
>
> There are two solutions that have been implemented recently at SLAC. Both will be booted on Monday 3/10/2014, in production.
> So we don't know if the problem is fixed yet.
> 1) Ziga added eieio in omsMAXv.cpp by including basicIoOps.h from Till Strauman.
>
> Example:
> static void writeReg8(volatile epicsUInt8 *a8, epicsUInt8 value)
> {
> #ifdef HAS_IOOPS_H
>      out_8(a8, value);
> #else
>      *a8 = value;
> #endif
> }
>
> 2) Mitch, changed the function sendOnly() to flush multiple responses.
>
> Details:
> Motor-R6-8
> Asyn-R4-12
> EPICS R3.14.12
> RTEMS-4.9.4
> MVME6100 CPU.
> MAXV-8000
>
> QUESTION:
> How shold asyn support implementation  respond in this sort of case, where the response messages get out of sync with the queries?
> Do you simply clear you the response buffer, buffer and start again, or is there another solution?  Has anyone else implemented a fix to this
> Particular MAxv but or to another asyn support driver?
>
> Regards,
>    Kristi
>
>
>
>
>



Replies:
RE: MAXv model-3 - error reading axis position after 5 attempts Jens . Eden
References:
MAXv model-3 - error reading axis position after 5 attempts Luchini, Kristi L.
RE: MAXv model-3 - error reading axis position after 5 attempts Mark Rivers
RE: MAXv model-3 - error reading axis position after 5 attempts D'Ewart, J. Mitch
Re: MAXv model-3 - error reading axis position after 5 attempts Jens Eden

Navigate by Date:
Prev: PCaPAC Abstract Deadline Elder Matias
Next: RE: MAXv model-3 driver - omsMAXvConfig problem D'Ewart, J. Mitch
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  <20142015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Re: MAXv model-3 - error reading axis position after 5 attempts Jens Eden
Next: RE: MAXv model-3 - error reading axis position after 5 attempts Jens . Eden
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  <20142015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 17 Dec 2015 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·