RE: XPS Q8 issue with indexHomeSearch sequence

[Russ Berg <Russ.Berg@lightsource.ca>]

HI Mark,

The version of the firmware I have is XPS-Q8 Firmware Standard V1.3.0 which dates to November of 2013, I will pass on your information to the tech support person I was talking to and see if they can give a straight answer regarding if and when it was fixed, when I find out I will repost back to tech talk

-Russ

 

 

 

From: Mark Rivers [mailto:rivers@cars.uchicago.edu]
Sent: March-01-16 10:01 AM
To: Russ Berg <Russ.Berg@lightsource.ca>; tech-talk@aps.anl.gov
Subject: RE: XPS Q8 issue with indexHomeSearch sequence

 

Hi Russ,

 

Ø  Recently I nearly destroyed a $600 in vac limit switch when the stage controlled by the Newport XPS Q8 failed to respond to a + limit switch it was already sitting on

Ø  and continued to move in the + direction.

Ø  I verified that the limit switch worked correctly and found that the homing procedure as it is defined in the manual (paste of relevant section below) worked as advertised but ONLY if the stage was not already sitting on the + limit switch. I hadn’t noticed this problem over the last year because in all the times I homed the stage I had never tried homing while on the + limit switch. It appears that the Q8 is waiting for the event of the limit switch being hit and does not get a current limit switch state or poll it during the homing sequence and as a result it will merrily continue moving + until a hard stop …or smoke.

 

Ø  I just received an email back from Newport acknowledging this issue and that they were able to recreate it. Tech support said that they will refer the issue to the “design team” to have it corrected.

 

This is a known problem which I reported to Newport in 2013.  I have appended the entire e-mail conversation I had with Mark White, who was then the Newport technical engineer based in California.  I also included Daniel Gvero who is the XPS lead product engineer in France.

 

You will see that Newport agreed to fix the problem on the XPS-Q8 on March 7, 2013.  They then claimed to have done so in a firmware release on April 23, 2013.

 

I don’t have any record showing whether or not I tested that the new firmware actually fixed the problem.

 

Do you know the date or version of your Q8 firmware?

 

 

**************************************************************************

This is my original message I sent to Mark White and Daniel Gvero on Feb. 12, 2013 (over 3 years ago):

 

Hi Mark,

 

I would like to report 2 serious problems with the XPS.  The first problem is happening with our XPS-Q8, I don't know if the same problem exists on the XPS-C8. The second problem occurs on both I am quite sure.

 

1) The GatheringConfigurationSet command fails (with no error message) if the TypeArray string containing the information to gather exceeds 500 characters.  This is a real problem.  One is allowed to specify up to 25 items to gather.  Each item has a syntax like the following:

GROUP4.ROLL.CurrentPosition,

That string is 28 characters long, and it is typical.  If I want to specify 25 such items my string will be 700 characters long.

 

The following demonstrates this problem. This was done in a telnet session to the XPS.  The following command worked (note that is all one very long line).  The XPS returned 0,,EndOfAPI, as it should.

 

GatheringConfigurationSet(GROUP1.THETA.SetpointPosition,GROUP1.THETA.CurrentPosition,GROUP1.HEIGHT.SetpointPosition,GROUP1.HEIGHT.CurrentPosition,GROUP3.PITCH.SetpointPosition,GROUP3.PITCH.CurrentPosition,GROUP4.ROLL.SetpointPosition,GROUP4.ROLL.CurrentPosition,GROUP5.HEIGHT.SetpointPosition,GROUP5.HEIGHT.CurrentPosition,GROUP6.P.SetpointPosition,GROUP6.P.CurrentPosition,GROUP7.P.SetpointPosition,GROUP7.P.CurrentPosition,GROUP8.P.SetpointPosition,GROUP8.P.CurrentPosition)
0,,EndOfAPI

This command failed because I added one more piece of gathering data that made the string exceed 500 characters.  The XPS did not return anything.  When I entered <Return> again then it returned -7,,EndofAPI. 

 

GatheringConfigurationSet (GROUP1.THETA.SetpointPosition,GROUP1.THETA.CurrentPosition,GROUP1.HEIGHT.SetpointPosition,GROUP1.HEIGHT.CurrentPosition,GROUP3.PITCH.SetpointPosition,GROUP3.PITCH.CurrentPosition,GROUP4.ROLL.SetpointPosition,GROUP4.ROLL.CurrentPosition,GROUP5.HEIGHT.SetpointPosition,GROUP5.HEIGHT.CurrentPosition,GROUP6.P.SetpointPosition,GROUP6.P.CurrentPosition,GROUP7.P.SetpointPosition,GROUP7.P.CurrentPosition,GROUP8.P.SetpointPosition,GROUP8.P.CurrentPosition,GROUP8.P.FollowingError)

-7,
,EndOfAPI

 

I was able to print the actual characters being sent to the XPS when my software was calling GatheringConfigurationGet and the entire string was sent correctly.

 

2) The following is a very serious problem, with safety implications.  It happens under the following circumstances:

   - An axis has "UseCurrentPositionAsHome"

  - The axis is at a hard limit when it is homed.

  - The axis is moved in the direction of the hard limit.

 

Clearly the XPS should not allow the motion into the hard limit direction.  BUT IT DOES!  This has led to serious equipment damage in several cases here.  The XPS appears to only stop a motion due to a limit when it sees a transition on that limit switch.  But clearly if a limit switch is activated the XPS should NEVER allow motion in that direction.  This is the first motor controller I have ever come across with this serious safety problem.

 

Thanks,

Mark

**************************************************************************

 From Mark White on Feb. 12, 2013:

 

Hi Mark,

 

Thanks for emailing about these two items.  I 'think' that I verified

the long string problem on the Cx version too.  Or, I should say, that

it didn't work and behaved the same as on the Qx version.

 

For the limit problem, I've reinforced the question to Daniel and, if he

doesn't email all of us, let you know if he emails me about it.  But, I

can confirm what you've observed.  At this point, my suggestion is to

check the status of the limit bits just after homing to the current

position.

 

Thanks,

Mark

**************************************************************************

 

From me on Feb. 28, 2013 to Mark White and Daniel Gvero:

 

Mark and Daniel,

 

It's been over 2 weeks and I have not heard from Daniel about the limit problem.

 

When we first purchased the XPS we complained about the fact that hitting a limit switch is treated so "seriously" by the XPS.  It disables all motors in the group, and all motors must be re-homed if a limit switch is hit.  There is no other controller that we have used that does this.  Newport said at the time that they did this because Newport controllers are used in factory automation and European Union safety regulations absolutely required it.

 

Now we find that under certain conditions the XPS completely ignores the limit switch, and happily allows motion in the direction of a stuck limit switch.   This is the most unsafe situation I have ever seen in a motor controller, no other controller I have ever seen will permit this.  It makes your safety arguments a bad joke.

 

If I don't receive a solution to this problem soon I will post a warning to the EPICS distribution list warning others of this serious safety issue with the XPS.

 

Mark

**************************************************************************

 

From Mark White on Feb. 28, 2013:

 

Hi Mark,

 

I was waiting to email you when the internal discussions were finished

and a plan of action was determined, but will send something now.  I

thought a change was coming a week ago, but the firmware group changed

course unexpectedly.  I've been fighting against the status-quo to get

this problem fixed, and believe that the decision will come tomorrow.

Believe it or not, but the reason that it's not considered a simple

change is that it technically changes the behavior of the controller and

there may be an installed user base expecting this behavior.

 

To summarize:  Please give me until Tuesday morning to confirm that a

correction to the behavior is made.  If I can't assure that the firmware

will be changed by that point, then I adamantly agree and support any

EPICS distribution about this problem.  You can even include that the

'Newport solution' is that when using Current Position home type, to

always check the EOR flags before moving.

 

 

Regarding the 500 character limit for API:  It is not planned for

change.  (I'm going to fight this battle after the EOR/hard limit one)

 

 

Wish I could do more to influence the change faster,

Mark

**************************************************************************

 

From me to Mark White on Feb. 28, 2013:

 

Thanks for the reply.  I'm fine with waiting longer.  I was not looking for a solution so soon, just an acknowledgment of the problem from France.

 

Is the 500 character limit documented?  If not then it is definitely a bug since what is documented is that 25 items can be specified, and the 500 character limit will be exceeded if the GROUP.POSITIONER names exceed some length, and they don't have a documented limit.

 

Checking EOR before moving is completely unacceptable.  This can happen to an inexperienced beamline user in the middle of the night and they cannot be expected to do this.  It is not my user's or my driver's responsibility to obey hardware limits, it is the controller's responsibility.

 

Mark

**************************************************************************

 

From Mark White on March 7, 2013:

 

Hi Mark,

Quick note to let you know that changes for the moves at the limits has been agreed to.  I’ll let you know more details, expected change timeframe, when I learn them.

 

Regards,
Mark

**************************************************************************

 

From Mark White on April 23, 2013:

 

Hi Mark,

An updated XPS-Qx firmware is now on the Newport FTP site.  It corrects

the limit signal problem.

ftp://download.newport.com/MotionControl/Current/MotionControllers/XPS-Q8/Updates/XPS-Q8%20Standard%20Installer%20Pack%2030011/

 

Thanks,

Mark

**************************************************************************

 

From me on April 23, 2013:

 

**************************************************************************

Hi Mark,

 

Is there also a version of the C8 firmware that fixes this problem?  I think the problem occurs on both the Q8 and C8?

 

Thanks,

Mark

**************************************************************************

 

From Mark White on April 24, 2013:

 

Hi Mark,

 

The C8 firmware does not have an update.  That version is not scheduled

for ongoing maintenance since it is considered obsolete.  However, I'm

doing what I can to have the Cx firmware updated as well.  How many XPS

do you know about at Argonne (and SLAC/elsewhere if you know) are used

in a manner to need this update?

 

One solution is upgrading the CPU in the XPS with the XPS-QKIT.  We can

discuss this option more when you like.

 

Thanks,

Mark

**************************************************************************

 

From me to Mark White on  April 24, 2013:

 

Hi Mark,

 

I think the C8 really needs the update, and I just talked to the users here at SLAC, and they agree.  I understand that the C8 is no longer being sold, and I don't expect new features.  But our beamline alone has $100,000 of the C8 controllers, and we plan to use them for many years to come.  Some will definitely be used with motors having "Use current position as home", and we cannot afford to crash past limit switches.  Just today here at SLAC a Newport stage had its encoder fail due to radiation damage, so they had to change to Use Current Position As Home.  They don't want to be able to move into a limit just because they were on the limit when they had to re-home.

 

Thanks,

Mark

 

**************************************************************************

 

From Mark White to me (and Matt Boyes at SLAC who also complained that XPS-C8 needed the same firmware fix) on  April 24, 2013:

 

Matt and Mark: Thank you for your emails.  Your feedback is critical in

driving resource allocation for updating the C8 firmware.  Any more

similar opinions can only help everyone understand your position.

 

Highest regards,

Mark W.

 

**************************************************************************

From me to Mark White on April 30, 2013:

 

We had an XPS-C8 fail today, and found that it is an older revision that Newport does not support upgrading with the XPS-QKIT.  Assuming we can fix the CPU card ourselves (Newport no longer supplies XPS-C8 CPU cards) we will be in the situation where we have an unsafe controller, one which merrily crashes through limit switches.

 

This makes it even more imperative that Newport provide upgraded firmware for the C8.

 

Mark

 

 

From: tech-talk-bounces@aps.anl.gov [mailto:tech-talk-bounces@aps.anl.gov] On Behalf Of Russ Berg
Sent: Tuesday, March 01, 2016 8:55 AM
To: tech-talk@aps.anl.gov
Subject: XPS Q8 issue with indexHomeSearch sequence

 

Hi All, a tech talk search for this issue resulted in 0 hits so I thought those of you with XPS Q8 controllers might not know.

 

Recently I nearly destroyed a $600 in vac limit switch when the stage controlled by the Newport XPS Q8 failed to respond to a + limit switch it was already sitting on and continued to move in the + direction.

I verified that the limit switch worked correctly and found that the homing procedure as it is defined in the manual (paste of relevant section below) worked as advertised but ONLY if the stage was not already sitting on the + limit switch. I hadn’t noticed this problem over the last year because in all the times I homed the stage I had never tried homing while on the + limit switch. It appears that the Q8 is waiting for the event of the limit switch being hit and does not get a current limit switch state or poll it during the homing sequence and as a result it will merrily continue moving + until a hard stop …or smoke.

 

I just received an email back from Newport acknowledging this issue and that they were able to recreate it. Tech support said that they will refer the issue to the “design team” to have it corrected.

 

The manual states the following:

  3. IndexHomeSearch is used with positioners that have a home index, but with no

hardware home switch signal. In this process, the positioner initially moves in the

positive direction to find the index. When a + limit switch is detected, the direction

of motion reverses until the index is found

 

a screenshot of the versions the Q8 is running:

 

 

~~~~~~~~~

Russ Berg                                     Software & Controls

e-mail: russ.berg@lightsource.ca   Canadian Light Source

ph: 306 657 3572                         University of Saskatchewan

fax: 306 657 3535                        Saskatoon SK S7N 0X4