RE: Problems with soft motors

[Mark Rivers <rivers@cars.uchicago.edu>]

Hi Tim,

 

Ø  Do you see the error message "cannot find dbCaLink task." in the IOC console output after iocInit?

 

No, but there are definitely problems in devSoftAux.cc.

 

I have found that there were 2 problems. 

 

1)      devSoftAux::soft_motor_task has calls to the CA library like this:

            SEVCHK(ca_search(mr->dinp.value.pv_link.pvname, &dinp),

                   "ca_search() failure");

 

This means that if there is a CA error then it kills the task and no other needed CA callbacks get established.  In one of my IOCs where no soft motor callbacks were happening at all, this was the issue.  The soft_motor_task did not exist, whereas it did exist in all the other IOCs.  I tracked the problem down to one soft motor in the system that does not have string defined for the DINP link.  In previous versions of base this worked OK.  However, in base 7 it caused the task to exit with the following error:

 

 

CA.Client.Exception...............................................

    Error: "Invalid string"

    Context: "ca_search() failure"

    Source File: ../devSoftAux.cc line 172

    Current Time: TUE SEP 04 2018 13:39:36.049462058

 

This is the code that did it:

 

        if (mr->dinp.value.constantStr == NULL)

            ptr->default_done_behavior = true;

        else

        {

            ptr->default_done_behavior = false;

            SEVCHK(ca_search(mr->dinp.value.pv_link.pvname, &dinp),

                   "ca_search() failure");

 

The problem is that since when the link is not defined dinp.value.pv_link.pvname is NULL, and ca_search returns an error.  I don’t know why this worked in 3.14 and 3.15, but it did.  I was able to fix this problem by not calling the CA functions if dinp.value.pv_link.pvname  is NULL.

 

        if (mr->dinp.value.constantStr == NULL)

            ptr->default_done_behavior = true;

        else if (mr->dinp.value.pv_link.pvname != NULL)

        {

            ptr->default_done_behavior = false;

printf("devSoftAux::soft_motor_task: adding dinp link for motor %s link=%s\n", mr->name, mr->dinp.value.pv_link.pvname);

            SEVCHK(ca_search(mr->dinp.value.pv_link.pvname, &dinp),

                   "ca_search() failure");

 

I also temporarily added the debugging printf for each type of link that gets callbacks established in this function.  That let me see what the second problem is.

 

2)      The second problem is exposed with the printf statements added when the callbacks are established.  This is what I see during iocInit:

devSoftAux::soft_motor_task: adding dinp link for motor 13BMA:pm1 link=13BMA:KBVB:Done

devSoftAux::soft_motor_task: adding urip link for motor 13BMA:pm1 link=13BMA:KBVB:Readback.G

devSoftAux::soft_motor_task: adding urip link for motor 13BMA:pm2 link=13BMA:KBVB:Readback.H

devSoftAux::soft_motor_task: adding dinp link for motor 13BMA:pm3 link=13BMA:KBVP:Done

devSoftAux::soft_motor_task: adding urip link for motor 13BMA:pm3 link=13BMA:KBVP:Readback.G

devSoftAux::soft_motor_task: adding urip link for motor 13BMA:pm4 link=13BMA:KBVP:Readback.H

devSoftAux::soft_motor_task: adding dinp link for motor 13BMA:pm5 link=13BMA:KBVT:Done

devSoftAux::soft_motor_task: adding urip link for motor 13BMA:pm5 link=13BMA:KBVT:Readback.G

devSoftAux::soft_motor_task: adding urip link for motor 13BMA:pm6 link=13BMA:KBVT:Readback.H

 

The above output shows the problem.  For each soft motor it should be adding callbacks for both the dinp (done moving) and urip (position) links.  It is correctly adding the urip link for each motor.  However, it is only adding the dinp links for every other motor (pm1, pm3, pm5).  It is not adding the dinp link for (pm2,pm4,pm6).

 

I don’t understand this, since pm1 and pm2 seem to be defined identically for their DINP links.

 

ioc13bma> dbpr "13BMA:pm1",2

ACCL: 0.1           ACKS: NO_ALARM      ACKT: YES           ADEL: 0

ASG:                ATHM: 0             BACC: 0.1           BDST: 0

BKPT: 00            BVEL: 1             CDIR: 0             CNEN: Disable

DCOF: 0             DESC: Curvature     DHLM: 10000         DIFF: 0

DINP:DB_LINK 13BMA:KBVB:Done NPP NMS    DIR: Pos            DISA: 0

DISP: 0             DISS: NO_ALARM      DISV: 1             DLLM: -10000

DLY: 0              DMOV: 1             DOL:CONSTANT        DRBV: 0.25

DTYP: Soft Channel  DVAL: 0.25          EGU: mm             ERES: 1.0e-04

EVNT:               FLNK:CONSTANT       FOF: 0              FOFF: Frozen

FRAC: 1             HHSV: NO_ALARM      HIGH: 0             HIHI: 0

HLM: 10000          HLS: 0              HLSV: NO_ALARM      HOMF: 0

HOMR: 0             HOPR: 0             HSV: NO_ALARM       HVEL: 0.1

ICOF: 0             IGSET: 0            INIT:               JAR: 10

JOGF: 0             JOGR: 0             JVEL: 1             LCNT: 0

LDVL: 0.25          LLM: -10000         LLS: 0              LLSV: NO_ALARM

LOCK: NO            LOLO: 0             LOPR: 0             LOW: 0

LRLV: 0             LRVL: 2500          LSPG: Go            LSV: NO_ALARM

LVAL: 0.25          LVIO: 0             MDEL: 0             MISS: 0

MOVN: 0             MRES: 1.0e-04       NAME: 13BMA:pm1     NSEV: NO_ALARM

NSTA: NO_ALARM      NTM: NO             NTMF: 2             OFF: 0

OMSL: supervisory   OUT:DB_LINK 13BMA:pm1DVAL.VAL PP MS     PACT: 0

PCOF: 0             PHAS: 0             PINI: NO            POST:

PP: 0               PREC: 3             PREM:               PRIO: LOW

PUTF: 0             RBV: 0.25           RCNT: 0             RDBD: 1.0e-04

RDBL:DB_LINK 13BMA:KBVB:Readback.G NPP NMS                  RDIF: 0

REP: 0              RHLS: 0             RINP:CONSTANT       RLLS: 0

RLNK:CONSTANT       RLV: 0              RMOD: Default       RMP: 2500

RPRO: 0             RRBV: 2500          RRES: 1             RTRY: 0

RVAL: 2500          RVEL: 0             S: 50               SBAK: 50

SBAS: 5             SCAN: Passive       SDIS:CONSTANT       SET: Use

SEVR: NO_ALARM      SMAX: 50            SPMG: Go            SREV: 200

SSET: 0             STAT: NO_ALARM      STOO:DB_LINK 13BMA:KBVB:Stop PP MS

STOP: 0             SUSE: 0             SYNC: 0             TDIR: 0

TIME: 2018-09-07 17:04:51.807438608     TPRO: 0             TSE: 0

TSEL:CONSTANT       TWF: 0              TWR: 0              TWV: 0.2

UDF: 0              UDFS: INVALID       UEIP: No            UREV: 0.02

URIP: Yes           VAL: 0.25           VBAS: 0.1           VELO: 1

VERS: 6.1           VMAX: 1             VOF: 0

value = 0 = 0x0

 

ioc13bma> dbpr "13BMA:pm2",2

ACCL: 0.1           ACKS: NO_ALARM      ACKT: YES           ADEL: 0

ASG:                ATHM: 0             BACC: 0.1           BDST: 0

BKPT: 00            BVEL: 1             CDIR: 0             CNEN: Disable

DCOF: 0             DESC: Ellipticity   DHLM: 10000         DIFF: 0

DINP:DB_LINK 13BMA:KBVB:Done NPP NMS    DIR: Pos            DISA: 0

DISP: 0             DISS: NO_ALARM      DISV: 1             DLLM: -10000

DLY: 0              DMOV: 1             DOL:CONSTANT        DRBV: -0.0999

DTYP: Soft Channel  DVAL: -0.0999       EGU: mm             ERES: 1.0e-04

EVNT:               FLNK:CONSTANT       FOF: 0              FOFF: Frozen

FRAC: 1             HHSV: NO_ALARM      HIGH: 0             HIHI: 0

HLM: 10000          HLS: 0              HLSV: NO_ALARM      HOMF: 0

HOMR: 0             HOPR: 0             HSV: NO_ALARM       HVEL: 0.1

ICOF: 0             IGSET: 0            INIT:               JAR: 10

JOGF: 0             JOGR: 0             JVEL: 1             LCNT: 0

LDVL: -0.0999       LLM: -10000         LLS: 0              LLSV: NO_ALARM

LOCK: NO            LOLO: 0             LOPR: 0             LOW: 0

LRLV: 0             LRVL: -999          LSPG: Go            LSV: NO_ALARM

LVAL: -0.0999       LVIO: 0             MDEL: 0             MISS: 0

MOVN: 0             MRES: 1.0e-04       NAME: 13BMA:pm2     NSEV: NO_ALARM

NSTA: NO_ALARM      NTM: NO             NTMF: 2             OFF: 0

OMSL: supervisory   OUT:DB_LINK 13BMA:pm2DVAL.VAL PP MS     PACT: 0

PCOF: 0             PHAS: 0             PINI: NO            POST:

PP: 0               PREC: 3             PREM:               PRIO: LOW

PUTF: 0             RBV: -0.0999        RCNT: 0             RDBD: 1.0e-04

RDBL:DB_LINK 13BMA:KBVB:Readback.H NPP NMS                  RDIF: 0

REP: 0              RHLS: 0             RINP:CONSTANT       RLLS: 0

RLNK:CONSTANT       RLV: 0              RMOD: Default       RMP: -999

RPRO: 0             RRBV: -999          RRES: 1             RTRY: 0

RVAL: -999          RVEL: 0             S: 50               SBAK: 50

SBAS: 5             SCAN: Passive       SDIS:CONSTANT       SET: Use

SEVR: NO_ALARM      SMAX: 50            SPMG: Go            SREV: 200

SSET: 0             STAT: NO_ALARM      STOO:DB_LINK 13BMA:KBVB:Stop PP MS

STOP: 0             SUSE: 0             SYNC: 0             TDIR: 0

TIME: 2018-09-07 17:04:51.807438608     TPRO: 0             TSE: 0

TSEL:CONSTANT       TWF: 0              TWR: 0              TWV: 0.1

UDF: 0              UDFS: INVALID       UEIP: No            UREV: 0.02

URIP: Yes           VAL: -0.0999        VBAS: 0.1           VELO: 1

VERS: 6.1           VMAX: 1             VOF: 0

value = 0 = 0x0

 

So one problem is fixed, and I know what the other problem is, but not yet how to fix it.

 

Mark

 

 

From: Mooney, Tim M. <mooney@aps.anl.gov>
Sent: Friday, September 7, 2018 12:15 PM
To: Mark Rivers <rivers@cars.uchicago.edu>; Peterson, Kevin M. <kmpeters@anl.gov>; Johnson, Andrew N. <anj@anl.gov>
Cc: tech-talk@aps.anl.gov
Subject: Re: Problems with soft motors

 

Hi Mark,

 

I'll make a wild guess:  The function, soft_init(), in motor/motorApp/SoftMotorSrc/devSoftAux.cc, seems vulnerable to changes in EPICS base that might not have had consequences for other code I know of.  This code has always seemed a little hinky to me.  Do you see the error message "cannot find dbCaLink task." in the IOC console output after iocInit?

 

Tim Mooney (mooney@aps.anl.gov) (630)252-5417
Beamline Controls Group (www.aps.anl.gov)
Advanced Photon Source, Argonne National Lab

---

From: Mark Rivers <rivers@cars.uchicago.edu>
Sent: Thursday, September 6, 2018 6:23:32 PM
To: Peterson, Kevin M.; Mooney, Tim M.; Johnson, Andrew N.
Cc: tech-talk@aps.anl.gov
Subject: Problems with soft motors

 

Hi,


I am having a serious problem with soft motor device support.  These are databases that have worked fine for over 10 years.  The databases are the ones in motorApp/Db: pseudoMotor.db and sumDiff2D.db.


The databases are very simple: there are 2 real motors and 2 soft motors.  One soft motor moves both real motors in the same direction, and the other soft motor moves the real motors in the opposite direction.  The readback of the first soft motor is the average of the real motors, while the other is the difference of the real motors.  The calculations are done in transform records.  There is also a transform record that calculates when the soft motors are done moving, which is just the logical AND of the real motor .DMOV fields.


The main problem I am seeing is that the transform records are updating OK, but for some motors these values are not getting back into the motor record.


Here is an example of the debugging output of the devSoft.cc when a real motor is moved, and the soft motors, pm5 and pm6 are updating during that real motor move.


soft_dinp_func(): HARDMOVE set for 13BMD:pm5.
soft_rdbl_func(): updated RMP = 901 for 13BMD:pm5.
update(): dmov=0 for 13BMD:pm5.
update(): DMOV=0 for 13BMD:pm5.
soft_rdbl_func(): updated RMP = 23 for 13BMD:pm6.
update(): dmov=1 for 13BMD:pm6.
update(): DMOV=1 for 13BMD:pm6.
soft_rdbl_func(): updated RMP = 930 for 13BMD:pm5.
update(): dmov=0 for 13BMD:pm5.
update(): DMOV=0 for 13BMD:pm5.
soft_rdbl_func(): updated RMP = 80 for 13BMD:pm6.
update(): dmov=1 for 13BMD:pm6.
update(): DMOV=1 for 13BMD:pm6.
soft_dinp_func(): Done moving set for 13BMD:pm5.
update(): dmov=0 for 13BMD:pm5.
update(): DMOV=1 for 13BMD:pm5.
soft_rdbl_func(): updated RMP = 940 for 13BMD:pm5.
update(): dmov=1 for 13BMD:pm5.
update(): DMOV=1 for 13BMD:pm5.
soft_rdbl_func(): updated RMP = 101 for 13BMD:pm6.
update(): dmov=1 for 13BMD:pm6.
update(): DMOV=1 for 13BMD:pm6.

In this case pm5 is working fine.  It detected that dmov initially went to 0, and so at the end of the move it set the VAL field to match the RBV field.

However, pm6 is not working correctly.  It is detecting the readback changes, but it does not get the correct value of dmov.  It always sees it as 1, not as 0 which it should.  This means that at the end of the move it does not set the VAL field to match RBV.  pm5 and pm6 are configured identically, and they both get their DMOV value from the same field of the same transform record.  Why does one work and one not work?

Here are some symptoms of the problem:

- Some soft motors work fine, and others do not, even when they are apparently configured identically.  For one set of soft motors in one of my IOCs there are no updates printed with debugging in devSoft.cc.  For these soft motors neither the RBV fields nor the VAL fields update at all when the real motors are moved.

- The behavior of a particular motor is always the same, through multiple reboots, i.e. it either works or does not work.

- This has all worked fine until this shutdown when I made a lot of changes:
    Base 3.15.5 > 7.0.1.1
    synApps modules (motor, calc, etc.): master January 2018 to master August 2018.

- The devSoft.cc has not changed since 2015 so that cannot be the problem.

- I tried reverting the motorRecord.cc to the commit from Jan. 4, 2018 which is what I was using in the last run when it worked fine.  That did NOT fix the problem, which was very surprising.  This suggests that perhaps the problem has something to do with links in the transform record, or some change in base?

- I reverted the entire IOC application to the version I was running in the last run (3.15.5, master branch of synApps from January 2018) and all of the soft motors work fine.  This seems to prove it is some software change in base 7.0.1 vs 3.15.5, or a change in synApps since January.  But it is NOT the motor record itself.

Help!

Thanks,
Mark