Problems with vxWorks IOCs on base 3.14.12.6

[Mark Rivers <rivers@cars.uchicago.edu>]

Folks,

We are having problems with all of our vxWorks IOCs.  We recently upgraded from base 3.14.12.5 to 3.14.12.6.  We have not changed the version of vxWorks, which is 6.9.4.1.  We did update the Linux server on which we build and boot the IOCs from Fedora 15 to Centos 7.

The IOCs work fine as CA servers, all Linux and Windows clients can connect to them fine.  But the IOCs are not working as CA clients.   All records with CA links to other IOCs fail to connect, they have STAT=LINK.    SNL programs can connect to local PVs but not to PVs on other IOCs.  This is true whether the IOC hosting the target PV is running on Linux or VxWorks.

Here is an example where the DOL field points to a valid PV in another VxWorks IOC.  When I test the record it has STAT=LINK.

ioc13idc> dbtr "13IDC:filter:EnergyBeamline",10
ACKS: INVALID       ACKT: YES           ADEL: 0
ALST: 74.9806201550387                  AOFF: 0             ASG:
ASLO: 0             BKPT: 00            DESC: 13-ID-C Filters beamline energy
DISA: 0             DISP: 0             DISS: NO_ALARM      DISV: 1
DOL:CA_LINK 13BMC:m1.RBV NPP NMS        DRVH: 0             DRVL: 0
DTYP: Soft Channel  EGU:                EGUF: 0             EGUL: 0
EOFF: 0             ESLO: 1             EVNT: 0
FLNK:DB_LINK 13IDC:filter:Energy        HHSV: NO_ALARM      HIGH: 0
HIHI: 0             HOPR: 0             HSV: NO_ALARM       HYST: 0
INIT: 0             IVOA: Continue normally                 IVOV: 0
LALM: 74.9806201550387                  LBRK: 0             LCNT: 0
LINR: NO CONVERSION LLSV: NO_ALARM      LOLO: 0             LOPR: 0
LOW: 0              LSV: NO_ALARM       MDEL: 0
MLST: 74.9806201550387                  NAME: 13IDC:filter:EnergyBeamline
NSEV: NO_ALARM      NSTA: NO_ALARM      OIF: Full           OMOD: 0
OMSL: closed_loop   ORAW: 75            ORBV: 0             OROC: 0
OUT:CONSTANT        OVAL: 74.9806201550387                  PACT: 0
PHAS: 0             PINI: NO            PREC: 4             PRIO: LOW
PROC: 1             PUTF: 0             PVAL: 74.9806201550387
RBV: 0              ROFF: 0             RPRO: 0             RVAL: 75
SCAN: Passive       SDIS:CONSTANT       SEVR: INVALID       SIML:CONSTANT
SIMM: NO            SIMS: NO_ALARM      SIOL:CONSTANT       STAT: LINK
TIME: 2017-01-11 18:11:02.007588962     TPRO: 0             TSE: 0
TSEL:CONSTANT       UDF: 0              VAL: 74.9806201550387
value = 0 = 0x0

However, that 13BMC:m1.RBV PV works fine from Linux caget:

corvette:~/support/CARS/iocBoot>caget 13BMC:m1.RBV
13BMC:m1.RBV                   -0.856875

The record above works fine with CP links as long as the PVs are local to that IOC.

This is on a different IOC where the SNL program connects fine to local PVs, but fails to connect to the 2 PVs that are in another IOC.

epics> seqShow BM13_Energy -
State Program: "BM13_Energy"
  thread priority = 50
  number of state sets = 1
  number of syncQ queues = 0
  number of channels = 26
  number of channels assigned = 26
  number of channels connected = 24
  number of channels monitored = 24
  options: async=0, debug=0, newef=1, reent=0, conn=1

  State Set: "Energy"
  thread name = BM13_Energy;  Thread id = 0x211e110
  First state = "init"
  Current state = "init"
  Previous state = "init"
  Elapsed time since state was entered = -Inf seconds
  Wake up delay = Inf seconds
  Get in progress = [00000000000000000000000000]
  Put in progress = [00000000000000000000000000]


epics> seqChanShow BM13_Energy -
State Program: "BM13_Energy"
Number of channels=26
View: State set Energy

#14 of 26:
  Variable name: "xt_ht"
    type = double
    count = 1
  Value = 0
  Assigned to "13BMD:m22.VAL"
  Not connected
  Monitored
  Not sync'ed
  Status = -2
  Severity = -1
  Message = disconnected
  Time stamp = <undefined>
Next? (+/- skip count)


This is the output of epicsPrtEnvParams:

ioc13idc> epicsPrtEnvParams
EPICS_AR_PORT: 7002
EPICS_CAS_AUTO_BEACON_ADDR_LIST is undefined
EPICS_CAS_BEACON_ADDR_LIST is undefined
EPICS_CAS_BEACON_PERIOD is undefined
EPICS_CAS_BEACON_PORT is undefined
EPICS_CAS_IGNORE_ADDR_LIST is undefined
EPICS_CAS_INTF_ADDR_LIST is undefined
EPICS_CAS_SERVER_PORT is undefined
EPICS_CA_ADDR_LIST is undefined
EPICS_CA_AUTO_ADDR_LIST: YES
EPICS_CA_BEACON_PERIOD: 15.0
EPICS_CA_CONN_TMO: 30.0
EPICS_CA_MAX_ARRAY_BYTES: 88000
EPICS_CA_MAX_SEARCH_PERIOD: 300.0
EPICS_CA_NAME_SERVERS is undefined
EPICS_CA_REPEATER_PORT: 5065
EPICS_CA_SERVER_PORT: 5064
EPICS_CMD_PROTO_PORT is undefined
EPICS_IOC_LOG_FILE_COMMAND is undefined
EPICS_IOC_LOG_FILE_LIMIT: 1000000
EPICS_IOC_LOG_FILE_NAME is undefined
EPICS_IOC_LOG_INET: 164.54.160.182
EPICS_IOC_LOG_PORT: 7004
EPICS_TIMEZONE: CUS::360:031302:110602
EPICS_TS_NTP_INET is undefined
IOCSH_HISTSIZE: 50
IOCSH_PS1: epics>
value = 0 = 0x0

This is the output of the "version" and "coreRelease" commands on vxWorks.

ioc13bma> version
VxWorks (for Motorola MVME2700 - MPC 750) version 6.9.4.1.
Kernel: WIND version 2.13.
Made on Jul 27 2015 17:59:12.
Boot line:
dc(0,0)corvette:/usr/local/vw/VX6941/mv2700-dev6-debug e=164.54.160.74:ffffff00 h=164.54.160.82 u=iocboot tn=ioc13bma s=/home/epics/support/CARS/iocBoot/ioc13bma/st.cmd
value = 180 = 0xb4

ioc13bma> coreRelease
############################################################################
## EPICS R3.14.12.6
## EPICS Base built Jan  9 2017
############################################################################
value = 0 = 0x0
ioc13bma>


Any ideas on what the problem could be and any diagnostics to run to track it down?

Thanks,
Mark