Experimental Physics and
Industrial Control System

Hinko Kocevar via Tech-talk <tech-talk at aps.anl.gov> · Wed, 25 Mar 2020 11:32:57 +0000

> However, there is a way to do what you want.  You need to create 2 new driver parameters for those functions (alarmStatus and alarmSeverity). Then you create 2 new longout records that get their VAL fields from the .STAT and .SEVR fields of your record.  Those longout records can send the status and severity to the driver.

Right.

This works for me as desired. Note that I had to set DOL in EvrLinkSevr to EvrLinkSevrIn.SEVR, and not EvrLinkSevrIn.VAL. As soon as EVR PV went away the VAL in EvrLinkSevrIn did not change, but SEVR/STAT did.

# get EVR link state SEVR from EVR PV
record(longin, "$(P)$(R)EvrLinkSevrIn") {
    field(INP,  "LAB-EVR-000:Link-Sts.SEVR CP MS")
    field(FLNK, "$(P)$(R)EvrLinkSevr.PROC PP MS")
    info(autosaveFields, "INP")
}

record(longout, "$(P)$(R)EvrLinkSevr")
{
    field(DOL,  "$(P)$(R)EvrLinkSevrIn.SEVR MS")
    field(OMSL, "closed_loop")
    field(DTYP, "asynInt32")
    field(OUT,  "@asyn($(PORT),$(ADDR=0),$(TIMEOUT=1))SIS8300.EVR.LINK_SEVERITY")
}

record(longin, "$(P)$(R)EvrLinkSevrR")
{
    field(DTYP, "asynInt32")
    field(INP,  "@asyn($(PORT),$(ADDR=0),$(TIMEOUT=1))SIS8300.EVR.LINK_SEVERITY")
    field(SCAN, "I/O Intr")
}

Thank you,
//hinko

________________________________________
From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Tuesday, March 24, 2020 7:51 PM
To: Hinko Kocevar; tech-talk at aps.anl.gov
Subject: Re: detecting remote PV disconnect (asyn)

Hi Hinko,

 added the following to writeInt32() method of my DAQ IOC that extends asynNDArrayDriver for the parameter that supports BeamState record:

        asynStatus paramStatus;

        getParamStatus(function, &paramStatus);

        int alarmStatus;

        getParamAlarmStatus(function, &alarmStatus);

        int alarmSeverity;

        getParamAlarmSeverity(function, &alarmSeverity);

I always get all three values 0, no matter if the SEVR/STAT are indicating fault.

The methods in asynPortDriver are intended to send the status from the driver back to the client, e.g. EPICS device support and thus to the record.  They cannot be used to get the status from the record.

asynPortDrivers inherently know nothing about EPICS records.  They operate at a lower level, and can be used in the absence of an EPICS IOC entirely.  In order to get that information to the driver you need to use the standard asyn interfaces (asynInt32, etc.).

However, there is a way to do what you want.  You need to create 2 new driver parameters for those functions (alarmStatus and alarmSeverity). Then you create 2 new longout records that get their VAL fields from the .STAT and .SEVR fields of your record.  Those longout records can send the status and severity to the driver.

Mark

________________________________
From: Hinko Kocevar <Hinko.Kocevar at ess.eu>
Sent: Tuesday, March 24, 2020 8:10 AM
To: Mark Rivers; tech-talk at aps.anl.gov
Subject: Re: detecting remote PV disconnect (asyn)

Now that my asyn longout record, called BeamState, has STAT = LINK and SEVR = INVALID set, how can I get a hold of this in the code?

I added the following to writeInt32() method of my DAQ IOC that extends asynNDArrayDriver for the parameter that supports BeamState record:

        asynStatus paramStatus;
        getParamStatus(function, &paramStatus);
        int alarmStatus;
        getParamAlarmStatus(function, &alarmStatus);
        int alarmSeverity;
        getParamAlarmSeverity(function, &alarmSeverity);

I always get all three values 0, no matter if the SEVR/STAT are indicating fault.

DB report:

epics> dbpr LAB:AMC-008-5:BeamState 4
ACKS: INVALID       ACKT: YES           ADEL: 0             ALST: 10
ASG :               ASP : PTR (nil)     BKLNK: ELL 1 [0x28674d8 .. 0x28674d8]
BKPT: 00            DESC:               DISA: 0             DISP: 0
DISS: NO_ALARM      DISV: 1
DOL : DB_LINK LAB:AMC-008-5:BeamStateIn NPP MS              DPVT: PTR 0x4d33af0
DRVH: 0             DRVL: 0             DSET: PTR 0x10212e0 DTYP: asynInt32
EGU :               EVNT:               FLNK: CONSTANT      HHSV: NO_ALARM
HIGH: 0             HIHI: 0             HOPR: 0             HSV : NO_ALARM
HYST: 0             IVOA: Continue normally                 IVOV: 0
LALM: 10            LCNT: 0             LLSV: NO_ALARM      LOLO: 0
LOPR: 0             LOW : 0             LSET: PTR 0x3cf66a0 LSV : NO_ALARM
MDEL: 0             MLIS: ELL 1 [0x7fde9c02bce8 .. 0x7fde9c02bce8]
MLOK: d0 07 f3 03 00 00 00 00           MLST: 10
NAME: LAB:AMC-008-5:BeamState           NSEV: NO_ALARM      NSTA: NO_ALARM
OLDSIMM: NO         OMSL: closed_loop
OUT : INST_IO @asyn(AMC,0,1)SIS8300.EVR.BEAM_STATE          PACT: 0
PHAS: 0             PINI: NO            PPN : PTR (nil)     PPNR: PTR (nil)
PRIO: LOW           PROC: 0             PUTF: 0             RDES: PTR 0x2550ed0
RPRO: 0             RSET: PTR 0x1022f20 SCAN: Passive       SDIS: CONSTANT
SDLY: -1            SEVR: INVALID       SIML: CONSTANT      SIMM: NO
SIMPVT: PTR (nil)   SIMS: NO_ALARM      SIOL: CONSTANT      SPVT: PTR (nil)
SSCN: <nil>         STAT: LINK          TIME: 2020-03-24 13:56:48.405431439
TPRO: 0             TSE : 0             TSEL: CONSTANT      UDF : 0
UDFS: INVALID       VAL : 10

Here is the record:

record(longout, "$(P)$(R)BeamState")
{
    field(DOL,  "$(P)$(R)BeamStateIn MS")
    field(OMSL, "closed_loop")
    field(DTYP, "asynInt32")
    field(OUT,  "@asyn($(PORT),$(ADDR=0),$(TIMEOUT=1))SIS8300.EVR.BEAM_STATE")
}

//hinko

________________________________________
From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Hinko Kocevar via Tech-talk <tech-talk at aps.anl.gov>
Sent: Tuesday, March 24, 2020 1:42:26 PM
To: Mark Rivers
Cc: tech-talk
Subject: Re: detecting remote PV disconnect (asyn)

After tinkering a little bit more I got this to show link disconnected on camonitor:

record(stringin, "$(P)$(R)TimestampClk") {
    field(INP, "LAB-EVR-008:EvtECnt-I.VAL CP MS")
    field(FLNK, "$(P)$(R)TimestampIn.PROC PP MS")
    #field(SCAN, "10 second")
}

LAB:AMC-008-5:TimestampClk     2020-03-24 13:31:56.866964 131
LAB:AMC-008-5:TimestampIn      2020-03-24 13:31:56.866317 1585053116.866316538
LAB:AMC-008-5:Timestamp        2020-03-24 13:31:56.868907 1585053116.866316538
LAB:AMC-008-5:TimestampClk     2020-03-24 13:31:56.872925 131 LINK INVALID

The alarm is not propagated to the TimestampIn record, though. Is it the DTYP = "Soft Timestamp" in TimestampIn that interferes ?

At the moment that is OK with me. I had more success with longin records from EVR that I'm going to have feed the value into my DAQ anyway.
On those longin records, with DB constructed pretty much similar as with Timestamp records, I can see the alarm propagation all the way to the asyn record:

LAB:AMC-008-5:BeamStateIn      2020-03-24 13:31:56.678547 20
LAB:AMC-008-5:BeamStateR       2020-03-24 13:31:56.678653 20
LAB:AMC-008-5:BeamState        2020-03-24 13:31:56.683638 20
LAB:AMC-008-5:BeamStateIn      2020-03-24 13:31:56.872914 20 LINK INVALID
LAB:AMC-008-5:BeamState        2020-03-24 13:31:56.890227 20 LINK INVALID

Here BeamState is asyn based record. I'll just rely on one of these PV alarm status in my code to detect if EVR IOC is still there or not.
One thing that I had to add was MS to the DOL field of the BeamState that points to BeamStateIn; that made it propagate the alarm from BeamStateIn to BeamState.

Thank you!
//hinko
________________________________________
From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Tuesday, March 24, 2020 12:30:01 PM
To: Hinko Kocevar
Cc: tech-talk
Subject: Re: detecting remote PV disconnect (asyn)

Hi Hinko,

What happens if you change TimeStampClk to use normal soft device support and read into the VAL field, rather than TSEL?  Does the record go into alarm?  This is simpler and hopefully it will work.

record(stringin, "$(P)$(R)TimestampClk") {
    field(INP, "LAB-EVR-008:EvtECnt-I.TIME CP MS")
    field(FLNK, "$(P)$(R)TimestampIn.PROC PP MS")
    field(SCAN, "10 second")
}

Mark

________________________________
From: Hinko Kocevar <Hinko.Kocevar at ess.eu>
Sent: Tuesday, March 24, 2020 3:58 AM
To: Mark Rivers
Cc: tech-talk
Subject: Re: detecting remote PV disconnect (asyn)

Mark, I have this DB now (with the rest of PVs the same as in original post):

record(stringin, "$(P)$(R)TimestampClk") {
    field(DTYP, "Soft Timestamp")
    # use dummy placeholder PV, change EVR prefix to a desired EVR IOC
    field(TSEL, "LAB-EVR-008:EvtECnt-I.TIME CP MS")
    field(INP,  "@%s.%f")
    field(FLNK, "$(P)$(R)TimestampIn.PROC PP MS")

    field(SCAN, "10 second")
}

# get timestamp from EVR event PV
# i.e. set TSEL: LAB-EVR-004:EvtECnt-I.TIME CP MS
record(stringin, "$(P)$(R)TimestampIn") {
    field(DTYP, "Soft Timestamp")
    # use dummy placeholder PV, change EVR prefix to a desired EVR IOC
    field(TSEL, "LAB-EVR-008:EvtECnt-I.TIME MS")
    field(INP,  "@%s.%f")
    field(FLNK, "$(P)$(R)Timestamp.PROC PP MS")
    info(autosaveFields, "TSEL")
}

When I kill the EVR IOC I get these messages instantaneously:

epics> DB CA Link Exception: "Virtual circuit disconnect", context "bd-cpu08.cslab.esss.lu.se:43805"
recGblGetTimeStamp: dbGetTimeStamp failed for LAB:AMC-008-5:TimestampClk.TSEL
recGblGetTimeStamp: dbGetTimeStamp failed for LAB:AMC-008-5:TimestampClk.TSEL
recGblGetTimeStamp: dbGetTimeStamp failed for LAB:AMC-008-5:TimestampIn.TSEL
recGblGetTimeStamp: dbGetTimeStamp failed for LAB:AMC-008-5:TimestampIn.TSEL

And they keep repeating on 10 second interval, all 4 of them.

epics> dbpr LAB:AMC-008-5:TimestampClk 4
ACKS: NO_ALARM      ACKT: YES           APST: On Change     ASG :
ASP : PTR (nil)     BKLNK: ELL 0 [(nil) .. (nil)]           BKPT: 00
DESC:               DISA: 0             DISP: 0             DISS: NO_ALARM
DISV: 1             DPVT: PTR (nil)     DSET: PTR 0x10249a0 DTYP: Soft Timestamp
EVNT:               FLNK: DB_LINK LAB:AMC-008-5:TimestampIn.PROC
INP : INST_IO @%s.%f                    LCNT: 0             LSET: PTR 0x3c75230
MLIS: ELL 1 [0x7f9e1c052d18 .. 0x7f9e1c052d18]
MLOK: b0 df e2 03 00 00 00 00           MPST: On Change
NAME: LAB:AMC-008-5:TimestampClk        NSEV: NO_ALARM      NSTA: NO_ALARM
OLDSIMM: NO         OVAL: 1585039504.002516850              PACT: 0
PHAS: 0             PINI: NO            PPN : PTR (nil)     PPNR: PTR (nil)
PRIO: LOW           PROC: 0             PUTF: 0             RDES: PTR 0x243f560
RPRO: 0             RSET: PTR 0x10236e0 SCAN: 10 second     SDIS: CONSTANT
SDLY: -1            SEVR: NO_ALARM      SIML: CONSTANT      SIMM: NO
SIMPVT: PTR (nil)   SIMS: NO_ALARM      SIOL: CONSTANT      SPVT: PTR 0x57cfd40
SSCN: <nil>         STAT: NO_ALARM      SVAL:
TIME: 2020-03-24 09:45:04.002516850     TPRO: 0             TSE : 0
TSEL: CA_LINK LAB-EVR-008:EvtECnt-I.TIME CP MS              UDF : 0
UDFS: INVALID       VAL : 1585039504.002516850

epics> dbpr LAB:AMC-008-5:TimestampIn 4
ACKS: NO_ALARM      ACKT: YES           APST: On Change     ASG :
ASP : PTR (nil)     BKLNK: ELL 2 [0x2726738 .. 0x26ef708]   BKPT: 00
DESC:               DISA: 0             DISP: 0             DISS: NO_ALARM
DISV: 1             DPVT: PTR (nil)     DSET: PTR 0x10249a0 DTYP: Soft Timestamp
EVNT:               FLNK: DB_LINK LAB:AMC-008-5:Timestamp.PROC
INP : INST_IO @%s.%f                    LCNT: 0             LSET: PTR 0x3c75390
MLIS: ELL 3 [0x7f9e6c1287a0 .. 0x7f9e940071f0]
MLOK: 20 e0 e2 03 00 00 00 00           MPST: On Change
NAME: LAB:AMC-008-5:TimestampIn         NSEV: NO_ALARM      NSTA: NO_ALARM
OLDSIMM: NO         OVAL: 1585039504.002516850              PACT: 0
PHAS: 0             PINI: NO            PPN : PTR (nil)     PPNR: PTR (nil)
PRIO: LOW           PROC: 0             PUTF: 0             RDES: PTR 0x243f560
RPRO: 0             RSET: PTR 0x10236e0 SCAN: Passive       SDIS: CONSTANT
SDLY: -1            SEVR: NO_ALARM      SIML: CONSTANT      SIMM: NO
SIMPVT: PTR (nil)   SIMS: NO_ALARM      SIOL: CONSTANT      SPVT: PTR (nil)
SSCN: <nil>         STAT: NO_ALARM      SVAL:
TIME: 2020-03-24 09:45:04.002516850     TPRO: 0             TSE : 0
TSEL: CA_LINK LAB-EVR-008:EvtECnt-I.TIME NPP MS             UDF : 0
UDFS: INVALID       VAL : 1585039504.002516850

Notice the TSEL is set to be CA_LINK in both cases above.

The camonitor is just stuck on last update, no indication of error/alarm:
....
LAB:AMC-008-5:TimestampClk     2020-03-24 09:45:03.931073 1585039503.931072848
LAB:AMC-008-5:TimestampIn      2020-03-24 09:45:03.931073 1585039503.931072848
LAB:AMC-008-5:TimestampClk     2020-03-24 09:45:04.002517 1585039504.002516850
LAB:AMC-008-5:TimestampIn      2020-03-24 09:45:04.002517 1585039504.002516850

Thanks,
//hinko
________________________________________
From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Monday, March 23, 2020 6:41:44 PM
To: Hinko Kocevar
Cc: tech-talk
Subject: Re: detecting remote PV disconnect (asyn)

Hi Hinko,

> Is there a way to check that the EVR PV disappeared?

If you are willing to live with a bit more overhead you could you could do the following:

Remove the CP attribute from TimestampIn.  That way it will attempt to do a normal read which should definitely return an error if the remote IOC is down.

Add this record.  It will process TimestampIn via a FLNK when the EVR changes, and also every 10 seconds.  That way TimestampIn will be periodically trying to read even if EVR IOC is down, and should detect disconnects.

record(stringin, "$(P)$(R)TimestampClk") {
    field(DTYP, "Soft Timestamp")
    # use dummy placeholder PV, change EVR prefix to a desired EVR IOC
    field(TSEL, "LAB-EVR-000:EvtECnt-I.TIME CP MS")
    field(INP,  "@%s.%f")
    field(FLNK, "$(P)$(R)TimestampIn.PROC PP MS")

    field(SCAN, "10 second")
}

Mark

________________________________
From: Hinko Kocevar <Hinko.Kocevar at ess.eu>
Sent: Monday, March 23, 2020 11:55 AM
To: Mark Rivers
Cc: tech-talk
Subject: Re: detecting remote PV disconnect (asyn)

Hi Mark,

> - Both $(P)$(R)TimestampIn and $(P)$(R)Timestamp are in the same IOC.  That is not the IOC you are killing.

These live in the DAQ IOC, not EVR IOC (which I'm killing).

> - The TSEL link is talking to another IOC (the EVR IOC), PV="LAB-
EVR-000:EvtECnt-I.TIME CP MS"

Yes LAB-
EVR-000:EvtECnt-I.TIME is on EVR IOC. Using MS flag there makes me think that alarm severity would be propagated to the record holding TSEL (I have never done this before so I might be wrong on this assumption).

> - Is the OUT link is talking to a local driver?  What does this driver do?  "@asyn($(PORT),$(ADDR=0),$(TIMEOUT=1))SIS8300.EVR.TIMESTAMP")

Yes, OUT is local to an asyn driver (DAQ driver). Asyn code for DAQ driver takes the value obtained from EVR IOC PV and writes it to the firmware on top of which the DAQ IOC sits.

> Does TimeStampIn record ever go into alarm state when the EVR IOC shuts down?  It may take the time that the OS requires to detect that a socket is closed, which can be many minutes.

I do not think so. I've been looking at the "dbpr TimeStampIn 4" and none of the fields change value when comparing alive EVR PV vs dead EVR PV.

> It seems to me that the record that should first detect the problem is not using asyn, it is the TimestampIn record.  So this is really not an asyn question, it is the behavior of EPICS base in general for CP links.

True. The first record is no using asyn.
Is there a way to check that the EVR PV disappeared?

//hinko

Mark

________________________________
From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Hinko Kocevar via Tech-talk <tech-talk at aps.anl.gov>
Sent: Monday, March 23, 2020 11:11 AM
To: tech-talk at aps.anl.gov
Subject: detecting remote PV disconnect (asyn)

Hi,

I have an EVR PV that delivers timestamp to my asyn based DAQ IOC. I want to know if and when that PV is inaccessible - i.e. EVR IOC not running / died.

I have this DB construct:

# get timestamp from EVR event PV
# i.e. set TSEL: LAB-EVR-004:EvtECnt-I.TIME CP MS
record(stringin, "$(P)$(R)TimestampIn") {
    field(DTYP, "Soft Timestamp")
    # use dummy placeholder PV, change EVR prefix to a desired EVR IOC
    field(TSEL, "LAB-EVR-000:EvtECnt-I.TIME CP MS")
    field(INP,  "@%s.%f")
    field(FLNK, "$(P)$(R)Timestamp.PROC PP MS")
    info(autosaveFields, "TSEL")
}

# holds TS value in 'sec.nsec' format
record(stringout, "$(P)$(R)Timestamp")
{
    field(DOL,  "$(P)$(R)TimestampIn")
    field(OMSL, "closed_loop")
    field(DTYP, "asynOctetWrite")
    field(OUT,  "@asyn($(PORT),$(ADDR=0),$(TIMEOUT=1))SIS8300.EVR.TIMESTAMP")
}

Works as expected. The $(P)$(R)Timestamp is refreshed upon each update of the LAB-EVR-000:EvtECnt-I PV, feeding the value to the asyn based IOC code.

Upon killing the EVR IOC the $(P)$(R)TimestampIn and $(P)$(R)Timestamp show no apparent state change that the LAB-EVR-000:EvtECnt-I went away. At least looking that the "dbpr ... 4" output for the above mentioned PVs.

I was expecting the alarm fields to be set to something like "not connected". Is there a way, in asyn based IOC, to detect this situation?

Thanks,
Hinko

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System