Experimental Physics and
Industrial Control System

Hinko Kocevar via Tech-talk <tech-talk at aps.anl.gov> · Tue, 24 Mar 2020 08:58:33 +0000

Mark, I have this DB now (with the rest of PVs the same as in original post):

record(stringin, "$(P)$(R)TimestampClk") {
    field(DTYP, "Soft Timestamp")
    # use dummy placeholder PV, change EVR prefix to a desired EVR IOC
    field(TSEL, "LAB-EVR-008:EvtECnt-I.TIME CP MS")
    field(INP,  "@%s.%f")
    field(FLNK, "$(P)$(R)TimestampIn.PROC PP MS")

    field(SCAN, "10 second")
}

# get timestamp from EVR event PV
# i.e. set TSEL: LAB-EVR-004:EvtECnt-I.TIME CP MS
record(stringin, "$(P)$(R)TimestampIn") {
    field(DTYP, "Soft Timestamp")
    # use dummy placeholder PV, change EVR prefix to a desired EVR IOC
    field(TSEL, "LAB-EVR-008:EvtECnt-I.TIME MS")
    field(INP,  "@%s.%f")
    field(FLNK, "$(P)$(R)Timestamp.PROC PP MS")
    info(autosaveFields, "TSEL")
}

When I kill the EVR IOC I get these messages instantaneously:

epics> DB CA Link Exception: "Virtual circuit disconnect", context "bd-cpu08.cslab.esss.lu.se:43805"
recGblGetTimeStamp: dbGetTimeStamp failed for LAB:AMC-008-5:TimestampClk.TSEL
recGblGetTimeStamp: dbGetTimeStamp failed for LAB:AMC-008-5:TimestampClk.TSEL
recGblGetTimeStamp: dbGetTimeStamp failed for LAB:AMC-008-5:TimestampIn.TSEL
recGblGetTimeStamp: dbGetTimeStamp failed for LAB:AMC-008-5:TimestampIn.TSEL

And they keep repeating on 10 second interval, all 4 of them.

epics> dbpr LAB:AMC-008-5:TimestampClk 4
ACKS: NO_ALARM      ACKT: YES           APST: On Change     ASG :
ASP : PTR (nil)     BKLNK: ELL 0 [(nil) .. (nil)]           BKPT: 00
DESC:               DISA: 0             DISP: 0             DISS: NO_ALARM
DISV: 1             DPVT: PTR (nil)     DSET: PTR 0x10249a0 DTYP: Soft Timestamp
EVNT:               FLNK: DB_LINK LAB:AMC-008-5:TimestampIn.PROC
INP : INST_IO @%s.%f                    LCNT: 0             LSET: PTR 0x3c75230
MLIS: ELL 1 [0x7f9e1c052d18 .. 0x7f9e1c052d18]
MLOK: b0 df e2 03 00 00 00 00           MPST: On Change
NAME: LAB:AMC-008-5:TimestampClk        NSEV: NO_ALARM      NSTA: NO_ALARM
OLDSIMM: NO         OVAL: 1585039504.002516850              PACT: 0
PHAS: 0             PINI: NO            PPN : PTR (nil)     PPNR: PTR (nil)
PRIO: LOW           PROC: 0             PUTF: 0             RDES: PTR 0x243f560
RPRO: 0             RSET: PTR 0x10236e0 SCAN: 10 second     SDIS: CONSTANT
SDLY: -1            SEVR: NO_ALARM      SIML: CONSTANT      SIMM: NO
SIMPVT: PTR (nil)   SIMS: NO_ALARM      SIOL: CONSTANT      SPVT: PTR 0x57cfd40
SSCN: <nil>         STAT: NO_ALARM      SVAL:
TIME: 2020-03-24 09:45:04.002516850     TPRO: 0             TSE : 0
TSEL: CA_LINK LAB-EVR-008:EvtECnt-I.TIME CP MS              UDF : 0
UDFS: INVALID       VAL : 1585039504.002516850

epics> dbpr LAB:AMC-008-5:TimestampIn 4
ACKS: NO_ALARM      ACKT: YES           APST: On Change     ASG :
ASP : PTR (nil)     BKLNK: ELL 2 [0x2726738 .. 0x26ef708]   BKPT: 00
DESC:               DISA: 0             DISP: 0             DISS: NO_ALARM
DISV: 1             DPVT: PTR (nil)     DSET: PTR 0x10249a0 DTYP: Soft Timestamp
EVNT:               FLNK: DB_LINK LAB:AMC-008-5:Timestamp.PROC
INP : INST_IO @%s.%f                    LCNT: 0             LSET: PTR 0x3c75390
MLIS: ELL 3 [0x7f9e6c1287a0 .. 0x7f9e940071f0]
MLOK: 20 e0 e2 03 00 00 00 00           MPST: On Change
NAME: LAB:AMC-008-5:TimestampIn         NSEV: NO_ALARM      NSTA: NO_ALARM
OLDSIMM: NO         OVAL: 1585039504.002516850              PACT: 0
PHAS: 0             PINI: NO            PPN : PTR (nil)     PPNR: PTR (nil)
PRIO: LOW           PROC: 0             PUTF: 0             RDES: PTR 0x243f560
RPRO: 0             RSET: PTR 0x10236e0 SCAN: Passive       SDIS: CONSTANT
SDLY: -1            SEVR: NO_ALARM      SIML: CONSTANT      SIMM: NO
SIMPVT: PTR (nil)   SIMS: NO_ALARM      SIOL: CONSTANT      SPVT: PTR (nil)
SSCN: <nil>         STAT: NO_ALARM      SVAL:
TIME: 2020-03-24 09:45:04.002516850     TPRO: 0             TSE : 0
TSEL: CA_LINK LAB-EVR-008:EvtECnt-I.TIME NPP MS             UDF : 0
UDFS: INVALID       VAL : 1585039504.002516850

Notice the TSEL is set to be CA_LINK in both cases above.

The camonitor is just stuck on last update, no indication of error/alarm:
....
LAB:AMC-008-5:TimestampClk     2020-03-24 09:45:03.931073 1585039503.931072848
LAB:AMC-008-5:TimestampIn      2020-03-24 09:45:03.931073 1585039503.931072848
LAB:AMC-008-5:TimestampClk     2020-03-24 09:45:04.002517 1585039504.002516850
LAB:AMC-008-5:TimestampIn      2020-03-24 09:45:04.002517 1585039504.002516850

Thanks,
//hinko
________________________________________
From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Monday, March 23, 2020 6:41:44 PM
To: Hinko Kocevar
Cc: tech-talk
Subject: Re: detecting remote PV disconnect (asyn)

Hi Hinko,

> Is there a way to check that the EVR PV disappeared?

If you are willing to live with a bit more overhead you could you could do the following:

Remove the CP attribute from TimestampIn.  That way it will attempt to do a normal read which should definitely return an error if the remote IOC is down.

Add this record.  It will process TimestampIn via a FLNK when the EVR changes, and also every 10 seconds.  That way TimestampIn will be periodically trying to read even if EVR IOC is down, and should detect disconnects.

record(stringin, "$(P)$(R)TimestampClk") {
    field(DTYP, "Soft Timestamp")
    # use dummy placeholder PV, change EVR prefix to a desired EVR IOC
    field(TSEL, "LAB-EVR-000:EvtECnt-I.TIME CP MS")
    field(INP,  "@%s.%f")
    field(FLNK, "$(P)$(R)TimestampIn.PROC PP MS")

    field(SCAN, "10 second")
}

Mark

________________________________
From: Hinko Kocevar <Hinko.Kocevar at ess.eu>
Sent: Monday, March 23, 2020 11:55 AM
To: Mark Rivers
Cc: tech-talk
Subject: Re: detecting remote PV disconnect (asyn)

Hi Mark,

> - Both $(P)$(R)TimestampIn and $(P)$(R)Timestamp are in the same IOC.  That is not the IOC you are killing.

These live in the DAQ IOC, not EVR IOC (which I'm killing).

> - The TSEL link is talking to another IOC (the EVR IOC), PV="LAB-
EVR-000:EvtECnt-I.TIME CP MS"

Yes LAB-
EVR-000:EvtECnt-I.TIME is on EVR IOC. Using MS flag there makes me think that alarm severity would be propagated to the record holding TSEL (I have never done this before so I might be wrong on this assumption).

> - Is the OUT link is talking to a local driver?  What does this driver do?  "@asyn($(PORT),$(ADDR=0),$(TIMEOUT=1))SIS8300.EVR.TIMESTAMP")

Yes, OUT is local to an asyn driver (DAQ driver). Asyn code for DAQ driver takes the value obtained from EVR IOC PV and writes it to the firmware on top of which the DAQ IOC sits.

> Does TimeStampIn record ever go into alarm state when the EVR IOC shuts down?  It may take the time that the OS requires to detect that a socket is closed, which can be many minutes.

I do not think so. I've been looking at the "dbpr TimeStampIn 4" and none of the fields change value when comparing alive EVR PV vs dead EVR PV.

> It seems to me that the record that should first detect the problem is not using asyn, it is the TimestampIn record.  So this is really not an asyn question, it is the behavior of EPICS base in general for CP links.

True. The first record is no using asyn.
Is there a way to check that the EVR PV disappeared?

//hinko

Mark

________________________________
From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Hinko Kocevar via Tech-talk <tech-talk at aps.anl.gov>
Sent: Monday, March 23, 2020 11:11 AM
To: tech-talk at aps.anl.gov
Subject: detecting remote PV disconnect (asyn)

Hi,

I have an EVR PV that delivers timestamp to my asyn based DAQ IOC. I want to know if and when that PV is inaccessible - i.e. EVR IOC not running / died.

I have this DB construct:

# get timestamp from EVR event PV
# i.e. set TSEL: LAB-EVR-004:EvtECnt-I.TIME CP MS
record(stringin, "$(P)$(R)TimestampIn") {
    field(DTYP, "Soft Timestamp")
    # use dummy placeholder PV, change EVR prefix to a desired EVR IOC
    field(TSEL, "LAB-EVR-000:EvtECnt-I.TIME CP MS")
    field(INP,  "@%s.%f")
    field(FLNK, "$(P)$(R)Timestamp.PROC PP MS")
    info(autosaveFields, "TSEL")
}

# holds TS value in 'sec.nsec' format
record(stringout, "$(P)$(R)Timestamp")
{
    field(DOL,  "$(P)$(R)TimestampIn")
    field(OMSL, "closed_loop")
    field(DTYP, "asynOctetWrite")
    field(OUT,  "@asyn($(PORT),$(ADDR=0),$(TIMEOUT=1))SIS8300.EVR.TIMESTAMP")
}

Works as expected. The $(P)$(R)Timestamp is refreshed upon each update of the LAB-EVR-000:EvtECnt-I PV, feeding the value to the asyn based IOC code.

Upon killing the EVR IOC the $(P)$(R)TimestampIn and $(P)$(R)Timestamp show no apparent state change that the LAB-EVR-000:EvtECnt-I went away. At least looking that the "dbpr ... 4" output for the above mentioned PVs.

I was expecting the alarm fields to be set to something like "not connected". Is there a way, in asyn based IOC, to detect this situation?

Thanks,
Hinko

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System