Re: detecting remote PV disconnect (asyn)

[Hinko Kocevar via Tech-talk <tech-talk at aps.anl.gov>]

After tinkering a little bit more I got this to show link disconnected on camonitor:

record(stringin, "$(P)$(R)TimestampClk") {
    field(INP, "LAB-EVR-008:EvtECnt-I.VAL CP MS")
    field(FLNK, "$(P)$(R)TimestampIn.PROC PP MS")
    #field(SCAN, "10 second")
}

LAB:AMC-008-5:TimestampClk     2020-03-24 13:31:56.866964 131
LAB:AMC-008-5:TimestampIn      2020-03-24 13:31:56.866317 1585053116.866316538
LAB:AMC-008-5:Timestamp        2020-03-24 13:31:56.868907 1585053116.866316538
LAB:AMC-008-5:TimestampClk     2020-03-24 13:31:56.872925 131 LINK INVALID

The alarm is not propagated to the TimestampIn record, though. Is it the DTYP = "Soft Timestamp" in TimestampIn that interferes ?

At the moment that is OK with me. I had more success with longin records from EVR that I'm going to have feed the value into my DAQ anyway.
On those longin records, with DB constructed pretty much similar as with Timestamp records, I can see the alarm propagation all the way to the asyn record:

LAB:AMC-008-5:BeamStateIn      2020-03-24 13:31:56.678547 20
LAB:AMC-008-5:BeamStateR       2020-03-24 13:31:56.678653 20
LAB:AMC-008-5:BeamState        2020-03-24 13:31:56.683638 20
LAB:AMC-008-5:BeamStateIn      2020-03-24 13:31:56.872914 20 LINK INVALID
LAB:AMC-008-5:BeamState        2020-03-24 13:31:56.890227 20 LINK INVALID

Here BeamState is asyn based record. I'll just rely on one of these PV alarm status in my code to detect if EVR IOC is still there or not.
One thing that I had to add was MS to the DOL field of the BeamState that points to BeamStateIn; that made it propagate the alarm from BeamStateIn to BeamState.

Thank you!
//hinko
________________________________________
From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Tuesday, March 24, 2020 12:30:01 PM
To: Hinko Kocevar
Cc: tech-talk
Subject: Re: detecting remote PV disconnect (asyn)

Hi Hinko,

What happens if you change TimeStampClk to use normal soft device support and read into the VAL field, rather than TSEL?  Does the record go into alarm?  This is simpler and hopefully it will work.


record(stringin, "$(P)$(R)TimestampClk") {
    field(INP, "LAB-EVR-008:EvtECnt-I.TIME CP MS")
    field(FLNK, "$(P)$(R)TimestampIn.PROC PP MS")
    field(SCAN, "10 second")
}

Mark

________________________________
From: Hinko Kocevar <Hinko.Kocevar at ess.eu>
Sent: Tuesday, March 24, 2020 3:58 AM
To: Mark Rivers
Cc: tech-talk
Subject: Re: detecting remote PV disconnect (asyn)

Mark, I have this DB now (with the rest of PVs the same as in original post):

record(stringin, "$(P)$(R)TimestampClk") {
    field(DTYP, "Soft Timestamp")
    # use dummy placeholder PV, change EVR prefix to a desired EVR IOC
    field(TSEL, "LAB-EVR-008:EvtECnt-I.TIME CP MS")
    field(INP,  "@%s.%f")
    field(FLNK, "$(P)$(R)TimestampIn.PROC PP MS")

    field(SCAN, "10 second")
}

# get timestamp from EVR event PV
# i.e. set TSEL: LAB-EVR-004:EvtECnt-I.TIME CP MS
record(stringin, "$(P)$(R)TimestampIn") {
    field(DTYP, "Soft Timestamp")
    # use dummy placeholder PV, change EVR prefix to a desired EVR IOC
    field(TSEL, "LAB-EVR-008:EvtECnt-I.TIME MS")
    field(INP,  "@%s.%f")
    field(FLNK, "$(P)$(R)Timestamp.PROC PP MS")
    info(autosaveFields, "TSEL")
}

When I kill the EVR IOC I get these messages instantaneously:

epics> DB CA Link Exception: "Virtual circuit disconnect", context "bd-cpu08.cslab.esss.lu.se:43805"
recGblGetTimeStamp: dbGetTimeStamp failed for LAB:AMC-008-5:TimestampClk.TSEL
recGblGetTimeStamp: dbGetTimeStamp failed for LAB:AMC-008-5:TimestampClk.TSEL
recGblGetTimeStamp: dbGetTimeStamp failed for LAB:AMC-008-5:TimestampIn.TSEL
recGblGetTimeStamp: dbGetTimeStamp failed for LAB:AMC-008-5:TimestampIn.TSEL

And they keep repeating on 10 second interval, all 4 of them.

epics> dbpr LAB:AMC-008-5:TimestampClk 4
ACKS: NO_ALARM      ACKT: YES           APST: On Change     ASG :
ASP : PTR (nil)     BKLNK: ELL 0 [(nil) .. (nil)]           BKPT: 00
DESC:               DISA: 0             DISP: 0             DISS: NO_ALARM
DISV: 1             DPVT: PTR (nil)     DSET: PTR 0x10249a0 DTYP: Soft Timestamp
EVNT:               FLNK: DB_LINK LAB:AMC-008-5:TimestampIn.PROC
INP : INST_IO @%s.%f                    LCNT: 0             LSET: PTR 0x3c75230
MLIS: ELL 1 [0x7f9e1c052d18 .. 0x7f9e1c052d18]
MLOK: b0 df e2 03 00 00 00 00           MPST: On Change
NAME: LAB:AMC-008-5:TimestampClk        NSEV: NO_ALARM      NSTA: NO_ALARM
OLDSIMM: NO         OVAL: 1585039504.002516850              PACT: 0
PHAS: 0             PINI: NO            PPN : PTR (nil)     PPNR: PTR (nil)
PRIO: LOW           PROC: 0             PUTF: 0             RDES: PTR 0x243f560
RPRO: 0             RSET: PTR 0x10236e0 SCAN: 10 second     SDIS: CONSTANT
SDLY: -1            SEVR: NO_ALARM      SIML: CONSTANT      SIMM: NO
SIMPVT: PTR (nil)   SIMS: NO_ALARM      SIOL: CONSTANT      SPVT: PTR 0x57cfd40
SSCN: <nil>         STAT: NO_ALARM      SVAL:
TIME: 2020-03-24 09:45:04.002516850     TPRO: 0             TSE : 0
TSEL: CA_LINK LAB-EVR-008:EvtECnt-I.TIME CP MS              UDF : 0
UDFS: INVALID       VAL : 1585039504.002516850

epics> dbpr LAB:AMC-008-5:TimestampIn 4
ACKS: NO_ALARM      ACKT: YES           APST: On Change     ASG :
ASP : PTR (nil)     BKLNK: ELL 2 [0x2726738 .. 0x26ef708]   BKPT: 00
DESC:               DISA: 0             DISP: 0             DISS: NO_ALARM
DISV: 1             DPVT: PTR (nil)     DSET: PTR 0x10249a0 DTYP: Soft Timestamp
EVNT:               FLNK: DB_LINK LAB:AMC-008-5:Timestamp.PROC
INP : INST_IO @%s.%f                    LCNT: 0             LSET: PTR 0x3c75390
MLIS: ELL 3 [0x7f9e6c1287a0 .. 0x7f9e940071f0]
MLOK: 20 e0 e2 03 00 00 00 00           MPST: On Change
NAME: LAB:AMC-008-5:TimestampIn         NSEV: NO_ALARM      NSTA: NO_ALARM
OLDSIMM: NO         OVAL: 1585039504.002516850              PACT: 0
PHAS: 0             PINI: NO            PPN : PTR (nil)     PPNR: PTR (nil)
PRIO: LOW           PROC: 0             PUTF: 0             RDES: PTR 0x243f560
RPRO: 0             RSET: PTR 0x10236e0 SCAN: Passive       SDIS: CONSTANT
SDLY: -1            SEVR: NO_ALARM      SIML: CONSTANT      SIMM: NO
SIMPVT: PTR (nil)   SIMS: NO_ALARM      SIOL: CONSTANT      SPVT: PTR (nil)
SSCN: <nil>         STAT: NO_ALARM      SVAL:
TIME: 2020-03-24 09:45:04.002516850     TPRO: 0             TSE : 0
TSEL: CA_LINK LAB-EVR-008:EvtECnt-I.TIME NPP MS             UDF : 0
UDFS: INVALID       VAL : 1585039504.002516850


Notice the TSEL is set to be CA_LINK in both cases above.


The camonitor is just stuck on last update, no indication of error/alarm:
....
LAB:AMC-008-5:TimestampClk     2020-03-24 09:45:03.931073 1585039503.931072848
LAB:AMC-008-5:TimestampIn      2020-03-24 09:45:03.931073 1585039503.931072848
LAB:AMC-008-5:TimestampClk     2020-03-24 09:45:04.002517 1585039504.002516850
LAB:AMC-008-5:TimestampIn      2020-03-24 09:45:04.002517 1585039504.002516850


Thanks,
//hinko
________________________________________
From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Monday, March 23, 2020 6:41:44 PM
To: Hinko Kocevar
Cc: tech-talk
Subject: Re: detecting remote PV disconnect (asyn)

Hi Hinko,


> Is there a way to check that the EVR PV disappeared?


If you are willing to live with a bit more overhead you could you could do the following:


Remove the CP attribute from TimestampIn.  That way it will attempt to do a normal read which should definitely return an error if the remote IOC is down.


Add this record.  It will process TimestampIn via a FLNK when the EVR changes, and also every 10 seconds.  That way TimestampIn will be periodically trying to read even if EVR IOC is down, and should detect disconnects.


record(stringin, "$(P)$(R)TimestampClk") {
    field(DTYP, "Soft Timestamp")
    # use dummy placeholder PV, change EVR prefix to a desired EVR IOC
    field(TSEL, "LAB-EVR-000:EvtECnt-I.TIME CP MS")
    field(INP,  "@%s.%f")
    field(FLNK, "$(P)$(R)TimestampIn.PROC PP MS")

    field(SCAN, "10 second")
}


Mark


________________________________
From: Hinko Kocevar <Hinko.Kocevar at ess.eu>
Sent: Monday, March 23, 2020 11:55 AM
To: Mark Rivers
Cc: tech-talk
Subject: Re: detecting remote PV disconnect (asyn)

Hi Mark,

> - Both $(P)$(R)TimestampIn and $(P)$(R)Timestamp are in the same IOC.  That is not the IOC you are killing.

These live in the DAQ IOC, not EVR IOC (which I'm killing).


> - The TSEL link is talking to another IOC (the EVR IOC), PV="LAB-
EVR-000:EvtECnt-I.TIME CP MS"

Yes LAB-
EVR-000:EvtECnt-I.TIME is on EVR IOC. Using MS flag there makes me think that alarm severity would be propagated to the record holding TSEL (I have never done this before so I might be wrong on this assumption).


> - Is the OUT link is talking to a local driver?  What does this driver do?  "@asyn($(PORT),$(ADDR=0),$(TIMEOUT=1))SIS8300.EVR.TIMESTAMP")

Yes, OUT is local to an asyn driver (DAQ driver). Asyn code for DAQ driver takes the value obtained from EVR IOC PV and writes it to the firmware on top of which the DAQ IOC sits.

> Does TimeStampIn record ever go into alarm state when the EVR IOC shuts down?  It may take the time that the OS requires to detect that a socket is closed, which can be many minutes.

I do not think so. I've been looking at the "dbpr TimeStampIn 4" and none of the fields change value when comparing alive EVR PV vs dead EVR PV.

> It seems to me that the record that should first detect the problem is not using asyn, it is the TimestampIn record.  So this is really not an asyn question, it is the behavior of EPICS base in general for CP links.

True. The first record is no using asyn.
Is there a way to check that the EVR PV disappeared?


//hinko

Mark

________________________________
From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Hinko Kocevar via Tech-talk <tech-talk at aps.anl.gov>
Sent: Monday, March 23, 2020 11:11 AM
To: tech-talk at aps.anl.gov
Subject: detecting remote PV disconnect (asyn)

Hi,

I have an EVR PV that delivers timestamp to my asyn based DAQ IOC. I want to know if and when that PV is inaccessible - i.e. EVR IOC not running / died.

I have this DB construct:

# get timestamp from EVR event PV
# i.e. set TSEL: LAB-EVR-004:EvtECnt-I.TIME CP MS
record(stringin, "$(P)$(R)TimestampIn") {
    field(DTYP, "Soft Timestamp")
    # use dummy placeholder PV, change EVR prefix to a desired EVR IOC
    field(TSEL, "LAB-EVR-000:EvtECnt-I.TIME CP MS")
    field(INP,  "@%s.%f")
    field(FLNK, "$(P)$(R)Timestamp.PROC PP MS")
    info(autosaveFields, "TSEL")
}

# holds TS value in 'sec.nsec' format
record(stringout, "$(P)$(R)Timestamp")
{
    field(DOL,  "$(P)$(R)TimestampIn")
    field(OMSL, "closed_loop")
    field(DTYP, "asynOctetWrite")
    field(OUT,  "@asyn($(PORT),$(ADDR=0),$(TIMEOUT=1))SIS8300.EVR.TIMESTAMP")
}


Works as expected. The $(P)$(R)Timestamp is refreshed upon each update of the LAB-EVR-000:EvtECnt-I PV, feeding the value to the asyn based IOC code.

Upon killing the EVR IOC the $(P)$(R)TimestampIn and $(P)$(R)Timestamp show no apparent state change that the LAB-EVR-000:EvtECnt-I went away. At least looking that the "dbpr ... 4" output for the above mentioned PVs.

I was expecting the alarm fields to be set to something like "not connected". Is there a way, in asyn based IOC, to detect this situation?

Thanks,
Hinko