RE: streamdevice + asyn stuck

["Mark Rivers" <rivers@cars.uchicago.edu>]

I think I will have to defer to Dirk on this.  If the underlying port is
still working, I don't understand why the record is stuck with PACT=1.

If you temporarily change the record from SCAN=Event to "1 second" does
it still hang up?

Mark


-----Original Message-----
From: Zhang, Dehong [mailto:dhzhang@slac.stanford.edu] 
Sent: Monday, July 11, 2011 1:24 PM
To: Mark Rivers; dirk.zimoch@psi.ch; tech-talk@aps.anl.gov
Subject: RE: streamdevice + asyn stuck

Hi Mark,

Thank you very much for spending time on this!

The SCAN field is "Event".  I set up an event dispatcher to send out a
few
different events at certain rates.  Those records listen to the events.

Those records talk to MKS937A controlers via digi port servers with TCP.

When the problem happens, the PACT field is "1", and the record is stuck
forever.  But, the port is still active, other records can still use it.

Please help to correct my setup/configuration, so that when anything bad
happens, the record can recover in the next cycle. ( -- could it be
input
mismatch?)

Here is one example:

------------------------------------------------------------------------
-----------------------------
record(mbbi, "TST:DSU:GCC:01:STATE") {
  field(DTYP, "stream")
  field(INP,  "@mks937a.proto.db state(2) TST:R54:GCT:16:B 0")
  field(DESC, "Slot=A1")
  field(SCAN, "Event")
  field(EVNT, "1")
  field(PHAS, "2")
  ...
  field(FLNK, "TST:DSU:GCC:01:STATUSCALC")
}

------------------------------------------------------------------------
-----------------------------
part of the protocol file:
locktimeout  = 5000;
terminator   = CR;
replytimeout = 1000;
readtimeout  = 1000;
extrainput   = Ignore;

@init{out " ";}

state    { out "R\$1"; in "%{\?.|
\?E|HI|A|Lo|F|H|W|L|CON|P|NOG|M|NOT|Not|C}"; }

------------------------------------------------------------------------
-----------------------------
asyn configure:
drvAsynIPPortConfigure("TST:R54:GCT:16:B", "TST-R59-SER-04:2104
TCP",0,0,0)

------------------------------------------------------------------------
-----------------------------
ioc-tst-vacuum> dbpr TST:DSU:GCC:01:STATE 10
ACKS: INVALID       ACKT: YES           ASG:                
ASP: 0x0000000000000000                 BKPT: 0x00          COSV:
NO_ALARM      
DESC: Slot=A1       DISA: 0             DISP: 0             DISS:
NO_ALARM      
DISV: 1             DPVT: 0x10c5fa0c00000000                
DSET: 0x40c0df18db2a0000                DTYP: stream        EIST: Below
Range   
EISV: NO_ALARM      EIVL: 0x0           ELST: No Gauge      ELSV: MAJOR

ELVL: 0x0           EVNT: 1             FFST: Locked Out    FFSV: MAJOR

FFVL: 0x0           FLNK:DB_LINK TST:DSU:GCC:01:STATUSCALC  FRST: Low
Emission  
FRSV: MAJOR         FRVL: 0x0           FTST: Bad Command   FTSV: MAJOR

FTVL: 0x0           FVST: Filament Off  FVSV: MAJOR         FVVL: 0x0

INP:INST_IO @mks937a.proto.db state(2) TST:R54:GCT:16:B 0   LALM: 0

LCNT: 11            LSET: 0x50ee070d00000000                MASK: 0

MLIS: 0x0000000000000000000000000000000000000000            
MLOK: 0xc0d7bf0c00000000                MLST: 0             
NAME: XXX:DSU:GCC:01:STATE              NIST: Controlled    NISV: MINOR

NIVL: 0x0           NOBT: 0             NSEV: NO_ALARM      NSTA:
NO_ALARM      
ONST: ON            ONSV: NO_ALARM      ONVL: 0x0           ORAW: 0

PACT: 1             PHAS: 2             PINI: NO            
PPN: 0x0000000000000000                 PPNR: 0x0000000000000000

PRIO: LOW           PROC: 0             PUTF: 0             
RDES: 0x6084830c00000000                RPRO: 0             
RSET: 0xc02f2219db2a0000                RVAL: 0             SCAN: Event

SDEF: 1             SDIS:CONSTANT       SEVR: INVALID       SHFT: 0

SIML:CONSTANT       SIMM: NO            SIMS: NO_ALARM
SIOL:CONSTANT       
SPVT: 0x6057100d00000000                STAT: SCAN          SVAL: 0

SVST: Startup Delay SVSV: MINOR         SVVL: 0x0           SXST: HV Off

SXSV: MAJOR         SXVL: 0x0           TEST: Protected State

TESV: MINOR         TEVL: 0x0           THST: At Atmosphere THSV: MINOR

THVL: 0x0           TIME: 2011-07-10 00:55:28.506299000     TPRO: 0

TSE: 0              TSEL:CONSTANT       TTST: Wrong Gauge   TTSV: MAJOR

TTVL: 0x0           TVST: Not Connected TVSV: MAJOR         TVVL: 0x0

TWST: Above Range   TWSV: MINOR         TWVL: 0x0           UDF: 0

UNSV: MAJOR         VAL: 0              ZRST: ON            ZRSV:
NO_ALARM      
ZRVL: 0x0           

------------------------------------------------------------------------
-----------------------------
ioc-tst-vacuum> asynReport 10 TST:R54:GCT:16:B
TST:R54:GCT:16:B multiDevice:No canBlock:Yes autoConnect:Yes
    enabled:Yes connected:Yes numberConnects 1
    nDevices 0 nQueued 0 blocked:No
    asynManagerLock:No synchronousLock:No
    exceptionActive:No exceptionUsers 1 exceptionNotifys 0
    interposeInterfaceList
        asynOctet pinterface 0x2adb18bbb1e0 drvPvt 0xc8a2b70
    interfaceList
        asynCommon pinterface 0x2adb18bb9100 drvPvt 0xc8a2070
        asynOctet pinterface 0xc8a20e8 drvPvt 0xc8a2070
    Port TST-R59-SER-04:2104 TCP: Connected
                    fd: 13
    Characters written: 5368555
       Characters read: 10201345
------------------------------------------------------------------------
-----------------------------

Thank you again, Best regards,
Dehong




________________________________________
From: Mark Rivers [rivers@cars.uchicago.edu]
Sent: Friday, July 08, 2011 8:28 PM
To: Zhang, Dehong; dirk.zimoch@psi.ch; tech-talk@aps.anl.gov
Subject: RE: streamdevice + asyn stuck

Hi Dehong,

We need more information:

- What is the .SCAN field of the record: it is periodic, I/O Intr,
Passive, etc?

- What type of device are you talking to: serial, TCP, etc?

canBlock=Yes is normal.  It means that this is a "slow" device, and so
it uses asynchronous device support.

numberConnects is the number of times the device has been connected to.
It would normally be 1.  If it is 2, that suggests the device has
disconnected, and then reconnects.

When it happens please send the complete output of "dbpr <record> 10"

Also send the complete output of "asynReport 10 <portName>"

Mark


________________________________

From: Zhang, Dehong [mailto:dhzhang@slac.stanford.edu]
Sent: Fri 7/8/2011 7:19 PM
To: Mark Rivers; dirk.zimoch@psi.ch; tech-talk@aps.anl.gov
Subject: streamdevice + asyn stuck



Hi Dirk, Mark and fellow EPICSers,

Sorry to bother you with this.  I tried to chase the source code and
search the tech-talk,
but could not find much hints.

We use base 3.14.9, streamdevice 2.4 and asyn 4.10, have been
experiencing this stuck
problem for some time.  The symptom is that randomly, a mbbi/ai/bi...
(input) record would
stop updating and no longer does FLNK, with no error/warning messages
printed in the log
file.  And dbpr *** 3 would show that:
LCNT=11
SEVR=INVALID
STAT=SCAN

And manually writing to the PROC field also does nothing.

It seems like the record is stuck waiting for a callback from
streamdevice (and asyn), and
the EPICS framework just ignores any "PROC" requests.

In the protocol file we do have the timeouts etc:
locktimeout  = 5000;
terminator   = CR;
replytimeout = 1000;
readtimeout  = 1000;
extrainput   = Ignore;

While chasing the problem, I noticed that "asynReport 10 ***" would
show:
multiDevice:No canBlock:Yes autoConnect:Yes
    enabled:Yes connected:Yes numberConnects 1

For some the numberConnects can be 2.

Is this "canBlock:Yes" related to my problem?  Should the numberConnects
be <=1?
We use one port to talk to one device.

Rebooting the IOC does fix the problems, but then they would come back,
randomly.

Please advise.  Thank you very much.

Best regards,
Dehong