RE: streamdevice + asyn stuck

["Zhang, Dehong" <dhzhang@slac.stanford.edu>]

Hi Mark,

Thank you very much for spending time on this!

The SCAN field is "Event".  I set up an event dispatcher to send out a few
different events at certain rates.  Those records listen to the events.

Those records talk to MKS937A controlers via digi port servers with TCP.

When the problem happens, the PACT field is "1", and the record is stuck
forever.  But, the port is still active, other records can still use it.

Please help to correct my setup/configuration, so that when anything bad
happens, the record can recover in the next cycle. ( -- could it be input
mismatch?)

Here is one example:

-----------------------------------------------------------------------------------------------------
record(mbbi, "TST:DSU:GCC:01:STATE") {
  field(DTYP, "stream")
  field(INP,  "@mks937a.proto.db state(2) TST:R54:GCT:16:B 0")
  field(DESC, "Slot=A1")
  field(SCAN, "Event")
  field(EVNT, "1")
  field(PHAS, "2")
  ...
  field(FLNK, "TST:DSU:GCC:01:STATUSCALC")
}

-----------------------------------------------------------------------------------------------------
part of the protocol file:
locktimeout  = 5000;
terminator   = CR;
replytimeout = 1000;
readtimeout  = 1000;
extrainput   = Ignore;

@init{out " ";}

state    { out "R\$1"; in "%{\?.|  \?E|HI|A|Lo|F|H|W|L|CON|P|NOG|M|NOT|Not|C}"; }

-----------------------------------------------------------------------------------------------------
asyn configure:
drvAsynIPPortConfigure("TST:R54:GCT:16:B", "TST-R59-SER-04:2104 TCP",0,0,0)

-----------------------------------------------------------------------------------------------------
ioc-tst-vacuum> dbpr TST:DSU:GCC:01:STATE 10
ACKS: INVALID       ACKT: YES           ASG:                
ASP: 0x0000000000000000                 BKPT: 0x00          COSV: NO_ALARM      
DESC: Slot=A1       DISA: 0             DISP: 0             DISS: NO_ALARM      
DISV: 1             DPVT: 0x10c5fa0c00000000                
DSET: 0x40c0df18db2a0000                DTYP: stream        EIST: Below Range   
EISV: NO_ALARM      EIVL: 0x0           ELST: No Gauge      ELSV: MAJOR         
ELVL: 0x0           EVNT: 1             FFST: Locked Out    FFSV: MAJOR         
FFVL: 0x0           FLNK:DB_LINK TST:DSU:GCC:01:STATUSCALC  FRST: Low Emission  
FRSV: MAJOR         FRVL: 0x0           FTST: Bad Command   FTSV: MAJOR         
FTVL: 0x0           FVST: Filament Off  FVSV: MAJOR         FVVL: 0x0           
INP:INST_IO @mks937a.proto.db state(2) TST:R54:GCT:16:B 0   LALM: 0             
LCNT: 11            LSET: 0x50ee070d00000000                MASK: 0             
MLIS: 0x0000000000000000000000000000000000000000            
MLOK: 0xc0d7bf0c00000000                MLST: 0             
NAME: XXX:DSU:GCC:01:STATE              NIST: Controlled    NISV: MINOR         
NIVL: 0x0           NOBT: 0             NSEV: NO_ALARM      NSTA: NO_ALARM      
ONST: ON            ONSV: NO_ALARM      ONVL: 0x0           ORAW: 0             
PACT: 1             PHAS: 2             PINI: NO            
PPN: 0x0000000000000000                 PPNR: 0x0000000000000000                
PRIO: LOW           PROC: 0             PUTF: 0             
RDES: 0x6084830c00000000                RPRO: 0             
RSET: 0xc02f2219db2a0000                RVAL: 0             SCAN: Event         
SDEF: 1             SDIS:CONSTANT       SEVR: INVALID       SHFT: 0             
SIML:CONSTANT       SIMM: NO            SIMS: NO_ALARM      SIOL:CONSTANT       
SPVT: 0x6057100d00000000                STAT: SCAN          SVAL: 0             
SVST: Startup Delay SVSV: MINOR         SVVL: 0x0           SXST: HV Off        
SXSV: MAJOR         SXVL: 0x0           TEST: Protected State                   
TESV: MINOR         TEVL: 0x0           THST: At Atmosphere THSV: MINOR         
THVL: 0x0           TIME: 2011-07-10 00:55:28.506299000     TPRO: 0             
TSE: 0              TSEL:CONSTANT       TTST: Wrong Gauge   TTSV: MAJOR         
TTVL: 0x0           TVST: Not Connected TVSV: MAJOR         TVVL: 0x0           
TWST: Above Range   TWSV: MINOR         TWVL: 0x0           UDF: 0              
UNSV: MAJOR         VAL: 0              ZRST: ON            ZRSV: NO_ALARM      
ZRVL: 0x0           

-----------------------------------------------------------------------------------------------------
ioc-tst-vacuum> asynReport 10 TST:R54:GCT:16:B
TST:R54:GCT:16:B multiDevice:No canBlock:Yes autoConnect:Yes
    enabled:Yes connected:Yes numberConnects 1
    nDevices 0 nQueued 0 blocked:No
    asynManagerLock:No synchronousLock:No
    exceptionActive:No exceptionUsers 1 exceptionNotifys 0
    interposeInterfaceList
        asynOctet pinterface 0x2adb18bbb1e0 drvPvt 0xc8a2b70
    interfaceList
        asynCommon pinterface 0x2adb18bb9100 drvPvt 0xc8a2070
        asynOctet pinterface 0xc8a20e8 drvPvt 0xc8a2070
    Port TST-R59-SER-04:2104 TCP: Connected
                    fd: 13
    Characters written: 5368555
       Characters read: 10201345
-----------------------------------------------------------------------------------------------------

Thank you again, Best regards,
Dehong




________________________________________
From: Mark Rivers [rivers@cars.uchicago.edu]
Sent: Friday, July 08, 2011 8:28 PM
To: Zhang, Dehong; dirk.zimoch@psi.ch; tech-talk@aps.anl.gov
Subject: RE: streamdevice + asyn stuck

Hi Dehong,

We need more information:

- What is the .SCAN field of the record: it is periodic, I/O Intr, Passive, etc?

- What type of device are you talking to: serial, TCP, etc?

canBlock=Yes is normal.  It means that this is a "slow" device, and so it uses asynchronous device support.

numberConnects is the number of times the device has been connected to.  It would normally be 1.  If it is 2, that suggests the device has disconnected, and then reconnects.

When it happens please send the complete output of "dbpr <record> 10"

Also send the complete output of "asynReport 10 <portName>"

Mark


________________________________

From: Zhang, Dehong [mailto:dhzhang@slac.stanford.edu]
Sent: Fri 7/8/2011 7:19 PM
To: Mark Rivers; dirk.zimoch@psi.ch; tech-talk@aps.anl.gov
Subject: streamdevice + asyn stuck



Hi Dirk, Mark and fellow EPICSers,

Sorry to bother you with this.  I tried to chase the source code and search the tech-talk,
but could not find much hints.

We use base 3.14.9, streamdevice 2.4 and asyn 4.10, have been experiencing this stuck
problem for some time.  The symptom is that randomly, a mbbi/ai/bi... (input) record would
stop updating and no longer does FLNK, with no error/warning messages printed in the log
file.  And dbpr *** 3 would show that:
LCNT=11
SEVR=INVALID
STAT=SCAN

And manually writing to the PROC field also does nothing.

It seems like the record is stuck waiting for a callback from streamdevice (and asyn), and
the EPICS framework just ignores any "PROC" requests.

In the protocol file we do have the timeouts etc:
locktimeout  = 5000;
terminator   = CR;
replytimeout = 1000;
readtimeout  = 1000;
extrainput   = Ignore;

While chasing the problem, I noticed that "asynReport 10 ***" would show:
multiDevice:No canBlock:Yes autoConnect:Yes
    enabled:Yes connected:Yes numberConnects 1

For some the numberConnects can be 2.

Is this "canBlock:Yes" related to my problem?  Should the numberConnects be <=1?
We use one port to talk to one device.

Rebooting the IOC does fix the problems, but then they would come back, randomly.

Please advise.  Thank you very much.

Best regards,
Dehong