Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  <20032004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  <20032004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020 
<== Date ==> <== Thread ==>

Subject: RE: Problem with seq 2.0.4
From: "Mark Rivers" <rivers@cars.uchicago.edu>
To: "Jeff Hill" <johill@lanl.gov>, <tech-talk@aps.anl.gov>
Date: Tue, 2 Sep 2003 20:33:12 -0500
Jeff,

> The database links are well tested in this mode of operation, 
> but perhaps there is a race condition in the sequencer which is influenced by the
> improved performance of CA in R3.14. There might also be a 
> problem in the R3.14 implementation of the CA client library. 

I am now quite convinced that the problem is CA related, and probably related to the fact that 2 of the 26 PVs I am connecting 2 are in another IOC.  The other 24 are local.  My other seq programs talk only to local PVs, and they work fine.

When the seq program fails to connect to some channels I see the following:
#####
seq &BM13_Energy, "E=13BMA:E, MONO=13BMA:m17, EXPTAB_Z=13BMD:m22, YXTAL=13BMA:MON:, ZXTAL=13BMA:m14" 
@(#)SEQ Version 2.0.5: Tue Sep 2 15:52:2 2 CDT 20030x1238a08:
Spawning state program "BM13_Energy", thread 0x11ae730: "BM13_Energy"
value = 18540336 = 0x11ae730
Done executing startup script /home/epics/3.14/CARS/iocBoot/ioc13bma/st.cmd
ioc13bma> 24 of 26 assigned channels have connected
DB CA Exception: "Virtual circuit disconnect", context "164.54.160.75:5064"
CA.Client.Exception...............................................
    Warning: "Virtual circuit disconnect"
    Context: "164.54.160.95:5064"
    Source File: ../cac.cpp line 1537
    Current Time: TUE SEP 02 2003 20:18:09.811156578
....................................................................
CA.Client.Exception...............................................
    Warning: "Virtual circuit disconnect"
    Context: "164.54.160.95:5064"
    Source File: ../cac.cpp line 1537
    Current Time: TUE SEP 02 2003 20:18:10.127823232
....................................................................
CA.Client.Exception...............................................
    Warning: "Virtual circuit disconnect"
    Context: "164.54.160.95:5064"
    Source File: ../cac.cpp line 1537
    Current Time: TUE SEP 02 2003 20:18:13.461156432
....................................................................
ioc13bma> 24 of 26 assigned channels have connected
####

When the seq program initializes OK (which is rarely) I don't see these CA.Client.Exception messages.  One puzzling thing is that the number of unconnected channels reported by seq is not 2 (as it was in this particular test, and as one might expect), but some integer times 2, so the number of connected channels is 16, 18, 20, 22, 24 or 26.

Is the above error message indicative of a problem with CA itself, or could it be a problem with the seq client?

The remote IOC is running EPICS 3.13.7.  It is healthy, lots of memory, no CPU load, etc.  Here are the network related commands executed by the startup script, in case I am doing something wrong:
###############################
# Set the default gateway
routeAdd "0","164.54.160.1"
# The following line eliminates "cksum: out of data" messages due to DHCP
proxyPortFwdOff(67)
# Define locations for logging and for APS EPICS process variables
putenv("EPICS_IOC_LOG_INET=164.54.160.82")
putenv("EPICS_CA_AUTO_ADDR_LIST=NO")
putenv("EPICS_CA_ADDR_LIST=164.54.160.127")
##############################

Note that even in EPICS 3.13 we have always had problems with PVs in remote IOCs taking a long time (order of one minute) to connect to seq programs after iocInit.  We also see similar error messages to those above, but the PVs always eventually did connect.  In 3.14 they never connect.  Andrew Johnson told me today that they are seeing similar things on 3.13.  I've been meaning to report this/track it down for quite a while!

Mark


Navigate by Date:
Prev: RE: Problem with seq 2.0.4 Jeff Hill
Next: RE: Problem with seq 2.0.4 Carl Lionberger
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  <20032004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020 
Navigate by Thread:
Prev: RE: Problem with seq 2.0.4 Jeff Hill
Next: RE: Problem with seq 2.0.4 Carl Lionberger
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  <20032004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020 
ANJ, 10 Aug 2010 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·