Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  <20122013  2014  2015  2016  2017  2018  2019  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  <20122013  2014  2015  2016  2017  2018  2019 
<== Date ==> <== Thread ==>

Subject: Re: Runaway connection count on IOC
From: Andrew Johnson <anj@aps.anl.gov>
To: tech-talk@aps.anl.gov
Date: Mon, 11 Jun 2012 09:41:55 -0500
Hi Michael,

On 2012-06-11 michael.abbott@diamond.ac.uk wrote:
> 
> 1. The server is vxWorks 5.5.1 and EPICS 3.14.11

> 3. The ethernet connection is horribly horribly overloaded (100MBit link),
>  EPICS would like to deliver far more image frames than the network will
>  permit

> 5. At some random point during operation the number of IOC connections
>  ($(IOC):CA:CNX as reported by vxStats) starts climbing steadily.

What is the CPU load on the IOC at this time?  Does the CPU have any idle time 
at all?  Can you run Spy to find out which threads are getting any work done?

Also, are you running a name-server for the gateway, or is it relying on UDP 
broadcasts for name resolution?

> 7. Running `casr 2` takes forever (well, several minutes), the connection
>  is over a 9600 baud serial line.  Of the 8697 channels, the same PV is
>  reported over and over and over and over again (it's a simple camera
>  STATUS provided by the asyn driver).
> 
> 8. After `casr 2` has completed, the bogus channels have gone away:

> Very odd.  Any thoughts?

Your 'casr 2' is being run at a much higher priority than the other Channel 
Access activities.  I suspect your CPU is too busy handling higher priority 
operations to spend any time actually running the low-level CA threads that 
communicate with client.

UDP name lookups normally happen in the lowest priority CA thread, so the IOC 
/should/ stop accepting new client connections if the CPU gets overloaded 
because the name resolver task never gets a chance to run.  However if the 
gateway uses a name-server instead of UDP broadcasts this natural throttling 
will not be happening and I could see your symptoms occurring as a result.

Does this help explain what might be going on?

- Andrew
-- 
Never interrupt your enemy when he is making a mistake.
-- Napoleon Bonaparte

Replies:
Re: Runaway connection count on IOC Andrew Johnson
RE: Runaway connection count on IOC -- Possible Gateway Issue? michael.abbott
References:
Runaway connection count on IOC michael.abbott

Navigate by Date:
Prev: Re: EPICS host architectures Benjamin Franksen
Next: Re: Runaway connection count on IOC Andrew Johnson
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  <20122013  2014  2015  2016  2017  2018  2019 
Navigate by Thread:
Prev: Runaway connection count on IOC michael.abbott
Next: Re: Runaway connection count on IOC Andrew Johnson
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  <20122013  2014  2015  2016  2017  2018  2019 
ANJ, 18 Nov 2013 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·