RE: Very slow reconnection to medm after IOC reboot

["Jeff Hill" <johill@lanl.gov>]

Mark,

How many channels were involved in this disconnect / reconnect scenario?

> Stop IOC, wait 10 seconds, restart IOC.
> Time for medm to reconnect all channels: less than 10 seconds.

> Stop IOC, wait 15 minutes, restart IOC.
> Time for medm to reconnect all channels: 4 minutes

When the CA library disconnects from the IOC it starts looking for the
consequently disconnected channels with a fast search interval, but if the
library isn't immediately successful it exponentially backs off to a much
lower rate. We want that behavior in order to reduce the load on the
network.

If, after 15 minutes, the IOC comes up and the CA client library manages to
find some of its disconnected channels it has no way of knowing that the
balance of its remaining disconnected channels are on this new IOC. Consider
a worst case scenario; what if this particular CA client had a very large
number of disconnected channels. From my perspective, it wouldn't be a good
idea to search at the highest rate for these channels just because a new IOC
was seen. The client library does however adopt a compromise design where it
searches at an intermediate rate if it sees a beacon anomaly, and depending
on how many channels are disconnected this intermediate rate might result in
a 4 minute reconnect time. Furthermore, I think I recall correctly that the
event of connecting to a new IOC will also cause this same intermediate
search rate boost that is granted when a beacon anomaly event is detected.

Part of the rationale behind the current design is that if it was necessary
to wait a long time for the IOC to come up then waiting as long as the
number of channels divided by the number of channels per search attempt
times this intermediate delay (maybe as much as 4 minutes in this situation)
for the channels to reconnect isn't going to shatter our world. It is
particularly important to avoid designs which cause unstable traffic
insertion feedback when there is network congestion.

Jeff


> -----Original Message-----
> From: Mark Rivers [mailto:rivers@cars.uchicago.edu]
> Sent: Thursday, January 22, 2009 3:28 PM
> To: Andrew Johnson
> Cc: Jeff Hill; epics
> Subject: RE: Very slow reconnection to medm after IOC reboot
> 
> Hi Andrew,
> 
> Thanks for the response.
> 
> Here is the output of all env settings that start with the string EPICS,
> before I manually set EPICS_CAS_BEACON_ADDR_LIST:
> 
> $ printenv | grep EPICS
> EPICS_CA_AUTO_ADDR_LIST=NO
> EPICS_HOST_ARCH=win32-x86
> EPICS_CA_ADDR_LIST=localhost 164.54.160.255
> EPICS_DISPLAY_PATH=C:\EPICS\adls\
> EPICS_CA_MAX_ARRAY_BYTES=10000000
> 
> If I run with just those env settings I get the following message when I
> boot the IOC:
> 
> The CA server's beacon address list was empty after initialization?
> 
> If I add the env you suggested so they now look like this:
> 
> $ printenv | grep EPICS
> EPICS_CA_AUTO_ADDR_LIST=NO
> EPICS_CAS_BEACON_ADDR_LIST=164.54.160.255
> EPICS_HOST_ARCH=win32-x86
> EPICS_CA_ADDR_LIST=localhost 164.54.160.255
> EPICS_DISPLAY_PATH=C:\EPICS\adls\
> EPICS_CA_MAX_ARRAY_BYTES=10000000
> 
> Then when I start the IOC I don't get the message about the empty beacon
> address list.
> 
> This is a new problem with 3.14.10.  I run identical IOCs built with
> 3.14.8.2 and I don't get that error message with exactly the same
> environment settings.
> 
> However, even though the error message is gone the performance is only
> marginally improved, if at all.
> 
> Here is the result with EPICS_CAS_BEACON_ADDR_LIST set:
> 
> ******************************
> Windows IOC (3.14.10), Windows medm client (built with 3.14.9) on same
> PC.
> 
> Stop IOC, wait 10 seconds, restart IOC.
> Time for medm to reconnect all channels: less than 10 seconds.
> 
> Stop IOC, wait 15 minutes, restart IOC.
> Time for medm to reconnect first channel: 10 seconds
> Time for medm to reconnect all channels: 4 minutes
> 
> 
> The differences from my previous test (when I had the CA server beacon
> address error):
> 
> The time for the very first channel to reconnect was reduced from 60
> seconds to 10 seconds.  I am not sure how reproducible this is.
> 
> The time for all channels to connect was reduced from 4 minutes and 25
> seconds to 4 minutes.  Again, this may not be a statistically
> significant improvement.
> 
> Thus, even with EPICS_CAS_BEACON_ADDR_LIST set the performance is really
> bad.  And I am not the only one seeing it, Lewis Muir reported similar
> long reconnection times earlier this afternoon.
> 
> Mark
> 
> 
> 
> -----Original Message-----
> From: Andrew Johnson [mailto:anj@aps.anl.gov]
> Sent: Thursday, January 22, 2009 3:47 PM
> To: Mark Rivers
> Cc: Jeff Hill; epics
> Subject: Re: Very slow reconnection to medm after IOC reboot
> 
> On Thursday 22 January 2009 15:32:47 Mark Rivers wrote:
> > I noticed that when I start win32-x86 IOCs built with 3.14.10 I get
> the
> > following message just after iocInit:
> >
> > *******
> > The CA server's beacon address list was empty after initialization?
> > *******
> >
> > I do not get this message when running the identical IOC built with
> > 3.14.8.2.
> >
> > Is this significant?  Why am I getting this message?
> 
> Highly, it means that the IOC is not sending out any CA beacons at all,
> which
> explains why the clients are not reconnecting to it immediately.  Try
> setting
> EPICS_CAS_BEACON_ADDR_LIST on the IOC to your 164.54.160.255 broadcast
> address, although it should default to the value of EPICS_CA_ADDR_LIST
> if not
> set so I don't understand why it's ignoring that (unless you have other
> env
> settings for the IOC that you haven't told us about).
> 
> - Andrew
> --
> The best FOSS code is written to be read by other humans -- Harold Welte