Very slow reconnection to medm after IOC reboot

["Mark Rivers" <rivers@cars.uchicago.edu>]

Folks,

I have noticed a problem which seems new, though I can't say exactly
when it started.

The time for EPICS CA clients to reconnect when an IOC reboots is quite
long.

Here are some specifics.  I am running a soft IOC on Linux or Windows
and running medm on Linux or Windows.  There are 4 medm screens opened
to the IOC with about 200 PVs total.  There are 2 computers under test:
a recent Redhat Linux machine and a Windows XP machine.  They are on the
same subnet.  caRepeater is running on both machines.

******************************
Linux IOC (3.14.10), Linux medm client (built with 3.14.8.2) on the same
machine.

Stop IOC, wait 10 seconds, restart IOC.  
Time for medm to reconnect all channels: less than 10 seconds.

Stop IOC, wait 8 minutes, restart IOC.  
Time for medm to reconnect first channel: 40 seconds
Time for medm to reconnect all channels: 60 seconds

******************************
Windows IOC (3.14.10), same Linux medm client (built with 3.14.8.2)

Stop IOC, wait 10 seconds, restart IOC.  
Time for medm to reconnect all channels: less than 10 seconds.

Stop IOC, wait 15 minutes, restart IOC.
After 3 minutes no channels had reconnected.  Open a new medm screen to
the same IOC.  It immediately connects all PVs on the new screen, but
the PVs on the previous screens are still not connected.
Time for medm to reconnect all channels: 4 minutes (finally after 4
minutes all PVs on the previous screens reconnected.

******************************
Windows IOC (3.14.10), Windows medm client (built with 3.14.9) on same
PC.

Stop IOC, wait 10 seconds, restart IOC.  
Time for medm to reconnect all channels: less than 10 seconds.

Stop IOC, wait 15 minutes, restart IOC.
Time for medm to reconnect first channel: 60 seconds
Time for medm to reconnect all channels: 4 minutes and 25 seconds.

******************************

This definitely seems like a new problem to me, it never used to take 4
minutes to reconnect.  Unfortunately I have recently upgraded both medm
(from a much older version, to the newer one built with 3.14.8.2) and
the IOCs (from 3.14.8.2 to 3.14.10).

On the Windows PC I have the following EPICS environment variables
defined:
EPICS_CA_AUTO_ADDR_LIST=NO
EPICS_HOST_ARCH=win32-x86
EPICS_CA_ADDR_LIST=localhost 164.54.160.255
EPICS_DISPLAY_PATH=C:\EPICS\adls\
EPICS_CA_MAX_ARRAY_BYTES=10000000

On the Linux machine I have the following EPICS environment variables
defined:
EPICS_HOST_ARCH=linux-x86
EPICS_DISPLAY_PATH=/home/epics/adl/all
EPICS_CA_MAX_ARRAY_BYTES=10000000

These delays make IOC development very painful, particularly on Windows.
Each time I restart the IOC if I wait more than 30 seconds or so it then
takes 4 minutes for the medm screens to refresh.  It becomes quicker to
close them all and re-open them, though this is very tedious.

Have others seen similar problems?  Any idea when they started?  Do they
go away if clients are built with 3.14.10 which I have not tried (but I
have used 3.14.9)

Thanks,
Mark