Re: flaky IOC problems at Jefferson Lab

[Rolf Keitel <rolf@Triumf.CA>]

watson@cebaf.gov wrote:
> 

> Many of you are aware that Jefferson Lab (fna CEBAF) has had problems with
> IOC's mysteriously crashing, and in fact having multiple IOC's crash because
> 1 crashed. 
> ....
> Symptoms of the problem:
>         1) one IOC is in a funny state; screens which are up continue
>            to work for signals NOT on that IOC
>         2) MEDM or DM session which attempts to connect to ANY signal on
>            that IOC will become COMPLETELY UNUSABLE, and will in fact not
>            be able to connect to ANY signal on ANY ioc; existing screens
>            continue to function for signals on other IOC's.
>
Boy, am I happy that Chip describes exactly the symptoms I have been 
observing on our ion source test-stand over the last six months. 
> Further investigation reveals:
>         1) a scan task is consuming all available CPU resources
> 
> Analyis:
>         1) the CAMAC serial highway is probably at fault, and the error
> ....
My analysis is not as detailed as we are (relatively) new to EPICS, I am
the only one supporting our system at the moment, and my "spare time"
was
very limited. But ...
a) I run only one VME crate and, like Chip, traced the problem to CAMAC
   (by isolating the CAMAC sub-system in a second CPU). The problem
shows
   both with a serial highway (HYTEC 2992 controller) and a SCSI CAMAC
   controller (Jorway J73, we used the LANL CAMAC driver software and 
   modified it for the Jorway).
b) I seem to recall running spy, but not seeing anything abnormal, but
   this may be just due to inexperience, being frustrated, etc.
c) We run name resolution at default priority 

We run R3.12.2.

					- rolf -
-- 
+----------------------------------------------------------+
: Rolf Keitel                |    rolf@triumf.ca           :
: TRIUMF                     |    Tel: (604) 222-7453      :
: 4004 Wesbrook Mall         |    Fax: (604) 222-7307      :
: Vancouver, B.C., V6T 2A3   |    www.triumf.ca            :
+----------------------------------------------------------+