Experimental Physics and
Industrial Control System

[email protected] · Thu, 17 Feb 2000 17:08:37 -0800 (PST)

Dear EPICS experts,

We have an IOC (mv177) here at BaBar that has recently started suffering random
crashes caused by the tNetTask process taking up all the cpu cycles.  The
symptom is that all channels served by that IOC become disconnected.  While the
IOC is in this state I cannot telnet to it, but I can login via xyplex through
the serial port.  Running spy I then see that the tNetTask is hogging cpu
time.  I am also able to do "casr" and look at the connected clients, but
without a process id it is difficult to track down any correlation with a
specific client application.  Rebooting seems to be the only solution to this
problem.  The crash rate is around once per week.

This particular cpu is required for BaBar/PEP-II operation and is normally
rock solid.  A few months ago we migrated to EPICS version 3.13.1, but that
was well before these mysterious crashes began.  All of our other 14 IOCs are
running the same version and do not suffer this problem.  The only relevant
difference is that the problem IOC is the only one that shares a subnet with
the PEP-II IOC (ie, sees additional network traffic).  However, we do not think 
the problem is related to the network since rebooting (over nfs) always gets us 
out of the crashed state.

The IOC statistics before the last crash were:

cpu load: 45%
ca clients: 65
ca connections: ~1500
free memory: 5MB

These numbers are typical steady-state values.  So there does not seem to be a
smoking gun event that caused the crash.

Can anyone out there shed some light on what may be going on here?  Are there 
any other diagnostic tools I can use to further evaluate the problem?  Comments
and suggestions are greatly appreciated.

Thank you,
Jim Olsen
[email protected]

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System