RE: Buffer problems

["Kevin M. Peterson" <kmpeters@anl.gov>]

Three of UNICAT's IOCs hung at the same time yesterday.  The three crates 
had been
idle (nothing had been written to the log files for at least 5 hours).  The 
three log files
show that the three IOCs (LGA, PIE, and FAT) experience the same error:

LGA:    2004-04-05,16:07:12: 0x760408 (tNetTask): ei driver: command field 
frozen
PIE:    2004-04-05,16:07:12: 0x760408 (tNetTask): ei driver: command field 
frozen
FAT:    2004-04-05,16:07:28: 0xf60408 (tNetTask): ei driver: command field 
frozen

Each of the log files shows that the ENOBUFS and save/restore errors 
occurred 1-3 minutes
later and continued until each IOC was rebooted.  Here are the errors that 
occurred:

CAC: unexpected select fail: 851971=S_iosLib_INVALID_FILE_DESCRIPTOR
0x2174b4 (save_restore): save_file: Backup file (auto_sr/settings.savB) is 
bad.  Writing a new one.
0x32d250 (CA_online): ../online_notify.c: CA beacon error was "S_errno_ENOBUFS"
CAC: error = "S_errno_ENOBUFS" sending UDP msg to 164.54.216.255:5064
0x2174b4 (save_restore): save_file - unable to open file auto_sr/settings.savB
0x2174b4 (save_restore): *** *** *** *** *** *** *** *** *** *** *** *** 
*** *** *** ***
0x2174b4 (save_restore): save_restore:save_file: Can't write new backup 
file.  I quit.
0x2174b4 (save_restore): *** *** *** *** *** *** *** *** *** *** *** *** 
*** *** *** ***
sendto failed: S_errno_ENOBUFS
CAC: error = "S_errno_ENOBUFS" sending UDP msg to 164.54.216.255:5064
sendto failed: S_errno_ENOBUFS

We have seen the ENOBUFS errors occur in IOCs before without seeing the 
tNetTask error,
however, in those cases only one IOC was affected.

This is the first time we have seen both the tNetTask error and multiple 
IOCs have ENOBUFS
problems simultaneously.

Are these "command field frozen" errors occuring anywhere else?


Kevin Peterson


At 12:42 PM 3/29/2004 -0600, Mark Rivers wrote:
> Hi Ernest,
>
> Thanks for the reply.
>
> I really don't think it's anything with CA.  These IOCs were running
> fine for a long time, years in some cases.  Suddenly last week they all
> hung up at the same time.  I am able to completely reproduce the problem
> by simply sending 2000 1500-byte ping packets to them as fast as
> possible.  So I am quite sure that some computer on the network is doing
> the equivalent of that.  It is happening every few minutes or hours. The
> problem is that the NSLS is one giant subnet with 512 addresses.  If we
> disconnnect the switches at the beamline from the NSLS subnet then
> things are fine, so it's not a problem with the local CA server and
> clients.  They need to get a sniffer on the network and find who the
> culprit is.  More importantly, Wind River needs to fix their network
> software so that sending a 2000 1500-byte packets does not crash the
> network.
>
> Mark