RE: vxWorks 6.7 tNet0 task crashed with fage fault: help needed from vxWorks guru

["Jeff Hill" <johill@lanl.gov>]

Hi Mathias,

Some possible ideas.

1) First and foremost, make certain that the latest vxWorks driver patches
for your particular network interface chip are installed. 

2) Also look for vxWorks 6 patches which could be installed.

3) Run ifShow (and switch diagnostics) to see if there are unusual link
level errors or problems where the switch and network interface haven?t auto
negotiated full / half duplex correctly (consistently).

4) Run netstackSysPoolShow netstackDataPoolShow to see if your system is low
on network buffers. The network stack shouldn?t crash when it runs low on
buffers, but nevertheless adding more buffers might be a temporary
workaround. I understand that there is also a driver buffer pool that can be
adjusted on vxWorks.


Jeff
______________________________________________________
Jeffrey O. Hill           Email        johill@lanl.gov
LANL MS H820              Voice        505 665 1831
Los Alamos NM 87545 USA   FAX          505 665 5107

Message content: TSPA


> -----Original Message-----
> From: core-talk-bounces@aps.anl.gov [mailto:core-talk-
> bounces@aps.anl.gov] On Behalf Of Matthias Clausen
> Sent: Thursday, September 30, 2010 2:39 AM
> To: EPICS Core Talk
> Cc: Siegfried Rettig
> Subject: vxWorks 6.7 tNet0 task crashed with fage fault: help needed
> from vxWorks guru
> 
>   Hi all.
> 
> Since the beginning of this year we have random crashes of the tNet
> task
> in one of our new IOCs.
> We do not get any help from Wind River.
> We are going to get a WRS specialist to DESY to help us set up the IOC
> to enable us tracing back the problem when the next crash occurs. I
> have
> my doubts that this will happen soon and that we can set it up with our
> IOCs where the shell is still running locally ... we'll see.
> 
> Meanwhile I hope to get some ideas from you.
> 
> Since the Net task is down we can only use the local console with VGS
> output.
> Therefore I have real screen shots taken with my camera.
> 
> You can find the page fault info and the tt output in the two jpg files
> attached.
> (also the lkAddr and the checkStack output)
> There's more information about the individual memory locations but I do
> not want to fill your mailbox with my garbage. In case you might have
> an
> idea an more debug information would help - let me know there's more
> information available.
> 
> For now we have no clue what could be the root cause.
> We have two IOCs running with the same vxWorks image one of them is
> more
> heavily loaded then the other - where heavy means 20% CPU instead of
> 15%.
> The two CPUs are compact PCI CPUs from Kontron in Germany
> vxWorks image based on version 6.7
> 
> Craches happen at intervals between a week and two months.
> 
> Our plans:
> - Get a WRS specialist to help us set up the IOC to catch the root
> cause
> and trace it back
> - Install a PC with wireshark and monitor the traffic to disk
> - analyze the Ethernet traffic and filter out any -non IP- traffic to
> the IOC by setting up filters in the Cisco router before the IOC
> 
> Any other idea?
> 
> Of course any help is highly appreciated.
> 
> Cheers
> Matthias