Argonne National Laboratory

Experimental Physics and
Industrial Control System

2002  2003  2004  2005  2006  2007  2008  2009  <20102011  2012  2013  2014  2015  2016  2017  2018  2019  Index 2002  2003  2004  2005  2006  2007  2008  2009  <20102011  2012  2013  2014  2015  2016  2017  2018  2019 
<== Date ==> <== Thread ==>

Subject: vxWorks 6.7 tNet0 task crashed with fage fault: help needed from vxWorks guru
From: Matthias Clausen <Matthias.Clausen@desy.de>
To: EPICS Core Talk <core-talk@aps.anl.gov>
Cc: Siegfried Rettig <siegfried.rettig-labusga@desy.de>
Date: Thu, 30 Sep 2010 10:38:42 +0200
 Hi all.

Since the beginning of this year we have random crashes of the tNet task in one of our new IOCs.
We do not get any help from Wind River.
We are going to get a WRS specialist to DESY to help us set up the IOC to enable us tracing back the problem when the next crash occurs. I have my doubts that this will happen soon and that we can set it up with our IOCs where the shell is still running locally ... we'll see.

Meanwhile I hope to get some ideas from you.

Since the Net task is down we can only use the local console with VGS output.
Therefore I have real screen shots taken with my camera.

You can find the page fault info and the tt output in the two jpg files attached.
(also the lkAddr and the checkStack output)
There's more information about the individual memory locations but I do not want to fill your mailbox with my garbage. In case you might have an idea an more debug information would help - let me know there's more information available.

For now we have no clue what could be the root cause.
We have two IOCs running with the same vxWorks image one of them is more heavily loaded then the other - where heavy means 20% CPU instead of 15%.
The two CPUs are compact PCI CPUs from Kontron in Germany
vxWorks image based on version 6.7

Craches happen at intervals between a week and two months.

Our plans:
- Get a WRS specialist to help us set up the IOC to catch the root cause and trace it back
- Install a PC with wireshark and monitor the traffic to disk
- analyze the Ethernet traffic and filter out any -non IP- traffic to the IOC by setting up filters in the Cisco router before the IOC

Any other idea?

Of course any help is highly appreciated.

Cheers
Matthias

Attachment: tNet0-TaskTrace.JPG
Description: JPEG image

Attachment: tNet0-PageFault.JPG
Description: JPEG image

Attachment: lkAddr-0x0039d09e.JPG
Description: JPEG image

Attachment: checkStack-0xdef0af0.JPG
Description: JPEG image


Navigate by Date:
Prev: Re: alarm acknowledge Matthias Clausen
Next: [Merge] lp:~anj/epics-base/expand-calc-size into lp:epics-base Andrew Johnson
Index: 2002  2003  2004  2005  2006  2007  2008  2009  <20102011  2012  2013  2014  2015  2016  2017  2018  2019 
Navigate by Thread:
Prev: Re: alarm acknowledge Matthias Clausen
Next: [Merge] lp:~anj/epics-base/expand-calc-size into lp:epics-base Andrew Johnson
Index: 2002  2003  2004  2005  2006  2007  2008  2009  <20102011  2012  2013  2014  2015  2016  2017  2018  2019 
ANJ, 02 Feb 2012 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·