Experimental Physics and
Industrial Control System

Michael Davidsaver via Core-talk <core-talk at aps.anl.gov> · Sat, 27 Aug 2022 09:53:45 -0700

On 8/26/22 06:05, Zimoch Dirk (PSI) via Core-talk wrote:

Hi fellow VxWorks users,

Since I migrated most of our IOCs from VxWorks 5.5 to VxWorks 6.9, I notices a severe network driver problem.
After running fine for weeks or months, suddenly the tNet0 tasks consumed 100% CPU and stays so, which of course makes
the IOC unusable. Only reboot helps. This happens on multiple IOCs but not synchronously.

Are there any commonalities in terms of application code / epics drivers?

Any other changes besides the OS version?  (eg. different Base version,
drivers, adding PVA?)

It looks like the tNet0 task is constantly timing out and trying to drop some connection. But I cannot find out which
one. Any network diagnostics function freezes in ipcom_block_wait at a semaphore.

Is there any network traffic associated with this?

Does vxworks expose any packet packet Tx/Rx counters?

Can you look at what other threads are doing?

eg. Maybe some code is in a tight loop trying to (re)connect
a TCP socket without a hold-off time in between?

As there is a TCP server for at least CA, such an offending
loop could be in client code running on another computer.

Has anyone seen this behavior before?

tt tNet0

0x0012c68c vxTaskEntry  +0x48 : ipcomNetTask ()
0x00113c40 ipcomNetTask +0x34 : jobQueueProcess ()
0x0029d9d8 jobQueueProcess+0xe8 : 0x00241584 ()
0x002415b4 ipcom_atomic_sub_and_return+0x9c : 0x001f9208 ()
0x001f9274 ipnet_timeout_cancel+0xe4 : 0x00226910 ()
0x0022699c iptcp_drop_connection+0x8a4: ipcom_list_remove ()
value = 0 = 0x0

tt tNet0

0x0012c68c vxTaskEntry  +0x48 : ipcomNetTask ()
0x00113c40 ipcomNetTask +0x34 : jobQueueProcess ()
0x0029d9d8 jobQueueProcess+0xe8 : 0x00241584 ()
0x002415b4 ipcom_atomic_sub_and_return+0x9c : 0x001f9208 ()
0x001f9274 ipnet_timeout_cancel+0xe4 : 0x00226910 ()
0x00226a48 iptcp_drop_connection+0x950: 0x0022b380 ()
0x0022c4e0 sockInfo     +0x2ca4: ipcom_list_insert_first (0x1f, 0x1f)
value = 0 = 0x0

tt tNet0

0x0012c68c vxTaskEntry  +0x48 : ipcomNetTask ()
0x00113c40 ipcomNetTask +0x34 : jobQueueProcess ()
0x0029d9d8 jobQueueProcess+0xe8 : 0x00241584 ()
0x002415b4 ipcom_atomic_sub_and_return+0x9c : 0x001f9208 ()
0x001f9274 ipnet_timeout_cancel+0xe4 : 0x00226910 ()
0x00226a48 iptcp_drop_connection+0x950: 0x0022b380 ()
0x0022c4bc sockInfo     +0x2c80: 0x0022c58c ()
value = 0 = 0x0

tt tNet0

0x0012c68c vxTaskEntry  +0x48 : ipcomNetTask ()
0x00113c40 ipcomNetTask +0x34 : jobQueueProcess ()
0x0029d9d8 jobQueueProcess+0xe8 : 0x00241584 ()
0x002415b4 ipcom_atomic_sub_and_return+0x9c : 0x001f9208 ()
0x001f9274 ipnet_timeout_cancel+0xe4 : 0x00226910 ()
0x00226a48 iptcp_drop_connection+0x950: 0x0022b380 ()
value = 0 = 0x0

tt tNet0

0x0012c68c vxTaskEntry  +0x48 : ipcomNetTask ()
0x00113c40 ipcomNetTask +0x34 : jobQueueProcess ()
0x0029d9d8 jobQueueProcess+0xe8 : 0x00241584 ()
0x002415b4 ipcom_atomic_sub_and_return+0x9c : 0x001f9208 ()
0x001f9274 ipnet_timeout_cancel+0xe4 : 0x00226910 ()
value = 0 = 0x0

Dirk

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System