RE: VxWorks Buffer Problems

["Jeff Hill" <johill@lanl.gov>]

Matt,

> CA.Client.Diagnostic...........................................
>    Message: "Network connection lost"
>    Severity: "Warning" Context: "10.2.2.104:5064"
>    Source File: ../iocinf.c Line Number: 1488
>..................................................................
> CAC: unexpected select fail: 851971=S_iosLib_INVALID_FILE_DESCRIPTOR

As I recall there was a benign message like that in R3.13 that resulted from
blocking in select() at the same time that a file descriptor select was
watching was closed.

>0x820dec (CA_event): CAS: TCP send to "10.2.2.63:55102" 
> failed because "S_errno_ENOBUFS"

>From my perspective this isn't a crash but instead a diagnostic, but I can
understand that the net result is the same - communication is lost. You
might fault isolate by checking to see if the IOC is ping-able, and or
responding to telnet, during this situation.

Are the mbuf messages occurring only with the MVME-2700 hardware? If so I
would make certain that you have all of the driver patches from wrs.

> When I run 'netStackSysPoolShow' I see 2048 mbufs

As I recall on vxWorks the ENOBUF error number can be returned when running
out of, in addition to mbufs, clusters of various sizes, and also driver
buffers.

They have some experience with vxWorks buffer starvation issues and PPC VME
cards at the SNS. Dave Thompson there might have some advice. 

Also, EPICS R3.14.7 or higher ca client library is better optimized to avoid
increasing traffic during isolated periods of network congestion compared to
previous versions. One way to ease into R3.14 would be to try linking your
console applications with R3.14 first.

Jeff

-----Original Message-----
From: tech-talk-bounces@aps.anl.gov [mailto:tech-talk-bounces@aps.anl.gov]
On Behalf Of Matt Rippa
Sent: Tuesday, June 12, 2007 7:45 PM
To: tech-talk@aps.anl.gov
Subject: VxWorks Buffer Problems

Hi All -

Lately we have several CA crashes which may be buffer problems on 
our IOC's. We're using EPICS R3.13.4 on MVME-167 and MVME-2700 
targets.

When I run 'netStackSysPoolShow' I see 2048 mbufs. This looks 
good to me. Are there other useful checks from the shell I can 
run to assert my image is built in the best manner possible?

The INVALID FILE DESCRIPTOR message below seems strange to me. 
Has anyone seen that before?

Thanks for any help,
-Matt Rippa


 > CA.Client.Diagnostic...........................................
 >    Message: "Network connection lost"
 >    Severity: "Warning" Context: "10.2.2.104:5064"
 >    Source File: ../iocinf.c Line Number: 1488
 >..................................................................
 > CAC: unexpected select fail: 
851971=S_iosLib_INVALID_FILE_DESCRIPTOR


 > dbCa:exceptionCallback stat Network connection lost channel 
unknown
 > dbCa:exceptionCallback stat Network connection lost channel 
unknown
 > 0x93f7f4 (CA_event): CAS: TCP send to "10.2.2.151:59106" 
failed because "S_errno_ENOBUFS"

 >0x820dec (CA_event): CAS: TCP send to "10.2.2.63:55102" failed 
because "S_errno_ENOBUFS"
 >0x7d47e8 (CA_event): CAS: TCP send to "10.1.10.18:37796" failed 
because "S_errno_ENOBUFS"
 >0x8408b4 (CA_event): CAS: TCP send to "10.1.5.50:56530" failed 
because "S_errno_ENOBUFS"
 >0x845558 (CA_event): CAS: TCP send to "10.2.2.55:46467" failed 
because "S_errno_ENOBUFS"
 >0x81b4ac (CA_event): CAS: TCP send to "10.2.2.31:41488" failed 
because "S_errno_ENOBUFSCAC: erro"
 >r = "S_errno_ENOBU0xFS" sending UDP ms819f0cg to 10.2.2.255 (:5064
 >CA_event): CAS: TCP send to "10.2.2.31:41489" failed because 
"S_errno_ENOBUFS"
 >0x8abe64 (CA_event): CAS: TCP send to "10.2.5.43:44426" failed 
because "S_errno_ENOBUFS"