EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  <20042005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  <20042005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Re: Buffer problems
From: john sinclair <[email protected]>
To: "Mark Rivers" <[email protected]>, "Andrew Johnson" <[email protected]>
Cc: <[email protected]>
Date: Wed, 24 Mar 2004 17:56:21 -0500
Mark,

Wind River gave me the code shown below. I load this before IocInit and have 
played with buffer size adjustments. For example:

  ld < /ata0/ethShow.o
  changeIfSendQMax("elt0",2000)

When I have network problems, I do the following:

  ifQValuesShow("elt0")

and I always see

  elt0 drops = <some number> queue length = 2000 max_len = 2000

Before things get into this state, ifQValuesShow("elt0") always gives

  elt0 drops = 0 queue length = 0 max_len = 2000

At this point, network communications have ceased and I must reboot. I don't 
know what causes this and I started down the path of upgrading to tornado 
2.2.1. EPICS, however, does not build.

I only see this on one IOC that sustains a high level of network traffic (CA, 
Allen/Bradley ethernet PLC, MOXA ethernet-to-serial devices).

The buffer size adjustment makes the problem less frequent, but it has not 
gone away.

Wind river hinted that their ethernet driver might have a problem. This 
inspired the attempt to upgrade to 2.2.1.

I'm running version 3.13.6.

John Sinclair, ORNL

====================================================================================================
/*******************************************************************************
 *
 *  THIS CODE IS NOT SUPPORTED CODE.  IT IS PROVIDED ON AN AS-IS BASIS.
 *
*******************************************************************************/
 
#include "vxWorks.h"
#include "ifLib.h"
#include "etherLib.h"
#include "stdio.h"
#include "net/if.h"
#include "net/unixLib.h"
#include "netBufLib.h"
#include "end.h"
#include "muxLib.h"
#include "netShow.h"
#include "stdio.h"


IMPORT struct ifqueue ipintrq;
IMPORT struct ifqueue arpintrq;

IMPORT void netPoolShow (NET_POOL_ID pNetPool);

/******************************************************************************
*
* changeProtocolInputQMax - Change IP and ARP input queue maximum length
*                       Default configuration sets these values to 50 packets 
*                       each.
* 
*/
void changeProtocolInputQMax
    (
    int ipQNewSize,   /* Default is 50 */
    int arpQNewSize   /* Default is 50 */
    )
    {
    int s;
   
    printf("IP input maximum queue length = %d\n", ipintrq.ifq_maxlen);
    printf("Arp input maximum queue length = %d\n", arpintrq.ifq_maxlen);
    
    s = splnet();
    arpintrq.ifq_maxlen = arpQNewSize;
    ipintrq.ifq_maxlen = ipQNewSize;
    splx(s);

    printf("New IP input maximum queue length = %d\n", ipintrq.ifq_maxlen);
    printf("New Arp input maximum queue length = %d\n", arpintrq.ifq_maxlen);

    return;

    }


void protocolQValuesShow (void)
    {
   
    printf("IP receive queue max size = %d\n", ipintrq.ifq_maxlen);
    printf("IP receive queue drops = %d\n\n", ipintrq.ifq_drops);
    printf("ARP receive queue max size = %d\n", arpintrq.ifq_maxlen);
    printf("ARP receive queue drops = %d\n", arpintrq.ifq_drops);


    return;

    }


/******************************************************************************
*
* ifQValuesShow - Show number of times packets were dropped because interface 
*  send was full.
*
* name
*     Expects the interface name plus the unit number: dc0, fei0 
* 
*/

void ifQValuesShow
    (
    char *name
    )
    {

    struct ifnet *ifp;

    ifp = ifunit(name);

    if (ifp == NULL)
        {
        printf("Could not find %s interface\n", name);
        return;
        }
    printf("%s drops = %d queue length = %d max_len = %d \n", name,     
     ifp->if_snd.ifq_drops,
     ifp->if_snd.ifq_len, ifp->if_snd.ifq_maxlen);
    return;
    }

/*
* changeIfSendQMax - Change interface send queue length.
*
* name expects the interface name plus the unit number: dc0, fei0 
*
* len:  New length.   Default configuration sets this value to 50.
*
*/

void changeIfSendQMax
    (
    char *name,
    int len
    )
    {

    struct ifnet *ifp;
    int s;

    ifp = ifunit(name);

    if (ifp == NULL)
        {
        printf("Could not find %s interface\n", name);
        return;
        }

    printf("%s previous queue length was %d\n", name, ifp->if_snd.ifq_maxlen);
    s = splnet();
    ifp->if_snd.ifq_maxlen = len;
    splx(s);
    printf("Queue length set at: %d\n", ifp->if_snd.ifq_maxlen);
    return;
    }

/*
* endMibShow - Show the contents of the MIB2 table for end drivers.
*
* Output errors can be errors while transmitting or dropped packets due to
* driver's out of buffers error condition.  Input errors can be packets
* received
* with errors or packets dropped due to driver's out of buffers error 
* condition.
* See endPoolShow below. 
* 
*/
void endMibShow
    (
    char * devName,
    int unit
    )
    {

    END_OBJ * pEnd;

    if ((pEnd = endFindByName (devName, unit)) == NULL)
        {
        printf ("Could not find device %s\n", devName);
        return;
        }

    printf("Mib table statistics for %s%d:\n", devName, unit);
    printf("Output errors = %lu   ", pEnd->mib2Tbl.ifOutErrors);
    printf("Output Packets = %lu\n", pEnd->mib2Tbl.ifOutUcastPkts);
    printf("Input errors = %lu\n", pEnd->mib2Tbl.ifInErrors);
    printf("Unknown protocols = %lu\n", pEnd->mib2Tbl.ifInUnknownProtos);

    return;

    }

/*
* endPoolShow - Show the status of the private memory pool for end drivers.
*
* The total number of clusters is usually equal to the sum of the number of 
* transmit,
* receive and loan buffer descriptors that a driver is initialized with.
* See the External Interface section for the driver's library description in 
* the
* vxWorks Reference manual (i.e dec21x40End), the driver's header file, and 
* conf igNet.h
* for the default values.
*
* If the number of free buffers is 0, the device will be unable to either 
* transmit 
* (Mib 2 output errors will increase)  or receive packets (The Mib table input 
* errors
*  statistics will increase).
*
* Why would the free pool drop to 0?  Most common reason is task not reading a 
* socket.
* Drivers lend buffers to the stack which are not freed until the socket is 
* read
.
*
*/

void endPoolShow
    (
    char * devName,  /* The inteface name: "dc", "ln" ...*/
    int unit         /* the unit number: usually 0 */
    )
    {
    END_OBJ * pEnd;

    if ((pEnd = endFindByName (devName, unit)) != NULL)
        netPoolShow (pEnd->pNetPool);
    else
        printf ("Could not find device %s\n", devName);
    return;
    }

/*
* changeMtu - Change default interface MTU.
* 
*/
void changeMtu
    (
    char *ifName,
    int newSize
    )
    {
    struct ifnet *ifp;
    int s;

    ifp = ifunit(ifName);

    if (ifp == NULL)
        {
        printf("Could not find %s interface\n", ifName);
        return;
        }

    printf("Previous MTU for %s was %ld\n", ifName, ifp->if_mtu);
    s = splnet();
    ifp->if_mtu = newSize;
    splx(s);
    printf("MTU is now set at: %ld\n", ifp->if_mtu);
    return;
    }

void changeSendQueueLength
   (
      char *name,
      int len
   )
   {

    struct ifnet *ifp;
    int s;

    ifp = ifunit(name);

    if (ifp == NULL)
    {
    printf("Could not find %s interface\n", name);
    return;
    }

    printf("%s previous queue length was %d\n", name,
             ifp->if_snd.ifq_maxlen);
    s = splnet();
    ifp->if_snd.ifq_maxlen = len;
    splx(s);
    printf("Queue length set at: %d\n", ifp->if_snd.ifq_maxlen);
    return;
    }
====================================================================================================

On Wednesday 24 March 2004 05:30 pm, Mark Rivers wrote:
> Andrew,
>
> >  I'm not quite sure what causes ENOBUFS errors when
> > sending CA beacons; I have seen similar errors on IOCs here that have
> > never run out of DataPool or SysPool buffers, and I don't yet
> > know what they're complaining about (any ideas Jeff?).  They do appear
> >
> > to be benign though, if my memory is correct - the loss of a beacon
>
> can cause some
>
> > unnecessary CA searches to occur, but nothing should break.
>
> Things are not benign for us, though the CA error messages may just be
> symptomatic of deeper problems.  We have lost NFS and CA connectivity,
> and we have to reboot.  It runs for a while, and then it happens again.
> I suspect there is a machine on the network there that is infected with
> some virus, and periodically blasting each machine on the network.
>
> Disconnecting the beamline switch from the network has fixed the
> problem, but of course that means I can't diagnose it from here at
> Argonne.
>
> Mark

-- 
John Sinclair


References:
RE: Buffer problems Mark Rivers

Navigate by Date:
Prev: RE: Buffer problems Mark Rivers
Next: RE: Buffer problems Jeff Hill
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  <20042005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: RE: Buffer problems Mark Rivers
Next: Re: Buffer problems Tim Mooney
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  <20042005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 10 Aug 2010 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·