Re: excessive ioc memory utilization

[Geoff Savage <savage@fnal.gov>]

Hi Jeff,

The culprit was a misbehaving channel access client.  Somehow it 
connected tens of thousands of channels which increased the memory usage 
by CA.  When it was stopped the memory on the IOC was not returned.  Are 
there tools to monitor the CA memory usage?  Is this the expected 
behaviour of CA?

Thanks,  Geoff

Jeff Hill wrote:

> Geoff,
>
> Use memShow to obtain status of the system pool. A common cause
> of system pool depletion will be system pool data structure
> corruption, or of course some program continuously allocating
> memory that it does not free. The former shows up instantaneously
> while the latter generally can be predicted as a trend. If the
> problem is easily reproducible you may find that selective use of
> the task suspend command can be used to isolate the responsible
> thread. Typically a binary search is used where you suspend one
> half of the threads initially followed by half of the remaining
> half etc to tighten the noose with minimized effort.
>
> With your version of vxWorks the network stack pool shouldn't be
> increasing past the limits set when vxWorks is built.
>
> There is an integer level argument to casr that can be used to
> get more information about what it has allocated. 
>
> Jeff
>
>  
>
> > -----Original Message-----
> > From: Geoff Savage [mailto:savage@fnal.gov]
> > Sent: Tuesday, May 10, 2005 9:48 AM
> > To: EPICS Tech-Talk
> > Subject: excessive ioc memory utilization
> >
> > Hi,
> >
> > When I came in this morning our high voltage iocs are using
> > excessive
> > amounts of memory, around 95%.  They are all mv2301 processors
> > with 16MB
> > of memory running epics 3.14.6 built with vxworks 5.5.1.  The
> > netStackSysPoolShow and netStackDataPoolShow outputs are
> > reasonable.
> > There are not an excessive number of channels connected.  What
> > can I use
> > to look and see where the memory is allocated?  Any other
> > suggestions?
> >
> > Thanks,  Geoff
> >
> > -> netStackSysPoolShow
> > type        number
> > ---------   ------
> > FREE    :    5991
> > DATA    :      0
> > HEADER  :      0
> > SOCKET  :     21
> > PCB     :     35
> > RTABLE  :     91
> > HTABLE  :      0
> > ATABLE  :      0
> > SONAME  :      0
> > ZOMBIE  :      0
> > SOOPTS  :      0
> > FTABLE  :      0
> > RIGHTS  :      0
> > IFADDR  :      4
> > CONTROL :      0
> > OOBDATA :      0
> > IPMOPTS :      0
> > IPMADDR :      2
> > IFMADDR :      0
> > MRTABLE :      0
> > TOTAL   :    6144
> > number of mbufs: 6144
> > number of times failed to find space: 0
> > number of times waited for space: 0
> > number of times drained protocols for space: 0
> > __________________
> > CLUSTER POOL TABLE
> > _______________________________________________________________
> > ________________
> > size     clusters  free      usage
> > ---------------------------------------------------------------
> > ----------------
> > 64       1024      965       15123
> > 128      1024      996       1647
> > 256      512       446       10893
> > 512      512       512       0
> > ---------------------------------------------------------------
> > ----------------
> > value = 80 = 0x50 = 'P'
> >
> > -> netStackSysPoolShow
> > type        number
> > ---------   ------
> > FREE    :    5991
> > DATA    :      0
> > HEADER  :      0
> > SOCKET  :     21
> > PCB     :     35
> > RTABLE  :     91
> > HTABLE  :      0
> > ATABLE  :      0
> > SONAME  :      0
> > ZOMBIE  :      0
> > SOOPTS  :      0
> > FTABLE  :      0
> > RIGHTS  :      0
> > IFADDR  :      4
> > CONTROL :      0
> > OOBDATA :      0
> > IPMOPTS :      0
> > IPMADDR :      2
> > IFMADDR :      0
> > MRTABLE :      0
> > TOTAL   :    6144
> > number of mbufs: 6144
> > number of times failed to find space: 0
> > number of times waited for space: 0
> > number of times drained protocols for space: 0
> > __________________
> > CLUSTER POOL TABLE
> > _______________________________________________________________
> > ________________
> > size     clusters  free      usage
> > ---------------------------------------------------------------
> > ----------------
> > 64       1024      965       15125
> > 128      1024      996       1647
> > 256      512       446       10894
> > 512      512       512       0
> > ---------------------------------------------------------------
> > ----------------
> > value = 80 = 0x50 = 'P'
> >
> >
> > -> casr
> > Channel Access Server V4.11
> > Connected circuits:
> > TCP 131.225.231.65:46261(d0olj.fnal.gov): User="d0cal", V4.8,
> > 70
> > Channels, Priority=0
> > TCP 131.225.230.156:1026(d0olctl38): User="vxworks", V4.11, 6
> > Channels,
> > Priority=80
> > TCP 131.225.231.39:45142(d0ol57.fnal.gov): User="d0fpd", V4.8,
> > 6
> > Channels, Priority=0
> > TCP 131.225.231.39:45149(d0ol57.fnal.gov): User="d0fpd", V4.8,
> > 4
> > Channels, Priority=0
> > TCP 131.225.231.252:60499(d0ol49.fnal.gov): User="d0fpd", V4.8,
> > 4
> > Channels, Priority=0
> > TCP 131.225.231.252:49088(d0ol49.fnal.gov): User="d0fpd", V4.8,
> > 30
> > Channels, Priority=0
> > TCP 131.225.231.39:49714(d0ol57.fnal.gov): User="d0fpd", V4.8,
> > 6
> > Channels, Priority=0
> > TCP 131.225.231.247:42004(d0ol45.fnal.gov): User="d0cal", V4.8,
> > 3
> > Channels, Priority=0
> > TCP 131.225.231.247:43286(d0ol45.fnal.gov): User="d0cal", V4.8,
> > 128
> > Channels, Priority=0
> > value = 0 = 0x0
> >    
>
>
>