Re: Gateway Crashes

["Steven Banks" <Steven.Banks@synchrotron.vic.gov.au>]

Guys,

Dirk's suggestion of reducing the stack size via the "ulimit -s 4096"
command seems to have worked for us, at least for the moment.  I also note
Jeff Hill's comments that in EPICS 3.14.7 and beyond, the gateway will no
longer terminate, just not create the thread.  This has increased the
incentive for us to move to 3.14.8.2 in the very near future.

Thanks for your help.  Tech-talk again proves its worth.

Steve.

-----Original Message-----
From: Dirk Zimoch <dirk.zimoch@psi.ch>
To: Russ Berg <Russ.Berg@lightsource.ca>
Cc: Steven Banks <steven.banks@synchrotron.vic.gov.au>, tech-talk@aps.anl.gov
Date: Tue, 27 Jun 2006 13:29:50 +0200
Subject: Re: Gateway Crashes

> I think, the EPICS_CA_MAX_ARRAY_BYTES problem is something else. In my 
> experience, the gateway just becomes braindead, uses 100% CPU, stops
> any 
> communication and does not even write an error message if you try to 
> read a waveform which is larger than EPICS_CA_MAX_ARRAY_BYTES. In my 
> opinion, the EPICS_CA_MAX_ARRAY_BYTES variable is a bad idea anyway, 
> because for a generic application like the gateway, you can't know the 
> maximum data size in advance. There should be something more flexible
> in 
> the underlying CA code. (Even though that could lead to memory 
> fragmentation.)
> 
> Dirk
> 
> Russ Berg wrote:
> > Steven,
> > I am not familiar with the gateway code and this is likely not your 
> > problem but it may give you some idea's, I have been able to crash
> our 
> > gateway when the EPICS_CA_MAX_ARRAY_BYTES environ var is not set and
> you 
> > try to pull a waveform or analog array back through the gateway, I
> think 
> > the default is 16k bytes, this needs to be set on the IOC and
> wherever 
> > your GUI is. Again I don't recall any messages about pthread_create 
> > calving, but setting the max bytes to a value large enough made the 
> > gateway happy.
> >  
> > When you dial back the number of connections the problem goes away?
> if 
> > so, at what number of connections does it die?
> >  
> > -Russ
> > 
> >
> -----------------------------------------------------------------------
> -
> > From: Steven Banks [mailto:steven.banks@synchrotron.vic.gov.au]
> > Sent: Monday, June 26, 2006 12:57 AM
> > To: tech-talk@aps.anl.gov
> > Subject: Gateway Crashes
> > 
> > We are having severe problems with the EPICS gateway, and I'm hoping 
> > that someone can help us resolve them.
> >  
> > The problem is that the gateway is continually crashing whenever we
> open 
> > up a GUI display with requires connection to multiple IOCs.  These
> are 
> > Libera BPM boxes, and we have 98 of them.  We have a status display 
> > which shows the status of 33 of these at once.  As we scroll from one
> > display to the other, the gateway invariably crashes.
> >  
> > (This is a gateway between our office network and our accelerator 
> > network, and is used for monitoring the status of the machine from
> our 
> > desktops.)
> >  
> > The gateway log provides the following information:
> >  
> > ***************************************************************
> > Jun 26 16:01:02 PV Gateway Version 2.0.0.0 [Jun 14 2006 15:10:22]
> > EPICS 3.14.6 PID=8840 ServerPID=8839
> > EPICS_CA_ADDR_LIST=BROADCAST-ACC
> > EPICS_CA_AUTO_ADDR_LIST=NO
> > EPICS_CA_SERVER_PORT=Not specified
> > EPICS_CA_MAX_ARRAY_BYTES=Not specified
> > EPICS_CAS_INTF_ADDR_LIST=CR01GW01-OFFICE
> > EPICS_CAS_SERVER_PORT=Not specified
> > EPICS_CAS_IGNORE_ADDR_LIST=Not specified
> > Running as user ics on host CR01GW01
> > Statistics PV prefix is CR01GW01
> > pthread_create  error Cannot allocate memory
> >  
> > Jun 26 16:36:12 !!! Errlog message received (message is above)
> > fatal error: epicsThreadCreate
> >  
> > Jun 26 16:36:12 !!! Errlog message received (message is above)
> > ***************************************************************
> >  
> > Has anyone else seen the memory problems that we are experiencing? 
> Is 
> > this an indication that we are trying to connect to too many PVs at 
> > once?  Will purchasing more memory for the gateway perhaps alleviate
> the 
> > problem.  It currently has 512MB but we are planning to upgrade it to
> 1 GB.
> >  
> > Dr. Steven M. Banks ? Control Systems Engineer
> > Australian Synchrotron Project
> > Major Projects Victoria
> > 800 Blackburn Road
> > CLAYTON VIC 3168
> > tel: +613 8540 4103
> > fax: +613 8540 4200
> > 
> > mailto:steven.banks@synchrotron.vic.gov.au
> > http://www.synchrotron.vic.gov.au
> <http://www.synchrotron.vic.gov.au/>
> > 
> 
> -- 
> Dr. Dirk Zimoch
> Swiss Light Source
> Paul Scherrer Institut
> Computing and Controls
> phone +41 56 310 5182
> fax   +41 56 310 4413