RE: task priorities, busy cpu, and timeout

["Jeff Hill" <johill@lanl.gov>]

> The longer running clients like EDM and MEDM, StripTool and ALH should
> connect eventually.  I suspect you'll have the most trouble with any
> scripts or similar short running programs that aren't expecting the
> connection process to take longer than a second.  If you can increase
> their timeouts that may be sufficient to solve the problem.

It just might be a good guess that timeouts have been specified too low for
script based ca get activity. 

> Is processing load continuous, or do you get periods of time (seconds or
> more long) when there's no idle at all and others when there is?

This is also an important question. If there are not bursts of CPU
saturation of duration similar in magnitude to the shortest timeout imposed
in the CA client application then you would expect decent level of
responsiveness with 37% idle.

It might be interesting to test the vxWorks system in question with command
line "ping -s" to see if there are any intermittent round trip delay
artifacts due to IP kernel congestion - independent of any delays imposed by
the CA server. Intermittent vxWorks IP kernel delays can sometimes be
eliminated by configuring additional buffering quota to the IP kernel when
the vxWorks kernel is built. Also, in my experience reliable operation
sometimes requires upgrading the network interface's vxWorks device driver
to include the latest available patches.

Also, watch out for t_netTask thread starvation resulting from interrupt
activity and or thread activity at priorities the same as or exceeding that
of the t_netTask thread. Frequently, mutex semaphores in the vxWorks system
are configured for priority inheritance. That can cause a thread to
intermittently jump its priority to a much higher level.

Jeff

> -----Original Message-----
> From: Andrew Johnson [mailto:anj@aps.anl.gov]
> Sent: Wednesday, February 28, 2007 3:19 PM
> To: Dennis Nicklaus
> Cc: tech-talk@aps.anl.gov
> Subject: Re: task priorities, busy cpu, and timeout
> 
> Hi Dennis,
> 
> Dennis Nicklaus wrote:
> > We have a Motorola MVME 5500 CPU running Epics on VxWorks.  It'sdoing
> > some cpu-intensive computations and runs about 37% idle.  Our problem is
> > that this makes EPICS connectivity iffy, resulting in a timeout to a
> > client doing a caget (for example) about 20% of the time.
> 
> Is processing load continuous, or do you get periods of time (seconds or
> more long) when there's no idle at all and others when there is?  If
> your system behaves like that then I can see why your CA connectivity
> might be poor for some applications (the default timeout for caget is 1
> second, although you can change that with the -w option).
> 
> > A large fraction (about half) of the used cpu time is spent in the Epics
> > cbLow task, processing genSub record routines which are processed based
> > on an epics "event" scan field (posted at about 1Hz). (see spy output
> > below)
> >
> > Is there an existing document/web page describing the duties and
> > relative priorities of the various Epics tasks somewhere?  Which task(s)
> > are responsible for responding to a CA Client (the CAS-client tasks, I'm
> > guessing)?
> 
> Sorry, there probably should be but no such document exists.  Your guess
> is right.  The three cbXxx tasks are for general-purpose operations that
> need to happen in task context
> 
> > Can I lower the priorty of the cbLow task (to below that of the
> > CAS-client tasks) and still expect epics to work correctly?
> 
> Actually you probably can - the core code shouldn't notice, but you will
> need to be aware that doing this permits CA clients to affect the
> internal operation of the IOC.  The reason why all CA tasks are lower
> than the database tasks is that the IOC's most important job is usually
> to execute the process database, and when we're running out of time we'd
> much rather drop networking activities than record activities.
> 
> > Why do the genSub event-driven records get
> > processed in cbLow?  Is it because they are genSubs or because they are
> > event-driven?
> 
> The latter; record processing can occur in several different tasks, but
> the cbXxx and the periodic scan tasks are the main ones.
> 
> > It's a relatively new system and re-designing our software in light of
> > the cpu-usage is certainly possible, but we'd like to know more about
> > the epics priorities and how exactly a client gets a channel access
> > timeout before proceeding.  We're running epics 3.14.8 on vxworks 6.1 if
> > it makes any difference.
> 
> The longer running clients like EDM and MEDM, StripTool and ALH should
> connect eventually.  I suspect you'll have the most trouble with any
> scripts or similar short running programs that aren't expecting the
> connection process to take longer than a second.  If you can increase
> their timeouts that may be sufficient to solve the problem.
> 
> - Andrew
> --
> The right to be heard does not automatically include
> the right to be taken seriously. -- Hubert H. Humphrey