Re: flaky IOC problems at Jefferson Lab

[Marty Kraimer <mrk@aps.anl.gov>]

watson@cebaf.gov wrote:
> 
> >2) Raising the priority of the name resolution task above the scan
> >   tasks gives me an uneasy feeling. An ioc should always give
> >   highest priority to local affairs before it pays attention to
> >   CA clients.
> 
> I disagree strongly.  If an operator can't bring up a control screen in a
> timely fashion, the control system isn't working.  If a low priority
> record isn't scanned for a few seconds, its no big deal (see LONG email
> previously about scan priorities).  Again, we have many applications
> split among multiple ioc's, so "local" doesn't address the whole problem.
> 
> Frankly, there is almost nothing in our EPICS databases which is truly
> real time.  We do all our hard real time stuff in VxWorks tasks.

> >No matter what improvements are made to the CA connection algorithm,
> >it will not solve the problem of an IOC having little or no idle time.
> 
> There's plenty of time if priorities are properly managed :)

EPICS has never made the claim that it is designed for hard real time
systems, i.e.
systems which guarantees the time at which results will be produced.
Because EPICS is flexable and expandable it should not take on
this responsibility. HOWEVER, EPICS, together with vxWORKS,
does provide features that allow an application developer
to develop systems that have close to Hard real time requirements.

I/O event scanned (and in 3.13 event scanned) records can be given a
priority.
The periodic scanned records have priorities assigned based on the scan
rate.
Chip made a suggestion, which I think is very good, that the periodioc
scanned
records also have a priority just like I/O event and event records.
For other tasks (GPIB task, Allen Bradley, etc) priorities are also
attached.

Using these priorities an application developer can create and test an
application to ensure that it meets the performance objectives.

The only thing not under the control of the application developer (at
least
not easily under his/her control) is Channel Access activity.
In order to ensure that this does not violate the IOC performance
requirements,
the priorities of all CA tasks was set lower than any of the scan tasks.
It is possible for a CA put request to trigger a lot of record
processing.
This, however, is a result of the database design, i.e. the application
developer wants this to happen. Thus part of testing the performance
requirements is testing all such puts.

Chip has also requested that CA gets/puts also have priorities. (I think
I recall this also being discussed previously by Jeff and others.)
This means multiple priority CA server tasks.
Should any of these priorities be higher than any of the scan tasks?
If they are and priorities are assigned by CA clients with no control
by the IOC developer then assignment of these priorities becomes an
important issue.

Now back to the two issues that have started this entire discussion.

1) Saturated CPUs. 

If CPUs are run at very high CPU utilization things get very difficult.
Ensuring that no task ever starves while also meeting real time
requirements will be an extremely difficult problem. Although better
priority management may help the problem it will never it go away.

2) Setting CA UDP and CA TCP to high priority.

This is not a good way to solve the slow connection problem.
Processing CA search requests can, for short periods of time,
put a significant load on an IOC. This make it extremely
difficult or impossible for the application developer
to test that the performance requirements are meet.
I seems strange that a high priority real time requirement for
an IOC is to make a new operator screen quickly connect.
If this was truly a time critical screeen why wasn't to already
up and running when it was needed?

Marty