RE: CA loss package on 1 Hz rate

["Liyu, Andrei" <liyua@ornl.gov>]

Jeff,

	I don't sleep night :-) Can you explain me why did you choice to
have "Each client circuit has one queue"?
	As I understood there is one client circuit (it goes to TCP/IP
features) per PV channel. One PV can have some channels. I tried to
compare reasons and I would prefer to have queue per PV. (Actually, I
did something similar for another project with DCOM protocol.)

Andrei.

-----Original Message-----
From: Jeff Hill [mailto:johill@lanl.gov] 
Sent: Monday, August 22, 2005 4:34 PM
To: Liyu, Andrei; tech-talk@aps.anl.gov
Subject: RE: CA loss package on 1 Hz rate


> 	Does record has one queue? Or PV channel has queue?

Each client circuit has one queue.

> Do scalar and array be similar queue(=1)?

The per-subscription queue quota for scalars is > 1, and for arrays is
== 1.
Furthermore, with arrays only, indication that an update is needed, but
not
the current value of the array, is placed in the event queue. The
subscription update payload is set to whatever array value happens to be
in
the database when the server processes the event queue.

> 	When Monitor() function is called data will be copied to network
> stack? Or Monitor() sets a "semaphore" to notify server layer about
new
> data?

It sets a semaphore after it updates the queue. It is very important of
course that the scan task does not block if a network circuit is
stalled.

Jeff

> -----Original Message-----
> From: Liyu, Andrei [mailto:liyua@ornl.gov]
> Sent: Monday, August 22, 2005 2:00 PM
> To: Jeff Hill; tech-talk@aps.anl.gov
> Subject: RE: CA loss package on 1 Hz rate
> 
> Jeff,
> 
> 	Does record has one queue? Or PV channel has queue? Do scalar
> and array be similar queue(=1)?
> 	When Monitor() function is called data will be copied to network
> stack? Or Monitor() sets a "semaphore" to notify server layer about
new
> data?
> 
> 	I can add one more behavior feature. Let say IOC had 20
> detectors. All time BLM thread sends new data in the same order to
Epics
> (this is in the same IOC). Let's say PV1, PV2, ..., PV20. Clearly, PC
> doesn't get data and all time it will be the tail of order, for
example,
> from PV15 to PV20. I see 6 lines (one per PV) that data was lost.
> 	Finally, all time I see one additional message that PV15 (first
> lost PV) came second time (=twice with the same serial number). I
> couldn't understand why I see this just one time? It should be 6
times.
> Or CA has buffer for a semaphore and the semaphore is once per all
> arrays?
> 	Or is there another source of the problem?
> 
> Thanks, Andrei.
> 
> 
> -----Original Message-----
> From: Jeff Hill [mailto:johill@lanl.gov]
> Sent: Monday, August 22, 2005 11:50 AM
> To: Liyu, Andrei; tech-talk@aps.anl.gov
> Subject: RE: CA loss package on 1 Hz rate
> 
> 
> Andrei,
> 
> > PC checks serial number and prints on terminal port if data was
> > lost or come twice. Time to time (very seldom) I read both problems.
> 
> Within an EPICS IOC the scan tasks run at relatively high priority and
> the
> CA server runs at relatively low priority. This design ensures proper
> independence of scan tasks from introduced client load.
> 
> When a record is processed, and it needs to provide a subscription
> update to
> the CA server, it places the update in a queue. When the CA server
gets
> an
> opportunity to run it reads any events that might be waiting on the
> queue
> and sends them over the wire to the CA clients that are subscribed.
> 
> To protect the IOC, there is a quota limiting the maximum length of
the
> CA
> subscription update event queue. When this quota is exceeded the last
> event
> on the queue for a particular subscription is replaced. This means
that
> if
> there is a (possibly brief) period where the scan tasks produce
> subscription
> updates at a rate that exceeds the rate at which the server can send
> subscription updates to the client or the rate at which the client
> processes
> subscription updates then the buffering capacity of the event queue
can
> be
> exceeded. The bursty nature of CPU and network loads can make this
> situation
> more likely to occur. CPU and network load reporting diagnostics tend
to
> report aggregate loads averaged over time, and do not tend to report
> load
> burst statistics. That would require a strip chart or a histogram.
> 
> Furthermore, the memory management for the CA event queue is currently
> optimized for use with scalar value / time stamp / alarm status tuple
> updates. This means that multiple value updates might be queued
together
> at
> once allowing intermediate values to not be discarded during brief
> periods
> of client or network load saturation. In contrast, array value / time
> stamp
> / alarm status tuples are not queued by the system. The quota for
these
> updates is exactly one update, and therefore it is very likely that
> intermediate values can be discarded during brief intervals where the
> client
> CPU load or network load saturate. This behavior is by design. We were
> very
> careful to not call malloc/free for queuing of random sized arrays
> because
> of concerns about excessive CPU loading and fragmentation of pool.
Worse
> yet, since array element values are not placed on the queue then, with
> arrays, the event is only a signal that an update is required. The
> actual
> value that is sent is whatever that might be in the database at the
time
> that the server reads the place holder array update event from the
> queue.
> This means that the same update value might be sent twice (as you have
> observed).
> 
> The above is a well known limitation of the system at this time. With
> EPICS
> V4 we intend to eliminate this problem by transferring event queue
> memory
> management responsibility to the event producer.
> 
> Jeff
> 
> > -----Original Message-----
> > From: Liyu, Andrei [mailto:liyua@ornl.gov]
> > Sent: Monday, August 22, 2005 8:43 AM
> > To: tech-talk@aps.anl.gov
> > Subject: CA loss package on 1 Hz rate
> >
> > Hi,
> >
> > 	SNS BLM system has 2 levels. First level is VME based/vxWorks
> > (IOC). Second level is PC/Windows (PC) - similar CAGateway. IOC
sends
> > data one time per second to PC. Data is arrays (one array (720
floats)
> > per detector; up to 32 detectors per IOC). Array has information
about
> > serial number of current package. PC checks serial number and prints
> on
> > terminal port if data was lost or come twice. Time to time (very
> seldom)
> > I read both problems.
> > 	SNS works on 1 Hz now. IOC CPU loading ~ 3-5% and PC CPU loading
> > 5%. On 60 Hz IOC will have 30% on 60 Hz. PC doesn't depend of
> > accelerator rate but depends from clients quantity.
> > 	I hoped to use CAArchive to find source of problem. But I met
> > another problem. Archive lost connection to IOC when IOC/PC
connection
> > was Ok.
> >
> > 	Does anyone see this problem? Any other suggestions to find the
> > source (IOC, network, PC)?
> >
> > Thanks, Andrei.