Re: diagnostics related to discarding an intermediate CA subscription update

[Claude Saunders <saunders@aps.anl.gov>]

Regarding the issue of "flow control" diagnostics, would it be 
reasonable to approach this like telecom and server manafacturers? I am 
suggesting using SNMP to offer these diagnostics much like how we 
monitor dropped packets on commercial routers. It is UDP-based, so would 
probably have little impact on IOC performance, especially if only 
polled from NMS clients slowly and at low priority. Like Jeff pointed 
out, I wouldn't be inclined to send flow controls warnings up the same 
pipe that is already backed up. I consider "flow control" diagnostics 
something that is better suited to out-of-band communication: 
out-of-band in the sense of either a separate protocol, or even a 
separate NIC.

We could register an EPICS IOC MIB and all that...

 - Claude

Jeff Hill wrote:
> Ernest,
>
>   
> > Jeff, is there a way to address the request for diagnostic info about CA
> > flow control.  Kay has asked about this one previously?  Would this be a
> > minor or major change?  We want to be able to see when flow control
> > "kicks in"
> >     
>
> First there are two situations that can cause an intermediate subscription
> update to be discarded.
> 1) Event queue entry quota saturation
> 2) Client library initiated flow control
>
> Currently only the server, and not the client library, knows about the
> discard of an intermediate updates due to event queue entry quota
> saturation. Furthermore, event queue entry quota saturation is a
> per-channel, and not a per-circuit phenomenon. In contrast, flow control is
> a per-circuit phenomenon which is initiated by the client library when it
> knows that it, or the application, isn't keeping up. However similarly, both
> mechanisms are characterized by the number of updates that are discarded in
> the server. For example, the client can be in flow control mode, but if
> there isn't more than one update while in this mode we will not discard an
> update. 
>
> It would of course be unwise to print a mandatory message on the server's or
> client's console when discarding an intermediate subscription update because
> printf is much higher overhead call compared to the per subscription update
> message cost in CA so this could be another "adding more fuel to the fire
> once it starts" situation that we would need to avoid.
>
> On the server side, we already have the dbel (<record name>, <interest
> level>) diagnostic which at sufficiently high interest levels reports the
> number of discards that have occurred for each subscription attached to a
> particular record.
>
> For the client to know about the magnitude of the problem we would need for
> the serve to send a message to the client. It makes no sense to inject a
> message into the protocol stream when there is a discard as this could be
> another "adding more fuel to the fire once it starts" situation that we
> would need to avoid. Perhaps the sensible thing to do would be to include a
> number of discards since the last update count in the update message. This
> would require a free slot in the message header (we unfortunately don't have
> one) or a change in the format of the message payload. Changing the format
> of the payload is possible, but we would need to send the modified payload
> only to clients of a particular protocol version range. This, and any
> changes required to convey the information to the client's callback, might
> be considered to be a large change for a point release such as R3.14.9. We
> would also slightly increase the per-update message size, and therefore
> potently reduce maximum throughput, but that might tend to be a negligible
> impact.
>
> So I am not predisposed to provide the type of functionality described in
> the proceeding paragraph in R3.14.9, but if there appears to be a general
> level of interest I am very willing to create a Mantis entry against R3.15
> summarizing a feature request for that release which could be brought up for
> consideration at a future EPICS developers meeting.
>
> Jeff
>
>   
> > -----Original Message-----
> > From: Ernest L. Williams Jr. [mailto:ernesto@ornl.gov]
> > Sent: Thursday, January 25, 2007 2:15 PM
> > To: Jeff Hill
> > Cc: ernesto@ornl.gov; 'EPICS core-talk'
> > Subject: Re: FW: R3.14.9: Any changes still pending?
> >
> > On Thu, 2007-01-25 at 14:01 -0700, Jeff Hill wrote:
> >     
> > > > Jeff - are all your fixes committed now?  I don't want to go ahead
> > > >         
> > with
> >     
> > > > the -RC1 tag until I get a positive response from you.
> > > >         
> > > I have no patches pending at this time.
> > >       
> > Jeff, is there a way to address the request for diagnostic info about CA
> > flow control.  Kay has asked about this one previously?  Would this be a
> > minor or major change?  We want to be able to see when flow control
> > "kicks in"
> >
> >
> > Thanks,
> > Ernest
> >
> >
> >
> >     
> > > Jeff
> > >
> > >       
> > > > -----Original Message-----
> > > > From: Andrew Johnson [mailto:anj@aps.anl.gov]
> > > > Sent: Thursday, January 25, 2007 11:18 AM
> > > > To: EPICS core-talk
> > > > Subject: R3.14.9: Any changes still pending?
> > > >
> > > > Does anyone have any changes for R3.14.9 that they haven't committed
> > > > yet?  I would like to tag the -RC1 release tomorrow, but my rules
> > > >         
> > imply
> >     
> > > > that there's no point doing that if someone has any patches to commit.
> > > >
> > > > I have a minor IOC shutdown issue on Linux that I should have pushed
> > > > harder with earlier, but the solution would affect many source files
> > > >         
> > and
> >     
> > > > we don't have time to discuss the fix now so I'll document it on the
> > > > Known Problems page and leave the resolution until later.
> > > >
> > > > I am also still in the process of running the mrkSoftTest suite.
> > > >
> > > > Jeff - are all your fixes committed now?  I don't want to go ahead
> > > >         
> > with
> >     
> > > > the -RC1 tag until I get a positive response from you.
> > > >
> > > > - Andrew
> > > > --
> > > > The right to be heard does not automatically include
> > > > the right to be taken seriously. -- Hubert H. Humphrey
> > > >         
>
>