RE: gateway enum writes

["Mark Rivers" <rivers@cars.uchicago.edu>]

Sorry, one sentence should have "Count" instead of "Start"

The symptom of the problem on some Linux systems is that after pressing
"Count" both the "Count" and "Done"
buttons on the medm Choice Button widget are displayed in the "pressed"
or "selected"  state.  

> -----Original Message-----
> From: Mark Rivers 
> Sent: Thursday, March 30, 2006 11:51 AM
> To: Kenneth Evans, Jr.; Jeff Hill; Dirk Zimoch; Leicester, PJ (Pete)
> Cc: Tim Mooney; Ralph Lange (BESSY); tech-talk@aps.anl.gov; 
> Ned Arnold; Andrew Johnson
> Subject: RE: gateway enum writes
> 
> Ken,
> 
> Here is a quick statement of the problem.  The scaler record 
> has a field
> called .CNT which in the .dbd file is a menu with 2 values, "Done" (0)
> and "Count" (1).  The medm screen has a "Choice Button" for 
> this field,
> on which the user presses "Count" to start the scaler, and 
> which goes to
> "Done" when the scaler is done.  The symptom of the problem on some
> Linux systems is that after pressing "Start" both the "Count" 
> and "Done"
> buttons on the medm Choice Button widget are displayed in the 
> "pressed"
> or "selected"  state.  Since this widget is a radio button, 
> that should
> never happen, the .CNT field is always 0 or 1, so one and only one of
> those buttons should be in the selected state.  Often when the "Count"
> button is not active, and the CNT field is "Done", pressing 
> the "Count"
> button does not cause the CNT field to go to "Count".  This 
> is verified
> with camonitor on the CNT field.
> 
> This does not happen on medm on Windows, it works perfectly there.
> 
> The problem occurs whether the XWindows display is local on Linux, or
> remote on a Windows box.  So the problem is not with the X 
> server, since
> the failure depends on where medm is running, not where it is being
> displayed.
> 
> I don't see any evidence that, as Ken stated, the problem is with the
> scaler record not working with Channel Access.  Otherwise why would it
> work fine on Windows and many Linux platforms?
> 
> I conclude that the problem is with the X windows in the Linux medm
> client, but perhaps I am wrong.
> 
> Mark
> 
> > -----Original Message-----
> > From: Kenneth Evans, Jr. [mailto:evans@aps.anl.gov] 
> > Sent: Thursday, March 30, 2006 8:25 AM
> > To: 'Jeff Hill'; 'Dirk Zimoch'; 'Leicester, PJ (Pete)'
> > Cc: 'Tim Mooney'; 'Ralph Lange (BESSY)'; 
> > tech-talk@aps.anl.gov; 'Ned Arnold'; 'Andrew Johnson'
> > Subject: RE: gateway enum writes
> > 
> > All,
> > 
> >      First I apologize for not giving this my full attention. 
> >  I have been
> > out of town, the Gateway is no longer my primary 
> > responsibility, and there
> > are only so many hours in the day.
> > 
> >      On looking at the original message I can guess that you 
> > are using a
> > Joerger scaler.  The Joerger scaler does not play well with 
> > channel access.
> > It does not work well with MEDM, either on Solaris or Linux.  
> > I investigated
> > this some time ago, to the extent I was convinced it was not 
> > MEDM.  The beam
> > line people that were using it at that time did not have 
> > enough expertise to
> > determine the real cause of the problem.  The same problem 
> > has appeared at
> > various sites since.
> > 
> >      Puts to the Gateway come from the Gateway's clients (EDM 
> > in this case).
> > Straight puts go through, puts with callback are queued.  (I 
> > have not looked
> > at the code again, as Jeff stated.)  I would expect EDM is 
> > doing a put with
> > callback.  I tested yesterday and had no trouble putting enum 
> > values such as
> > SCAN rates through the Gateway.  Moreover, this is done here 
> > on a regular,
> > daily basis without problems, and has been tested extensively 
> > in the past.
> > 
> >      My recollection is that the workaround for the Joerger 
> > scaler is to
> > push the buttons alternately to get it to be done.  You can 
> > also have two
> > screens and use the buttons on both of them.  You'll have to 
> > experiment.
> > 
> >      Thus I think the problem is with the instrument rather than the
> > Gateway.  You will probably have problems without a Gateway.  
> > Having said
> > this, it should not be giving Virtual circuit unresponsive.  
> > VCR's have been
> > an ongoing problems with CA since they were introduced.  (On 
> > the positive
> > side, they solve some significant problems.)  I made a fix to 
> > some of the
> > VCR problems, which I tested.  Jeff fixed it in another way 
> > for 3.14.8,
> > which I have not tested, but I have no evidence here that 
> > Jeff's fix is not
> > working, and I would expect it to be.  Note that it is the 
> client that
> > decides it has a VCR.  In many cases there was nothing wrong 
> > with the server
> > (Gateway in this case).
> > 
> > 	-Ken
> > 
> > -----Original Message-----
> > From: Jeff Hill [mailto:johill@lanl.gov] 
> > Sent: Wednesday, March 29, 2006 11:36 AM
> > To: 'Kenneth Evans, Jr.'; 'Dirk Zimoch'; 'Leicester, PJ (Pete)'
> > Cc: 'Tim Mooney'; 'Ralph Lange (BESSY)'; tech-talk@aps.anl.gov
> > Subject: RE: gateway enum writes
> > 
> > 
> > Ok, so where are we. 
> > 
> > I think I understand that:
> > 
> > 1) Strange behavior occurs only when the GW is attached to a 
> > PV that takes a
> > long time to complete a put callback.
> > 
> > 2) Ken looked in the GW source code and verified that a put 
> > callback request
> > will postponed, when a put callback for that PV is already in 
> > progress,
> > until that put callback in progress completes.
> > 
> > Ken, perhaps you attempt to reproduce this issue against a 
> > record of this
> > type (with a very long put callback delay) at the APS?
> > 
> > Jeff
> > 
> > > -----Original Message-----
> > > From: Kenneth Evans, Jr. [mailto:evans@aps.anl.gov]
> > > Sent: Wednesday, March 29, 2006 9:04 AM
> > > To: 'Jeff Hill'; 'Dirk Zimoch'; 'Leicester, PJ (Pete)'
> > > Cc: 'Tim Mooney'; tech-talk@aps.anl.gov
> > > Subject: RE: gateway enum writes
> > > 
> > > > One could speculate that the original GW design might 
> > have intended to
> > > > perform all puts with completion notification in order to 
> > know exactly
> > > when
> > > > to update the GW's cache.
> > > 
> > > > One could further speculate that a solution, for the GW, 
> > might be to not
> > > > start a write for a channel if a write is already 
> > outstanding on that
> > > same
> > > > channel. Instead the GW might save the value from the 
> > last write request
> > > for
> > > > a busy channel and initiate another write request using 
> > this saved value
> > > as
> > > > soon as the outstanding write request completes.
> > > 
> > >      That is what it does.
> > > 
> > > 	-Ken
> > > 
> > > -----Original Message-----
> > > From: Jeff Hill [mailto:johill@lanl.gov]
> > > Sent: Wednesday, March 29, 2006 9:57 AM
> > > To: 'Dirk Zimoch'; 'Leicester, PJ (Pete)'
> > > Cc: 'Tim Mooney'; tech-talk@aps.anl.gov; 'Ken Evans'
> > > Subject: RE: gateway enum writes
> > > 
> > > 
> > > Here my two centi-euro's worth.
> > > 
> > > > if (ctx.getMsg()->m_cmmd == CA_PROTO_WRITE_NOTIFY) {
> > > >    docallback == GATE_DOCALLBACK;
> > > > } else {
> > > >    docallback == GATE_NOCALLBACK;
> > > > }
> > > 
> > > The context (in this case the variable is named ctx) is a 
> > class whose
> > > definition is private to the portable server and subject to 
> > change in the
> > > future. Its header file is typically not installed for 
> > public consumption.
> > > We would not want, for example, to have the gateway code 
> become too
> > > dependent on the internal details of the server or its 
> > protocol. That
> > > might
> > > create maintenance headaches.
> > > 
> > > One could speculate that the original GW design might have 
> > intended to
> > > perform all puts with completion notification in order to 
> > know exactly
> > > when
> > > to update the GW's cache.
> > > 
> > > Nevertheless, use of put callback just might cause behavior 
> > problems for
> > > the
> > > gateway if the destination record takes a long time to 
> > complete the write
> > > request, and a 2nd write is started while the GW has a 
> > write to the same
> > > record in progress. That would block the incoming protocol 
> > stream from the
> > > GW to the IOC (for all channels on that IOC). The CA server 
> > is unable to
> > > initiate another put notify request with this record until 
> > it completes
> > > the
> > > put notify request that is pending, and in that situation 
> > the current
> > > server
> > > design suspends processing of incoming requests until it 
> > can initiate the
> > > put notify request (when the record completes processing).
> > > 
> > > One could further speculate that a solution, for the GW, 
> > might be to not
> > > start a write for a channel if a write is already 
> > outstanding on that same
> > > channel. Instead the GW might save the value from the last 
> > write request
> > > for
> > > a busy channel and initiate another write request using 
> > this saved value
> > > as
> > > soon as the outstanding write request completes.
> > > 
> > > Jeff
> > > 
> > > > -----Original Message-----
> > > > From: Dirk Zimoch [mailto:dirk.zimoch@psi.ch]
> > > > Sent: Wednesday, March 29, 2006 4:41 AM
> > > > To: Leicester, PJ (Pete)
> > > > Cc: Tim Mooney; tech-talk@aps.anl.gov; Ken Evans
> > > > Subject: Re: gateway enum writes
> > > >
> > > > This is from the gateway code (gateVc.cc):
> > > >
> > > > caStatus gateVcData::write(const casCtx& ctx, const gdd& 
> > dd, gateChan
> > > > &/*chan*/)
> > > > {
> > > > 	int docallback=GATE_DOCALLBACK;
> > > >
> > > > [...]
> > > > 	switch(at) {
> > > > [...]
> > > > 	case gddAppType_ackt:
> > > > 	case gddAppType_acks:
> > > > 		docallback = GATE_NOCALLBACK;
> > > > 		// Fall through
> > > > 	default:
> > > > [...]
> > > > 			caStatus stat = pv->put(&dd, 
> docallback);
> > > > 			if(stat != S_casApp_success) 
> return stat;
> > > >
> > > > 			if(docallback) {
> > > > 				// Start a pending write
> > > > #if DEBUG_GDD
> > > > 				fflush(stderr);
> > > > 				printf("pending_write\n");
> > > > 				fflush(stdout);
> > > > #endif
> > > > 				pending_write = new
> > > gatePendingWrite(*this,ctx,dd);
> > > > 				if(!pending_write) return 
> > S_casApp_noMemory;
> > > > 				else return 
> S_casApp_asyncCompletion;
> > > > 			} else {
> > > > 				return S_casApp_success;
> > > > 			}
> > > >
> > > > It seems that ALL puts except alarm handler ackt and acks 
> > are done with
> > > > callback. There is no direct flag to write() from the 
> > generic CA server
> > > > which indicates writes with or without callback, as far 
> > as I can see.
> > > > But the gateway should have look at the CA command in 
> > ctx, probably like
> > > > this:
> > > >
> > > > if (ctx.getMsg()->m_cmmd == CA_PROTO_WRITE_NOTIFY) {
> > > >    docallback == GATE_DOCALLBACK;
> > > > } else {
> > > >    docallback == GATE_NOCALLBACK;
> > > > }
> > > >
> > > >
> > > > Dirk
> > > >
> > > >
> > > > Leicester, PJ (Pete) wrote:
> > > > > I can confirm that the problem appears isolated to the 
> > scaler record
> > > CNT
> > > > > field and not enums in general.
> > > > >
> > > > > It looks like Tim's ca_put_callback theory may be 
> > correct. Is there a
> > > > > Gateway expert out there who can confirm this? How are 
> > writes done
> > > from
> > > > > the Gateway. Does it always use ca_put_callback 
> > regardless of how the
> > > > > put reached the gateway?
> > > > >
> > > > > Thanks for everyones responses on this.
> > > > >
> > > > > Pete Leicester
> > > > > Diamond
> > > > >
> > > > > -----Original Message-----
> > > > > From: Tim Mooney [mailto:mooney@aps.anl.gov]
> > > > > Sent: 28 March 2006 21:07
> > > > > To: Leicester, PJ (Pete)
> > > > > Cc: tech-talk@aps.anl.gov
> > > > > Subject: Re: gateway enum writes
> > > > >
> > > > >
> > > > > Leicester, PJ (Pete) wrote:
> > > > >
> > > > >>I am getting some strange behaviour when writing 
> > enumerations through
> > > > >>the Gateway (version 2.0.0.0 on 3.14.8.2 and RedHat 
> > Enterprise 4).
> > > > >>
> > > > >>The problem first showed itself in edm when pressing a 
> > button on a edm
> > > > >>screen to send the value resulted in the following error:
> > > > >>
> > > > >>    
> > CA.Client.Exception...............................................
> > > > >>        Warning: "Virtual circuit unresponsive"
> > > > >>        Context: "diamrs0005l.diamond.ac.uk:6064"
> > > > >>        Source File: ../tcpiiu.cpp line 896
> > > > >>        Current Time: Tue Mar 28 2006 16:19:01.334736000
> > > > >>    
> > ..................................................................
> > > > >>
> > > > >>I did some further tests using caput with similar results:
> > > > >>
> > > > >>    [pjl45@pc0005 pjl45]$ caput  -w15 GDA:scaler2.CNT Count
> > > > >>    Old : GDA:scaler2.CNT                Done
> > > > >>    Read operation timed out: PV data was not read.
> > > > >>    New : GDA:scaler2.CNT
> > > > >>    
> > CA.Client.Exception...............................................
> > > > >>        Warning: "Virtual circuit disconnect"
> > > > >>        Context: "op=0, channel=GDA:scaler2.CNT, 
> > type=DBR_TIME_STRING,
> > > > >>count=1, ctx="diamrs0005l.diamond.ac.uk:6064""
> > > > >>        Source File: ../getCopy.cpp line 82
> > > > >>        Current Time: Tue Mar 28 2006 17:04:24.488496000
> > > > >>.
> > > > >>Despite the above error message the write does actually 
> > reach the IOC.
> > > > >>However if I now try the change the value back as follows:
> > > > >>
> > > > >>    [pjl45@pc0005 pjl45]$ caput  -w15 GDA:scaler2.CNT Done
> > > > >>    Old : GDA:scaler2.CNT                Count
> > > > >>    Read operation timed out: PV data was not read.
> > > > >>    New : GDA:scaler2.CNT
> > > > >>    
> > CA.Client.Exception...............................................
> > > > >>        Warning: "Virtual circuit disconnect"
> > > > >>        Context: "op=0, channel=GDA:scaler2.CNT, 
> > type=DBR_TIME_STRING,
> > > > >>count=1, ctx="diamrs0005l.diamond.ac.uk:6064""
> > > > >>        Source File: ../getCopy.cpp line 82
> > > > >>        Current Time: Tue Mar 28 2006 17:22:36.167980000
> > > > >>    
> > ..................................................................
> > > > >>
> > > > >>I get an error again however this time the 'Done' value 
> > gets written
> > > > >>to
> > > > >>the IOC exactly ONE MINUTE after I entered the caput 
> > command. This is
> > > > >>long after the caput command has timed out so it 
> > appears the gateway
> > > > >
> > > > > is
> > > > >
> > > > >>responsible for the delay?
> > > > >>
> > > > >>Has anyone any idea what may be happening? Is there a 
> 60 second
> > > > >>timeout
> > > > >>in CA or the gateway which may give a clue as to what I 
> > am seeing?
> > > > >>
> > > > >>(For the record this test was done with a very lightly 
> > loaded test
> > > > >>gateway serving only 20 or so PV's. Also using caput 
> > -w70 results in
> > > > >
> > > > > the
> > > > >
> > > > >>same timeouts)
> > > > >
> > > > >
> > > > > Do you get this kind of result with *all* enum fields 
> > that you write
> > > to
> > > > > through the gateway, or only this one?  The scaler 
> > record's CNT field
> > > is
> > > > > different from other enum's in that it starts an 
> > operation that may
> > > take
> > > > > a long time to complete.  I don't know what the gateway 
> > is using to do
> > > > > put's, but if it should happen to be ca_put_callback(), 
> > the gateway
> > > may
> > > > > be waiting for a callback that will come only after the 
> > scaler has
> > > > > finished counting.
> > > > >
> > > >
> > > > --
> > > > Dr. Dirk Zimoch
> > > > Swiss Light Source
> > > > Paul Scherrer Institut
> > > > Computing and Controls
> > > > phone +41 56 310 5182
> > > > fax   +41 56 310 4413
> > 
> > 
> > 
> > 
>