RE: CAC problem between RTEMS and vxWorks

["Hill, Jeff" <johill@lanl.gov>]

> Jeff:
> 
> Dynamic re-assign means in CA terms that the sequencer will issue a
> ca_clear_channel (for some connected channel), immediately followed by a
> ca_create_channel (to some other PV). 

Be careful about race conditions where a thread continues to use a chid 
after the channel that the chid references has been destroyed.

> Hmmm, what happens if there is a
> race between the ca_clear_channel call and the disconnect event caused by the
> IOC reboot? (Just think loud here.)
> 

Reading Wesley's later post I think we can assume that it's the client (i.e. 
sequencer) side that is being rebooted.

> It should be possible to devise a simple test client that issues a long
> sequence of such create/clear calls to check whether this is a CA problem.
> Unfortunately my automatic test setup does not yet support RTEMS, so this
> will take some time for me to do.

I had a look at the CA client side regression tests and I don't see a test
of this nature. Presumably such a test would create a few thousand channels
and then begin destroying them while at least some of them are in the process of connecting. This would be perhaps an unusual test in that we aren't testing 
any metric other than that the test program can survive a somewhat unusual 
sequence of events, and because successfully testing of anything at all might
be highly dependent on the various ratios of server/client cpu speeds and 
the speed of the network.

Considering this particular situation further. All we know is that we have 
an RTEMS IOC with a sequencer CA client that is not reconnecting reliably 
when the RTEMS IOC undergoes a rapid (close together in time) series of 
reboots. One has to guess that the vxWorks IOC is running low on some
resource perhaps its network buffers, file descriptors, ... 

Wesley:
I think that we see evidence that the vxWorks IOC is taking some time 
to determine that the previous lifetimes of the RTEMS IOC have passed out
of existence (since the RTEMS IOC didn't cleanup its CA circuits when it 
was shutdown). Have you seen the situation improve if you leave it for a 
sufficiently long interval? This would be long enough for the TCP watchdog 
timers to fire; that delay varies between different OS and different OS 
versions but is typically on the order, as I recall, of about 20 minutes.

Jeff

> 
> On Friday, September 21, 2012 11:37:03 Wesley Moore wrote:
> > From: "J. Lewis Muir" <jlmuir@imca-cat.org>
> > > There have been a number of bug fixes related to pvAssign.  You
> > > can find these in the release notes (just search in the page for
> > >
> > > "pvAssign"):
> > >   http://www-csr.bessy.de/control/SoftDist/sequencer/Notes.html
> > >
> > > What version of the sequencer are you using?
> >
> > I'm using 2-1-9.
> 
> In 2.1.9 all issues know to me have been resolved.
> 
> > Looking through all the release notes and also seq_if.c.
> > I'm thinking of trying to de-assign the channel between uses.
> 
> As a short-term work-around, this seem sensible.
> 
> Cheers
> Ben
> 
> --
> Ben Franksen
> ()  ascii ribbon campaign - against html e-mail
> /\  www.asciiribbon.org   - against proprietary attachments