Experimental Physics and
Industrial Control System

"Jeff Hill" <[email protected]> · Thu, 12 Mar 2009 11:54:25 -0600

Hi Andrew,

> Could this be the beacon port network order problem that you fixed
> recently?
> That might explain why she's seeing no beacons from the gateway.

I considered this possibility also.

Mantis entry 331 definitely might explain an absence of beacons from the GW
however it isn't a good match with their primary symptom.

my memory is that Mark's symptoms were as follows
-------------------------------------------------
O The CA server (IOC or GW) must be running R3.14.10
O The CA server (IOC or GW) must run on little-endian (i.e. Intel) processor
O The CA client library's channel connect delays are normal when initially
creating the channel. They had to restart the application to get an
immediate reconnect.
O The CA client library's channel re-connection delays were abnormally on
the order of 0-15 minutes but not longer - depending on where the IOC reboot
is synchronized with the CA client's periodic search request. In this
situation they are _not_ restarting the client application.

in contrast, at Diamond they have these symptoms
------------------------------------------------
O The CA client library's channel connect delays are normal when rebooting
the IOC and as a consequence also restarting the TCP circuit.
O The CA client library does not reconnect the channel when the link changes
to another type of link and back to a CA link. This would cause the CA
channel to be destroyed and then recreated in the CA client library, but the
TCP circuit might not be restarted unless said channel was the last channel
on the circuit. When you recreate the CA channel its search interval is
initially quite fast (on the order of 100's of mS), but with an exponential
back-off.
O They have a persistent disconnected DB CA link on the order of several
days. 

So, based on the time period that the channel remains disconnected, the fact
that it doesn't reconnect when it is recreated, and assuming also that they
don't have a very large non-default value for EPICS_CA_MAX_SEARCH_PERIOD in
the IOC with the persistently disconnected link, then I wouldn't expect that
Mantis entry 331 would fix their issue. 

Jeff

> -----Original Message-----
> From: Andrew Johnson [mailto:[email protected]]
> Sent: Thursday, March 12, 2009 10:43 AM
> To: [email protected]
> Cc: Jeff Hill; 'Shepherd, EL (Emma)'
> Subject: Re: Inter-IOC link problems
> 
> Jeff,
> 
> Could this be the beacon port network order problem that you fixed
> recently?
> That might explain why she's seeing no beacons from the gateway.
> 
> - Andrew
> 
> On Thursday 12 March 2009 11:12:56 Jeff Hill wrote:
> > Emma,
> >
> > > There doesn't seem to be any other obvious problems that I can see
> (CPU
> > > usage very low) - I've attached some of the console output.  I did a
> "tt"
> > > on the dbca link thread but wasn't sure where to go from there - is
> there
> > > anything else I should try before I reboot the IOC?
> >
> > First try to determine if it's an IP kernel related issue (you should
> see
> > some aspects of TCP/UDP that are not working using protocols that are
> not
> > CA if it's an IP kernel issue). Does telnet (verifying TCP) and ping
> > (verifying IP) work with the IOC when it is in this state? If your
> vxWorks
> > system has an echo server (listening on port seven) you could test UDP
> with
> > that.
> >
> > Here is a talk by Dave Thomson (which has some info on diagnosing
> vxWorks
> > buffer starvation related issues).
> >
> >
> http://www.diamond.ac.uk/CMSWeb/Downloads/diamond/Events/EPICS/MBUF_Proble
> m
> >s .ppt
> >
> > This one might help also.
> >
> > http://www.xs4all.nl/~borkhuis/vxworks/troubleshooting.txt
> >
> > And here is some info on how to configure vxWorks to run EPICS.
> >
> > http://www.aps.anl.gov/epics/base/tornado.php?format=printer
> >
> >
> > The output from ifShow, endPoolShow("name", 0), netStackDataPoolShow(),
> > netStackSysPoolShow(), and maybe also udpStatShow are probably most
> likely
> > to provide some hints at the cause of your problems if you are
> experiencing
> > troubles with the vxWorks IP kernel (or below). The output from ifShow
> can
> > be very interesting if there are low level media transmission errors.
> >
> > Look at the output from inetStatShow. In particular, look at TCP
> circuits
> > that consistently indicate the same large number of bytes pending in
> their
> > buffers (in multiple samples dumped with inetStatShow). Pending output
> > bytes can indicate congestion problems with the IP kernel, network,
> routing
> > system, and or the server (possibly a CA server (GW or IOC) this IOC is
> > connected to). Pending input bytes usually indicate issues with the code
> > consuming bytes from the socket (in this case the CA client library).
> >
> > > > I would also look very closely at the output from dbcar at higher
> > > > interest levels. As the interest level increases you should be able
> to
> > > > see if CA thinks that the channel is connected or not (the output
> from
> > > > void nciu::show ()). Of particular interest would be any situations
> > > > where CA thinks the channel is connected, but the DB CA link code
> does
> > > > not.
> > > > Also look for situations where the DB CA Link code thinks that it's
> a
> > > > CA link, but the CA channel hasn't been created (yet).
> >
> > I would definitely dump the output of dbcar when specifying a very high
> > magnitude interest level (a level of 1000 should be sufficient) so that
> you
> > see all of the gory details. We need to fault isolate so look for
> > situations where the CA client library marks a particular channel as
> being
> > connected, but the db ca link facility marks this channel as being
> > disconnected. Also look for situations where a channel hasn't been
> created
> > in the CA client library, but the db ca link facility considers the link
> to
> > be a CA link, and of course the third possibility would be that the
> channel
> > exists in the CA client library and both the CA client library and the
> DB
> > CA link facility consider the channel to be disconnected.
> >
> > If you can somehow capture the entire output from dbcar at interest
> level
> > 1000, and send it to me in an email, I would be happy to have a look.
> One
> > possibility would be to forward the output of the vxWorks command to a
> > file. Also send the name of the channels that should be connected, but
> > aren't.
> >
> > It will be time consuming, but you might also capture a tt from the
> thread
> > running the db ca link facility, and hopefully also all of the threads
> > managing the CA client context created for the db ca link facility. If
> you
> > could send that information I might be able to determine what has
> happened.
> > The tornado, host based debugging system, might help to automate the
> stack
> > trace collection process.
> >
> > Jeff
> >
> > > -----Original Message-----
> > > From: Shepherd, EL (Emma) [mailto:[email protected]]
> > > Sent: Tuesday, March 10, 2009 9:12 AM
> > > To: Jeff Hill
> > > Subject: RE: Inter-IOC link problems
> > >
> > > Hi Jeff,
> > >
> > > You may remember this problem I reported on tech-talk a little while
> ago.
> > > It has occurred again, and I have managed to do a little more
> debugging.
> > > I loaded a standalone CA client as you suggested and it works fine, so
> it
> > > appears that it is not a global CA issue.
> > >
> > > There doesn't seem to be any other obvious problems that I can see
> (CPU
> > > usage very low) - I've attached some of the console output.  I did a
> "tt"
> > > on the dbca link thread but wasn't sure where to go from there - is
> there
> > > anything else I should try before I reboot the IOC?
> > >
> > > Thanks again for your help,
> > >
> > > Emma
> > >
> > > Emma Shepherd
> > > Software Systems Engineer
> > > Beamline Controls - I06, I07, I24
> > >
> > > +44 (0)1235-778235
> > > http://www.diamond.ac.uk
> > >
> > > > -----Original Message-----
> > > > From: Jeff Hill [mailto:[email protected]]
> > > > Sent: 20 October 2008 17:24
> > > > To: Shepherd, EL (Emma); [email protected]
> > > > Subject: RE: Inter-IOC link problems
> > > >
> > > >
> > > > Presumably, the IP stack on this IOC is operating correctly when
> this
> > > > happens - as verified by {telnet, ping, ifShow, ...}?
> > > >
> > > > When this occurs, you might try running a small standalone CA client
> > > > that you have dynamically loaded into vxWorks. Its best to spawn
> this
> > > > type of client so that a CA context will not end up getting attached
> > > > to the vxWorks shell. The intent of course would be to isolate
> between
> > > > a global CA issue, and one that is isolated to the CA client / DB CA
> > > > Link code combination.
> > > >
> > > > I would also look very closely at the output from dbcar at higher
> > > > interest levels. As the interest level increases you should be able
> to
> > > > see if CA thinks that the channel is connected or not (the output
> from
> > > > void nciu::show ()). Of particular interest would be any situations
> > > > where CA thinks the channel is connected, but the DB CA link code
> does
> > > > not.
> > > > Also look for situations where the DB CA Link code thinks that it's
> a
> > > > CA link, but the CA channel hasn't been created (yet).
> > > >
> > > > Also, do a "tt" on the DBCA Link thread, and the satellite threads
> for
> > > > its CA context. Look for any situations where threads are hanging
> > > > around in unusual places which might indicate some form of deadlock.
> > > > If you see anything out of the ordinary please send the tt output
> and
> > > > I will have a look. In lightly loaded situations, "out of the
> > > > ordinary"
> > > > usually means a thread that isn't parked in the normal place (as
> seen
> > > > by snapshots with tt) for an extended length of time. One of course
> > > > needs to compare tt output from when the IOC is normal to tt output
> > > > from when the IOC is misbehaving.
> > > > Needless to say, a CPU starvation situation on this IOC would also
> > > > cause issues (could be the cause of your issue).
> > > >
> > > > In the past, quite some years back actually, I have seen UDP issues
> if
> > > > there were too many machines on a network with the wrong subnet mask
> > > > configuration. I think that there used to be some issues in
> particular
> > > > with HP workstations because they would reply with "ICMP network
> > > > unreachable" if their network mask was set incorrectly and this
> could
> > > > cause the IOC's search response to be discarded off the end of the
> > > > finite length UDP input queue (depending on which response got there
> > > > first and how many bogus ICMP messages are sent in response to each
> > > > search request). ICMP traffic can be seen with Ethernet snoopers
> like
> > > > wireshark or tcpdump. However, on modern switched networks, it may
> be
> > > > best to be on the same hub (not a switch) with the IOC so that you
> can
> > > > see unicast traffic that the switch sends only between the IOC and
> its
> > > > message peers. Admittedly, this is perhaps contraindicated based on
> > > > your not seeing any search traffic from the IOC in casnooper.
> > > >
> > > > You might have a look at the output from utpStatShow (presuming that
> > > > something is wrong with UDP and not IP).
> > > > Also, have a look at ifShow and verify that the broadcast address
> > > > remains correctly configured, and that there are not high error
> rates.
> > > >
> > > > Jeff
> > > >
> > > > > -----Original Message-----
> > > > > From: [email protected]
> > > > > [mailto:[email protected]]
> > > > > On Behalf Of Shepherd, EL (Emma)
> > > > > Sent: Friday, October 17, 2008 9:12 AM
> > > > > To: [email protected]
> > > > > Subject: RE: Inter-IOC link problems
> > > > >
> > > > > I've done a little more investigation and I think that in this
> case
> > > > > the gateway is not to blame.  It seems that other CA links
> > > >
> > > > on this IOC
> > > >
> > > > > are also not working, and they are not all going through
> > > >
> > > > the gateway
> > > >
> > > > > (some are on other IOCs on the same network).
> > > > >
> > > > > I setup caSnooper to monitor connection requests on one of
> > > >
> > > > the PVs my
> > > >
> > > > > IOC is failing to link to.  When I change the link to a
> > > >
> > > > constant and
> > > >
> > > > > change it back again, caSnooper does not report any new
> > > >
> > > > requests for
> > > >
> > > > > the PV.  However when I do the same on a 'healthy' IOC which has
> > > > > working links, I see the new request on caSnooper when I
> > > >
> > > > put the link
> > > >
> > > > > back.
> > > > >
> > > > > I'm not sure what that tells me except that it looks like
> > > >
> > > > the IOC has
> > > >
> > > > > somehow stopped broadcasting search requests..?
> > > > >
> > > > > Emma
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: [email protected]
> > > > > > [mailto:[email protected]] On Behalf Of Shepherd, EL
> > > > > > (Emma)
> > > > > > Sent: 17 October 2008 12:28
> > > > > > To: Ralph Lange
> > > > > > Cc: [email protected]
> > > > > > Subject: RE: Inter-IOC link problems
> > > > > >
> > > > > >
> > > > > > Hi there,
> > > > > >
> > > > > > Thanks for the replies, it seems that the 'undefined' entry
> might
> > > > > > have been a red herring.
> > > > > >
> > > > > > The IOC I am looking at is the client of the PV
> > > >
> > > > connection, and the
> > > >
> > > > > > IP address listed is the server side of the CA gateway.
> > > >
> > > > There are
> > > >
> > > > > > in fact two gateways on this machine - one for each
> > > >
> > > > direction as you
> > > >
> > > > > > suggested. The configuration is really very simple, it is
> > > >
> > > > setup to
> > > >
> > > > > > allow read access for all PVs.  Do you need to know anything
> more
> > > > > > specific?
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Emma
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Ralph Lange [mailto:[email protected]]
> > > > > > > Sent: 17 October 2008 08:52
> > > > > > > To: Shepherd, EL (Emma)
> > > > > > > Cc: [email protected]
> > > > > > > Subject: Re: Inter-IOC link problems
> > > > > > >
> > > > > > >
> > > > > > > Hi Emma,
> > > > > > >
> > > > > > > I would need a bit more information about your setup to
> > > >
> > > > be able to
> > > >
> > > > > > > fully understand your report.
> > > > > > >
> > > > > > > You are looking at the CA client side of an IOC. When you are
> > > > > > > losing connections between IOCs, is the IOC you're
> > > >
> > > > looking at the
> > > >
> > > > > > > server or the client of that PV connection?
> > > > > > > It seems there are no beacons coming from the CA Gateway
> > > > > > > (172.23.106.35). Is that the client side or the server side
> > > > > >
> > > > > > of the CA
> > > > > >
> > > > > > > Gateway? Are two (or more) Gateway processes running on
> > > > > >
> > > > > > that machine
> > > > > >
> > > > > > > (i.e. one for each direction)? What is the CA configuration
> for
> > > > > > > the
> > > > > > > Gateway(s) on that machine?
> > > > > > >
> > > > > > > CA configuration of a Gateway is difficult and subtle.
> > > >
> > > > There are a
> > > >
> > > > > > > lot of environment variables for CA server and client (see the
> > > > > >
> > > > > > CA Manual)
> > > > > >
> > > > > > > which influence the behaviour of a CA application. Some
> > > > > >
> > > > > > variables are
> > > > > >
> > > > > > > using other variables' values as default, which simplifies
> > > > > > > configuration of pure CA client or server apps, but may lead
> to
> > > > > > > unwanted behaviour for a CA Gateway (whis is one of the few
> apps
> > > > > > > that is as well CA server and client). E.g, it is quite easy
> to
> > > > > > > create a setup where the
> > > > > >
> > > > > > Gateway is
> > > > > >
> > > > > > > sending out beacons on the wrong (i.e. client) side.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Ralph
> > > > > > >
> > > > > > > Shepherd, EL (Emma) wrote:
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > We still seem to suffer quite a bit from problems with
> > > > > > >
> > > > > > > database links
> > > > > > >
> > > > > > > > between IOCs, particularly when a gateway is
> > > >
> > > > involved.  For some
> > > >
> > > > > > > > reason the links can become disconnected and a reboot
> > > >
> > > > is usually
> > > >
> > > > > > > > necessary to get them working again.  I have just had an
> > > > > > >
> > > > > > > opportunity
> > > > > > >
> > > > > > > > to do some diagnosis on one such problem and found a clue
> > > > > >
> > > > > > in the CA
> > > > > >
> > > > > > > > beacon hashtable part of the dbcar report.  The entry for
> > > > > > >
> > > > > > > the gateway
> > > > > > >
> > > > > > > > (172.23.106.35) is 'undefined', although the gateway itself
> > > > > > >
> > > > > > > seems to
> > > > > > >
> > > > > > > > be working just fine and I can use caget through the
> > > >
> > > > gateway as
> > > >
> > > > > > > > normal.
> > > > > > > >
> > > > > > > > Any ideas what could cause this to happen, or how to fix
> > > > > >
> > > > > > it when it
> > > > > >
> > > > > > > > does?  None of the tasks are suspended, CPU usage is low and
> > > > > > > > everything else looks fine.
> > > > > > > >
> > > > > > > > CA beacon hash entry for 172.23.106.32:5064 with
> > > >
> > > > period estimate
> > > >
> > > > > > > > 15.000521
> > > > > > > >         beacon number 168436, on THU OCT 16 2008 14:27:46 CA
> > > > > > > > beacon hash entry for 172.23.106.35:5064 <no period
> estimate>
> > > > > > > >         beacon number 0, on <undefined> CA beacon hash entry
> > > > > > > > for 172.23.106.97:5064 with
> > > >
> > > > period estimate
> > > >
> > > > > > > > 14.988265
> > > > > > > >         beacon number 76356, on THU OCT 16 2008 14:27:52 CA
> > > > > > > > beacon hash entry for 172.23.106.96:5064 with period
> estimate
> > > > > > > > 14.988637
> > > > > > > >         beacon number 39491, on THU OCT 16 2008 14:27:53 CA
> > > > > > > > beacon hash entry for 172.23.106.98:5064 with period
> estimate
> > > > > > > > 14.980477
> > > > > > > >         beacon number 58989, on THU OCT 16 2008 14:27:47 CA
> > > > > > > > beacon hash entry for 172.23.106.102:5064 with period
> > > >
> > > > estimate
> > > >
> > > > > > > > 14.990867
> > > > > > > >         beacon number 39993, on THU OCT 16 2008 14:27:53 CA
> > > > > > > > beacon hash entry for 172.23.106.32:5064 with period
> estimate
> > > > > > > > 15.000521
> > > > > > > >         beacon number 168436, on THU OCT 16 2008 14:27:46 CA
> > > > > > > > beacon hash entry for 172.23.106.35:5064 <no period
> estimate>
> > > > > > > >         beacon number 0, on <undefined> CA beacon hash entry
> > > > > > > > for 172.23.106.97:5064 with
> > > >
> > > > period estimate
> > > >
> > > > > > > > 14.988265
> > > > > > > >         beacon number 76356, on THU OCT 16 2008 14:27:52 CA
> > > > > > > > beacon hash entry for 172.23.106.96:5064 with period
> estimate
> > > > > > > > 14.988637
> > > > > > > >         beacon number 39491, on THU OCT 16 2008 14:27:53 CA
> > > > > > > > beacon hash entry for 172.23.106.98:5064 with period
> estimate
> > > > > > > > 14.980477
> > > > > > > >         beacon number 58989, on THU OCT 16 2008 14:27:47 CA
> > > > > > > > beacon hash entry for 172.23.106.102:5064 with period
> > > >
> > > > estimate
> > > >
> > > > > > > > 14.990867
> > > > > > > >         beacon number 39993, on THU OCT 16 2008 14:27:53
> > > > > > >
> > > > > > > entries per
> > > > > > >
> > > > > > > > bucket: mean = 0.011719 std dev = 0.107617 max = 1
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks in advance....
> > > > > > > >
> > > > > > > > Emma
> > > > > >
> > > > > > <DIV><FONT size="1" color="gray">This e-mail and any
> > > >
> > > > attachments may
> > > >
> > > > > > contain confidential, copyright and or privileged
> > > >
> > > > material, and are
> > > >
> > > > > > for the use of the intended addressee only. If you are not the
> > > > > > intended addressee or an authorised recipient of the addressee
> > > > > > please notify us of receipt by returning the e-mail and
> > > >
> > > > do not use,
> > > >
> > > > > > copy, retain, distribute or disclose the information in
> > > >
> > > > or attached
> > > >
> > > > > > to the e-mail. Any opinions expressed within this e-mail are
> those
> > > > > > of the individual and not necessarily of Diamond Light Source
> Ltd.
> > > > > > Diamond Light Source Ltd. cannot guarantee that this e-mail or
> any
> > > > > > attachments are free from viruses and we cannot accept liability
> > > > > > for any damage which you may sustain as a result of software
> > > > > > viruses which may be transmitted in or with the message. Diamond
> > > > > > Light Source Limited (company no. 4375679).
> > > > > > Registered in England and Wales with its registered office at
> > > > > > Diamond House, Harwell Science and Innovation Campus, Didcot,
> > > > > > Oxfordshire, OX11 0DE, United Kingdom </FONT></DIV>
> > > > >
> > > > > <DIV><FONT size="1" color="gray">This e-mail and any
> > > >
> > > > attachments may
> > > >
> > > > > contain confidential, copyright and or privileged material, and
> are
> > > > > for
> > > >
> > > > the
> > > >
> > > > > use of the intended addressee only. If you are not the intended
> > > > > addressee or an authorised recipient of the addressee
> > > >
> > > > please notify us
> > > >
> > > > > of receipt by returning the e-mail and do not use, copy, retain,
> > > > > distribute or disclose the information in or attached to
> > > >
> > > > the e-mail.
> > > >
> > > > > Any opinions expressed within this e-mail are those of the
> > > >
> > > > individual
> > > >
> > > > > and not necessarily of Diamond Light Source Ltd. Diamond
> > > >
> > > > Light Source
> > > >
> > > > > Ltd. cannot guarantee that this e-mail or any attachments are free
> > > > > from viruses and we cannot accept liability for any damage
> > > >
> > > > which you
> > > >
> > > > > may sustain as a result of software viruses which may be
> > > >
> > > > transmitted
> > > >
> > > > > in or with the message. Diamond Light Source Limited (company no.
> > > > > 4375679). Registered in England and Wales with its
> > > >
> > > > registered office
> > > >
> > > > > at Diamond House, Harwell Science and Innovation Campus, Didcot,
> > > > > Oxfordshire, OX11 0DE, United Kingdom </FONT></DIV>
> > >
> > > <DIV><FONT size="1" color="gray">This e-mail and any attachments may
> > > contain confidential, copyright and or privileged material, and are
> for
> > > the use of the intended addressee only. If you are not the intended
> > > addressee or an authorised recipient of the addressee please notify us
> of
> > > receipt by returning the e-mail and do not use, copy, retain,
> distribute
> > > or disclose the information in or attached to the e-mail.
> > > Any opinions expressed within this e-mail are those of the individual
> and
> > > not necessarily of Diamond Light Source Ltd.
> > > Diamond Light Source Ltd. cannot guarantee that this e-mail or any
> > > attachments are free from viruses and we cannot accept liability for
> any
> > > damage which you may sustain as a result of software viruses which may
> be
> > > transmitted in or with the message.
> > > Diamond Light Source Limited (company no. 4375679). Registered in
> England
> > > and Wales with its registered office at Diamond House, Harwell Science
> > > and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
> > > </FONT></DIV>
> > >
> > > --
> > >
> > > Scanned by iCritical.
> 
> 
> 
> --
> The best FOSS code is written to be read by other humans -- Harold Welte

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System