Re: Darwin and EPICS_CA_AUTO_ADDR_LIST issue

[Burkhard Kolb <b.kolb@gsi.de>]

Jeff Hill wrote:
> > I use the same list of IOCs on other linux boxes and have no problem
> > there.
>
> This might be because what UDP responses are discarded off the end of the
> finite length UDP queue might be highly dependent on what hosts manage to
> jamb their responses in first - and therefore on the topology of the
> network.
>
> > Anyhow this IOC is in the list but is actually really down - on purpose.
>
> So, I still think it would be good idea to peek under the hood at the ICMP
> traffic in case something unusual is going on (its not hard to do), but this
> may only reveal that there is one ICMP response reflected off of each host
> that isn't currently available.
>
> > We run a flat switched network.
>
> This probably means of course that the control system is all contained
> within one subnet? If so then perhaps you can get by using ordinary
> broadcasted search requests? Listing each host in the EPICS_CA_ADDR_LIST
> certainly does have the disadvantage of competition from ICMP responses when
> the routing system knows that the host that is being sent a search message
> is unreachable. One wouldn't generally have that specific problem with
> broadcast based search messages.
>
> Do you have an important reason to list the hosts individually in the
> EPICS_CA_ADDR_LIST?
Well...
As our network people still refuse to have routed networks here we have 
one large switched network with several thousand nodes. Many of them 
constantly produce broadcast messages contributing to quite some load.
My concern was really the difference between Macs and the standard linux 
boxes here, where I don't see this behaviour.
Maybe some of the Mac enthusiasts out there have an idea?

> Are you listing specific hosts in the EPICS_CA_ADDR_LIST in order to avoid
> contacting certain hosts on the network? If so, one solution might be to use
> a different UDP port for a subsystem of the overall control system. You
> could possibly also use a ca gateway to selectively couple between control
> subsystems operating on different ports. This might transparently provide
> the appearance of one control system operating on a single port to certain
> clients even though there are actually multiple underlying control systems
> operating on independent ports. 
>
> Multicast groups might provide a better way to manage grouping of hosts
> independent of subnets - in contrast to using different UDP ports. This
> isn't implemented yet but it's probably not a large amount of programming to
> add that feature. The primary benefit would be avoidance of configuration
> complexity - especially if the control system crosses over multiple subnets.
>
> Jeff
>
> > -----Original Message-----
> > From: Burkhard Kolb [mailto:b.kolb@gsi.de]
> > Sent: Friday, March 09, 2007 2:02 AM
> > To: Jeff Hill
> > Cc: tech-talk@aps.anl.gov
> > Subject: Re: Darwin and EPICS_CA_AUTO_ADDR_LIST issue
> >
> > Jeff Hill wrote:
> > > > On my MAC (OSX 10.4.8), EPICS base-3.14.9, medm 3.11:
> > > >
> > > > When I set EPICS_CA_AUTO_ADDR_LIST to NO and have the
> > EPICS_CA_ADDR_LIST
> > > > pointing to the list of IOCs medm does not find all PVs. Sometimes the
> > > > connections work for a short time then several IOCs are not seen
> > anymore.
> > > > If I set it to YES, medm finds them always.
> > > >
> > > > In the xterm window I get some error messages from really not existing
> > > > IOCs:
> > > > CAC: error = "Host is down" sending UDP msg to 140.181.98.50:5064
> > > > But the lost ones are not reported!
> > > >
> > > > I use the same list of IOCs on other linux boxes and have no problem
> > > there.
> > >
> > > What is the response to "ping -s 140.181.98.50"?
> > ping: invalid packet size: `140.181.98.50'
> > Did you mean -R or -r?
> > Anyhow this IOC is in the list but is actually really down - on purpose.
> > But most of the other ones are VxWorks 68040 cpus all running the same
> > kernel, same setup...
> > What bothers me most, is that sometimes when I start a medm with a
> > screen where vxStats infos from the IOCs is displayed, the IOCs show up
> > and then drop out - in the case when EPICS_CA_AUTO_ADDR_LIST=NO.
> > > PS: Behavior is known to vary between IP kernels for such unicast
> > addresses
> > > if there are two or more daemons (in this case two or more CA UDP
> > servers)
> > > listening to the same {IP address, port} tuple on the same host. The
> > typical
> > > behavior difference is that on certain IP kernels only one daemeon
> > listening
> > > on UDP port 5064 will receive a UDP message with a unicast destination
> > > address, but in contrast on other IP kernels all daemeons listening on
> > UDP
> > > port 5064 will receive such messages. In my experience UDP frames with
> > > broadcast address destinations always go to all registered listeners
> > > listening on UDP port 5064 - a uniform behavior across IP kernel
> > > implementations.
> > >
> > > PPS: One possible workaround to the above dilemma is to use directed
> > > broadcast addresses (router forwarded broadcast addresses) in the
> > > EPICS_CA_ADDR_LIST. This sometimes requires enabling of "broadcast
> > > forwarding" features in your routers. We might also add support for
> > > multicasting to future versions of CA.
> > We run a flat switched network.
> >
> > > Jeff
> > >
> > > > -----Original Message-----
> > > > From: Burkhard Kolb [mailto:b.kolb@gsi.de]
> > > > Sent: Thursday, March 08, 2007 9:28 AM
> > > > To: 'tech-talk@aps.anl.gov'
> > > > Subject: Darwin and EPICS_CA_AUTO_ADDR_LIST issue
> > > >
> > > > On my MAC (OSX 10.4.8), EPICS base-3.14.9, medm 3.11:
> > > >
> > > > When I set EPICS_CA_AUTO_ADDR_LIST to NO and have the
> > EPICS_CA_ADDR_LIST
> > > > pointing to the list of IOCs medm does not find all PVs. Sometimes the
> > > > connections work for a short time then several IOCs are not seen
> > anymore.
> > > > If I set it to YES, medm finds them always.
> > > >
> > > > In the xterm window I get some error messages from really not existing
> > > > IOCs:
> > > > CAC: error = "Host is down" sending UDP msg to 140.181.98.50:5064
> > > > But the lost ones are not reported!
> > > >
> > > > I use the same list of IOCs on other linux boxes and have no problem
> > > there.
> > > > Any idea?
> > > > --
> > > > -------------------------------------------------------------------
> > > > Dr. Burkhard Kolb
> > > > GSI mbH  |   KP1   |  Planckstr. 1  |  D-64291 Darmstadt
> > > > Email: B.Kolb@gsi.de                |  Tel.: +49 (0)6159 / 71 2667
> > > > -------------------------------------------------------------------
> >
> > --
> > -------------------------------------------------------------------
> > Dr. Burkhard Kolb
> > GSI mbH  |   KP1   |  Planckstr. 1  |  D-64291 Darmstadt
> > Email: B.Kolb@gsi.de                |  Tel.: +49 (0)6159 / 71 2667
> > -------------------------------------------------------------------


--
-------------------------------------------------------------------
Dr. Burkhard Kolb
GSI mbH  |   KP1   |  Planckstr. 1  |  D-64291 Darmstadt
Email: B.Kolb@gsi.de                |  Tel.: +49 (0)6159 / 71 2667
-------------------------------------------------------------------