EPICS Home

Experimental Physics and Industrial Control System


 
1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  <2011>  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  <2011>  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>
Subject: RE: CA beacon routing error S_errno_ECONNREFUSED
From: "Jeff Hill" <[email protected]>
To: "'J. Lewis Muir'" <[email protected]>, "'EPICS Tech Talk'" <[email protected]>
Date: Fri, 20 May 2011 10:08:57 -0600 
Hi Lewis,

The CA server's beacon thread uses the connect function call on its UDP
socket and thereby determines which network interface on the local system
will be used by the network kernel to send the udp frame to the "connected"
destination address. Once this network interface's address has been
obtained, it is included in the CA beacon protocol as the source address of
the server. It's this function call that is returning ECONNREFUSED, and
normally we expect an error to be returned there, if for example, there was
no route to the beacon's destination address (a different error number).

However, there is a low level error transmission internet protocol called
ICMP which transmits errors out of band from udp and tcp. It turns out that
BSD derived network kernels will sometimes return ECONNRESET errno from a
connected socket send/recv function call if that socket's source port was
the target of an ICMP "destination unreachable" ICMP error response in the
recent past. Perhaps your situation is similar but a slightly different
variant with the socket connect call returning ECONNREFUSED.

In any case, the indirect cause might be an explicit unicast address in
EPICS_CAS_BEACON_ADDR_LIST (defaulting to EPICS_CA_ADDR_LIST). The RFC's say
that the network kernel can respond with "destination unreachable" ICMP
error response to a IP packet with a unicast destination address, but not to
a IP packet with a broadcast destination address. Since the ICMP destination
unreachable isn't returned if there is a valid udp receiver for the
destination frame then starting and stopping the other ca server might have
an impact on the occurrence of the message.

I just now added some additional code to the new server so that it will
print a better diagnostic message in the future. The situation is complex
(modal) because the socket function that causes the error is a different one
than the socket function that actually receives it as an error return.

Jeff
______________________________________________________
Jeffrey O. Hill           Email        [email protected]
LANL MS H820              Voice        505 665 1831
Los Alamos NM 87545 USA   FAX          505 665 5107

Message content: TSPA

With sufficient thrust, pigs fly just fine. However, this is
not necessarily a good idea. It is hard to be sure where they
are going to land, and it could be dangerous sitting under them
as they fly overhead. -- RFC 1925

> -----Original Message-----
> From: [email protected] [mailto:[email protected]]
> On Behalf Of J. Lewis Muir
> Sent: Thursday, May 19, 2011 3:28 PM
> To: EPICS Tech Talk
> Subject: CA beacon routing error S_errno_ECONNREFUSED
> 
> I'm getting CA beacon routing error messages on the consoles of
> two IOCs and I'm hoping someone can help me figure out why.
> 
> The two IOCs are ioc1 (IP address: 192.168.0.133) and ioc2 (IP
> address: 192.168.0.134).  They're on the same subnet and connect
> to PVs provided by each other.
> 
> I'm getting messages like the following on ioc1:
> 
> ===
> ../online_notify.c: CA beacon routing (connect to
> "192.168.0.134:5065") error was "S_errno_ECONNREFUSED"
> ===
> 
> And I'm getting similar messages on ioc2:
> 
> ===
> ../online_notify.c: CA beacon routing (connect to
> "192.168.0.133:5065") error was "S_errno_ECONNREFUSED"
> ===
> 
> For a sample 24 hour period, ioc1 had 16 of these messages while
> ioc2 had 2.  Attached are the console logs for this 24 hour
> period for these two IOCs w/ hourly "marker" lines.
> 
> Ioc1 has the following environment configuration:
> 
> ===
> putenv "EPICS_CA_AUTO_ADDR_LIST=NO"
> putenv "EPICS_CA_ADDR_LIST=192.168.0.2 192.168.0.134
> 192.168.0.135"
> ===
> 
> And ioc2 has the following environment configuration:
> 
> ===
> putenv "EPICS_CA_AUTO_ADDR_LIST=NO"
> putenv "EPICS_CA_ADDR_LIST=192.168.0.2 192.168.0.133
> 192.168.0.135 192.168.0.135:55064"
> ===
> 
> I am able to successfully use a calc record on each IOC to
> connect to a PV on the other and read its value.  The IOCs seem
> to be working fine.
> 
> Here's the dbcar output for ioc1:
> 
> ===
> ioc1> dbcar(0,1)
> CA links in all records
> 
> Total 707 CA links; 707 connected, 0 not connected.
>     0 can't read, 10 can't write.  (40 disconnects, 0 writes
> prohibited)
> 
> value = 0 = 0x0
> ===
> 
> And here's the dbcar output for ioc2:
> 
> ===
> ioc2> dbcar(0,1)
> CA links in all records
> 
> Total 1251 CA links; 1251 connected, 0 not connected.
>     0 can't read, 10 can't write.  (47 disconnects, 8 writes
> prohibited)
> 
> value = 0 = 0x0
> ===
> 
> I'm a little surprised to see the number of disconnects for
> each.  However, that might be normal (or not).  Ioc2 has some
> records which connect to soft IOCs which may get started and
> stopped a number of times which might cause the disconnect count
> to increase.  (I'd have to test that to confirm.)  On the other
> hand, ioc1 doesn't connect to other soft IOCs, so why is its
> disconnect count so high?  Both connect to PVs provided by the
> APS PV gateway for my sector.  (When I ran dbcar, both IOCs had
> been up for much longer than the 24 hour time window of the
> attached console logs.)
> 
> Hardware and software for ioc1:
> 
> * VME processor module: Motorola MVME-5100 (vxWorks-ppc604_long)
>   - CPU: MPC 750
>   - Memory: 512 MB
> * OS: VxWorks 5.4.2
> * Kernel: WIND 2.5
> * BSP: mv5100-asd9 (by Andrew Johnson)
> * Build toolchain: Tornado 2.0.2
> * EPICS 3.14.11 (+ Known Problems patches)
> * SynApps 5.5 (+ asyn 4-15, mca 6-12-2, motor 6-5-1, tpmac 3-7)
> 
> Hardware and software for ioc2:
> 
> * VME processor module: Motorola MVME-5100 (vxWorks-ppc604_long)
>   - CPU: MPC 750
>   - Memory: 512 MB
> * OS: VxWorks 5.4.2
> * Kernel: WIND 2.5
> * BSP: mv5100-asd9 (by Andrew Johnson)
> * Build toolchain: Tornado 2.0.2
> * EPICS 3.14.11 (+ Known Problems patches)
> * SynApps 5.5 (+ asyn 4-15, mca 6-12-2, motor 6-5-1, tpmac 3-7)
> 
> Thanks,
> 
> Lewis
> 
> --
> J. Lewis Muir
> Software Engineer
> IMCA-CAT
Replies:
Re: CA beacon routing error S_errno_ECONNREFUSED J. Lewis Muir
References:
CA beacon routing error S_errno_ECONNREFUSED J. Lewis Muir
Navigate by Date:
Prev: trouble with 64-bit Window7 Matt Newville
Next: RE: caGateway crashes / use of *MustSucceed functions Jeff Hill
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  <2011>  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: CA beacon routing error S_errno_ECONNREFUSED J. Lewis Muir
Next: Re: CA beacon routing error S_errno_ECONNREFUSED J. Lewis Muir
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  <2011>  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024