EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Re: caget randomly returns Channel connect timed out
From: Dirk Zimoch via Tech-talk <[email protected]>
To: <[email protected]>
Date: Mon, 16 Jul 2018 09:42:29 +0200
Hi Matt,

Do you have new switches?

We at PSI hat problems with new Cisco switches (I don't know the model) dropping CA directed broadcasts (i.e. broadcasts into another subnet, such as 172.17.2.255 172.17.3.255) and even blocking them for a while when the rate went over a certain limit (e.g. when a huge UI starts up). I think that was supposed to be a counter measure against denial of service attacks. It took over a year of discussion with Cisco to get a fix.

Tracing the IP packets in all networks should show if anything is dropped by the switch.

Dirk


On 13.07.2018 01:24, Matt Rippa wrote:
Thanks for the ideas. It's intriguing this 1 second delay is not long enough. Our change from VxWorks to RTEMS did not change the fundamental application logic. The changes were OSI semantics. Further, the CPU loads are only ~10-20% viewing iocStats. Is that the best assessment of a system being "busy" ? Perhaps iocStats doesn't give enough
resolution.

We're collecting data from new tests at both our sites and should know more
tomorrow. I ran a quick ping test and interestingly the VxWorks IOC shows higher rtt mdev:

RTEMS:
53 packets transmitted, 53 received, 0% packet loss, time 52045ms
rtt min/avg/max/mdev = 1.126/1.153/1.204/0.038 ms

VxWorks:
53 packets transmitted, 53 received, 0% packet loss, time 52059ms
rtt min/avg/max/mdev = 0.951/1.052/1.902/0.184 ms

-Matt



On Thu, Jul 12, 2018 at 10:47 AM Michael Davidsaver <[email protected] <mailto:[email protected]>> wrote:

    On 07/12/2018 01:12 PM, Andrew Johnson wrote:
     > Hi Matt,
     >
     > On 07/12/2018 02:54 PM, Matt Rippa wrote:
     >> We ran a simple caget (once per second) from a linux host to an
    upgraded
     >> IOC. With the new rtems/3.14.12.7 <http://3.14.12.7>
    <http://3.14.12.7> IOC, it often
     >> experiences timeouts as shown below. With our legacy
    vxworks/3.13.9 IOC
     >> there are no timeouts logged. Relevant info is shown below. The
     >> caRepeater is running on the linux host.
     >
     > If your IOC is busy the caget default search timeout of 1 second
    may be
     > too short, have you tried using 'caget -w 5 tcs:ak:astCtx' say?

    @Matt, If this doesn't have an effect I'd suggest taking a packet
    capture
    to see whether these is a search reply coming back.

    As it happens, I had an symptom similar to this while playing around
    with an mvme31000 emulator.  The issue turned out to be dropped RX
    interrupts due to a bug in the emulator.  The effect was due to
    a hard coded 1 second timeout in CA during TCP connect().

    Have you tried the simple test of letting 'ping' run for a while?
    If the reported latencies aren't stable, it may be a sign of a
    lower level issue.

     >> I'm thinking that setting the linux host EPICS_CA_ADDR_LIST=YES may
     >> help. I'm running that test now.
     >
     > I have never seen an EPICS_CA_ADDR_LIST as long as yours, but
    given that
     > you seem to be searching multiple subnets maybe you do need that.

    Agreed.  For this test I'd suggest setting EPICS_CA_ADDR_LIST with only
    the IP of the IOC you're testing.  Along with EPICS_CA_AUTO_ADDR_LIST=NO

     > I don't think setting EPICS_CA_AUTO_ADDR_LIST to YES will make any
     > difference to this behaviour (your epicsPrtEnvParams showed it as YES
     > already, so unless you have explicitly changed it for the client
    you're
     > probably already searching your local subnet as well).

    Also if both IOC and client tools are 3.14.12.x then you can try

    EPICS_CA_AUTO_ADDR_LIST=NO EPICS_CA_ADDR_LIST=
    EPICS_CA_NAME_SERVERS=<IP>

    This will skip the UDP search altogether and do a TCP search.


     > HTH,
     >
     > - Andrew
     >
     >
     >> EPICS_CA_ADDR_LIST=172.17.2.255 172.17.3.255 172.17.102.130
     >> 172.17.105.20 172.16.102.130 aom-vme sbfmcao01.cl.gemini.edu
    <http://sbfmcao01.cl.gemini.edu>
     >> mcao-stealth.cl.gemini.edu <http://mcao-stealth.cl.gemini.edu>
    172.17.106.111 172.17.105.37
     >> 172.17.107.50 172.17.55.101 172.17.101.101 172.17.65.255
     >> 172.17.102.139 172.17.102.138
     >>
     >> linux> caget tcs:ak:astCtx
     >> tcs:ak:astCtx                  0
     >> linux> caget tcs:ak:astCtx
     >> Channel connect timed out: 'tcs:ak:astCtx' not found.
     >> linux> caget tcs:ak:astCtx
     >> Channel connect timed out: 'tcs:ak:astCtx' not found.
     >> linux> caget tcs:ak:astCtx
     >> tcs:ak:astCtx                  0
     >



--


Replies:
Re: caget randomly returns Channel connect timed out Andrew Johnson
References:
caget randomly returns Channel connect timed out Matt Rippa
Re: caget randomly returns Channel connect timed out Andrew Johnson
Re: caget randomly returns Channel connect timed out Michael Davidsaver
Re: caget randomly returns Channel connect timed out Matt Rippa

Navigate by Date:
Prev: GeSys+Cexp on RTEMS 4.10+ Anton Derbenev
Next: RE: Failed caput not notified when using ca_array_put_callback michael.abbott--- via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Re: caget randomly returns Channel connect timed out Matt Rippa
Next: Re: caget randomly returns Channel connect timed out Andrew Johnson
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020  2021  2022  2023  2024 
ANJ, 16 Jul 2018 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·