Experimental Physics and
Industrial Control System

윤상원 <[email protected]> · Fri, 7 Sep 2018 11:22:30 +0000

Hello Ralph,

Sorry for the late reply as I didn't check the tech-talk digest for several days.

Thanks a lot again for your clear and detailed explanation with guidance.

Since my last e-mail mentioning a performance issue on VM, we continued monitoring with some tries.
Through further monitoring, we could confirmed that the message had been also occurred on our physical machine when we changed network routing configuration on the machine on which a CA gateway was running.
Actually the network configuration we changed was same one which we were using on gateway VMs for several years.
Without the network configuration, the message has been disappeared for normal IOCs on gateway VM and we could service without any significant issue so far.

Consequently in our case the reason of the message was our network configuration not performance on VM.
Sorry for providing wrong information.

After this year campaign, I'll try to upgrade EPICS base as per your guidance.

Thanks again.

Best regards,
Sangwon

----------------------------------------------------------------------

Message: 1
Date: Wed, 29 Aug 2018 09:12:08 +0200
From: Ralph Lange <[email protected]>
To: EPICS Tech Talk <[email protected]>
Subject: Re: Virtual circuit unresponsive on CA Gateway
Message-ID:
	<CAGJXJ2G85WxW6KSP1Pg_nPMn44jzhJrjj-EZ+1KsKD9KA4dSwA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi Sangwon,

As you say, possible limitations in network traffic processing might be something that affects Gateway operation on VMs. But I would expect this only to happen if a Gateway is very busy and/or forwards large amounts of data, e.g. images.

Channel Access' detection of unresponsive IOC connections is based on two
mechanisms:
The beacons (by default 5065/UDP broadcasts) sent by the IOC (by default every 15 seconds) must reach the Gateway, which requires the caRepeater program to be running on the Gateway host before the Gateway starts.
If the Gateway does not receive a beacon within twice the configured beacon period, it uses the TCP connection to send a Channel Access alive message to the IOC. If the IOC does not return that alive message within 5 seconds, the Gateway declares the connection unresponsive.

Check if the IOC beacons are reaching the Gateway host. If that Gateway is not in the IOC's local network, the IOC needs to be configured to send beacons to the Gateway.
Also check if the VM's system clock is updating smoothly. I have seen VMs with jumping system clocks, which may affect the described detection mechanism.

Generally, I would suggest using a more recent version of base for the Gateway, e.g. 3.15.5. There is no problem running the Gateway on a different version than IOCs, and there are issues in Channel Access that have been fixed since 3.14.12.4 (December 2013). If you want a simple setup, create statically linked Gateway and caRepeater binaries. Those will not need an installation of EPICS Base to run.

Good luck!
~Ralph

On Wed, 29 Aug 2018 at 08:35, ??? <[email protected]> wrote:

> Hello Yong,
>
>
>
> Thanks a lot for your reply with the information.
>
> We are using the 3.14.12.4 for the CA gateways.
>
>
>
> For the information, we were running virtualized CA gateways and 
> yesterday we have built an additional CA gateway on another physical 
> machine without virtualization for comparison of virtualized and physical CA gateways.
>
> As the result, we observed that the message is often being occurred on 
> a virtualized CA gateway but no such message on the physical machine 
> based CA gateway.
>
> In our case for the moment it seems the message is related with 
> performance of system such as network traffic processing.
>
>
>
> Thanks again.
>
>
>
> Regards,
>
> Sangwon
>
>
>
> *From:* Hu, Yong <[email protected]>
> *Sent:* Wednesday, August 29, 2018 12:29 AM
> *To:* ??? <[email protected]>; [email protected]
> *Subject:* Re: Virtual circuit unresponsive on CA Gateway
>
>
>
> Hi Sangwon,
>
>
>
> Take a look at this: https://bugs.launchpad.net/epics-base/+bug/541238
>
> At the end, Andrew said ?Probably fixed in later versions of Base ?.?,
> ?R3.14.10 released?.
>
> Which version of Base is your CA gateway built against?
>
>
>
> Best,
>
> Yong
>
>
>
> *From: *<[email protected]> on behalf of ??? 
> <[email protected]
> >
> *Date: *Tuesday, August 28, 2018 at 3:29 AM
> *To: *"[email protected]" <[email protected]>
> *Subject: *Virtual circuit unresponsive on CA Gateway
>
>
>
> Hello,
>
>
>
> We are running several CA gateways for providing access to PVs for 
> clients including OPIs.
>
> I observed the warning message ?Virtual circuit unresponsive? on log 
> files of the CA gateways.
>
> For instance,
>
> Aug 28 15:29:19 Warning: Virtual circuit unresponsive 
> 172.17.101.21:5064
>
> Aug 28 15:53:40 Warning: Virtual circuit unresponsive 
> 172.17.101.80:5064
>
> ?
>
>
>
> In addition, when the message is logged on a log file, connections to 
> PVs of the IOC server related with the message (ex. 172.17.101.21) are 
> disconnected.
>
> At the time, connection to the same PVs which directly connects to the 
> IOC server instead of CA gateway is still continued and client 
> (camonitor) can receive changes from the IOC without disturbance.
>
>
>
> For the information, we are running two versions of CA gateways, 
> v2.0.4.0 and v2.1.0.
>
>
>
> Could anyone please share any guidance or comments in the case?
>
> Thanks in advance for your guidance or any comments.
>
>
>
> Regards,
>
> Sangwon

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System