Re: problems with running EPICS Gateway on virtual machines?

[Ru Igarashi via Tech-talk <tech-talk at aps.anl.gov>]

[normally, I'd inline my response but I'm stuck with Web Outlook.  My apologies.]

Hello Andrew,

re: camonitor: the errant sequence is the same with and without camonitor
running in parallel.  My thinking was that the reliance on the Channel
Access monitors meant that update occasionally necessarily would
eventually be out of sync.  But I couldn't reconcile that with the low incident
rate on the old gateway.  It didn't occur to me that there would be a setting
to not cache.

Your suggestion of the -no_cache option verified that theory: zero incidents.
But I still can't reconcile the low incident rate on the old gateway running on
old hardware.  I have to wonder if the different server performance specs
are impacting this.  Does the -no_cache result point to a possible solution?

In the long run, the -no_cache option is not acceptable.  I am using the
gateway as a CA proxy or concentrator, thus I require caching.  

Looking forward, the way I see things unfolding are either:
A) find a different server that doesn't have this problem (or at least as bad)
B) determine that this is not impacting anything I do and just live with it
C) determine that the additional volume of traffic is not that bad after all
and use -no_cache.

ru
________________________________________
From: Johnson, Andrew N. <anj at anl.gov>
Sent: Tuesday, April 13, 2021 15:52
To: Ru Igarashi
Cc: Hugo Slepicka; EPICS tech-talk
Subject: Re: External: Re: problems with running EPICS Gateway on virtual machines?

What happens when you run a camonitor on the PV(s) at the same time as you do the put, does it show the full sequence that you're expecting?

ISTR there is a configuration setting for the CA gateway which can turn off cacheing, which may be what you want for one that you’re going caputs through; maybe someone else with more knowledge can comment on that.

- Andrew


> On Apr 13, 2021, at 4:33 PM, Ru Igarashi via Tech-talk <tech-talk at aps.anl.gov> wrote:
>
> Hello Hugo,
>
> Neither camonitor nor caget shows bad timestamps.  Also, I forgot
> to mention that, when I do a caget right after a caput -c it returns
> with the correct value, and running a parallel camonitor exacerbates
> the problem.  I plan on running more detailed tests with caget and
> camonitor, and with debugging turned on.  But I would like to know
> if this is a familiar problem or that running CA Gateways on VMWare
> is a bad idea to begin with (I had big latency problems many years
> ago, so I went to a hardware solution at that time).
>
> ru
> ________________________________________
> From: Hugo Slepicka <hhslepicka at gmail.com>
> Sent: Tuesday, April 13, 2021 12:56
> To: Ru Igarashi
> Cc: tech-talk at aps.anl.gov
> Subject: External: Re: problems with running EPICS Gateway on virtual machines?
>
> Hi Ru,
>
> Could you check if doing a caget -a shows the timestamp as undefined?
>
> Another thing is: could you try to downgrade R2.0.6 and see if it works?
>
> Cheers,
> Hugo
>
> On Tue, Apr 13, 2021 at 10:44 AM Ru Igarashi via Tech-talk <tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>> wrote:
>
> Hello,
> I have at least one problem with running EPICS CA Gateway 2.1.2
> on Scientific Linux/Redhat Enterprise Linux 7 with EPICS 3.14.12.4.
> For IOC's running EPICS 7 on CentOS 8 and clients on older OS and
> EPICS 3.14.12, the readback value for asynchronous 'caput' commands
> (caput -c) are often the old value, not new value.  I have an older
> gateway running on old hardware, OS and EPICS which also does this
> but 100 times LESS frequently for the same type of IOC and client.
> The other difference between the two set-ups is that the new
> gateway is running on a virtual server (VMWare) and the old gateway
> is running on hardware.
>
>
> Further, when I see these errors, packet captures show the only
> differences in behaviour are that:
> - on the client side of the gateway, in the bad transaction, the
> "update" packet that follows the "write" arrives after the write
> is acknowledged whereas normal transactions have the
> acknowledgement packet arrive after the "update" packet.
> - on the server side of the gateway, in the bad transaction, the
> "update" packet carries a data payload of zeros in all bytes,
> whereas normal transactions have non-zero data payloads.
>
> It looks as if updates from the IOC are arriving late and
> the gateway is returning unusable data.
>
> I haven't ruled out server performance yet, but before I sink
> any more time into this, I'd like to ask, has anyone
> experienced problems running Gateways on virtual servers?
> Or could this be due to running the gateway on newer, higher
> performance machines, or due to mismatches in the way channel
> access is handled by EPICS 7 versus 3.14.12.4?
>
> ru
>

--
Complexity comes for free, simplicity you have to work for.