Re: External: Re: problems with running EPICS Gateway on virtual machines?

[Ru Igarashi via Tech-talk <tech-talk at aps.anl.gov>]

Hello Hugo,

Neither camonitor nor caget shows bad timestamps.  Also, I forgot
to mention that, when I do a caget right after a caput -c it returns
with the correct value, and running a parallel camonitor exacerbates
the problem.  I plan on running more detailed tests with caget and
camonitor, and with debugging turned on.  But I would like to know
if this is a familiar problem or that running CA Gateways on VMWare
is a bad idea to begin with (I had big latency problems many years
ago, so I went to a hardware solution at that time).

ru
________________________________________
From: Hugo Slepicka <hhslepicka at gmail.com>
Sent: Tuesday, April 13, 2021 12:56
To: Ru Igarashi
Cc: tech-talk at aps.anl.gov
Subject: External: Re: problems with running EPICS Gateway on virtual machines?

Hi Ru,

Could you check if doing a caget -a shows the timestamp as undefined?

Another thing is: could you try to downgrade R2.0.6 and see if it works?

Cheers,
Hugo

On Tue, Apr 13, 2021 at 10:44 AM Ru Igarashi via Tech-talk <tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>> wrote:

Hello,
I have at least one problem with running EPICS CA Gateway 2.1.2
on Scientific Linux/Redhat Enterprise Linux 7 with EPICS 3.14.12.4.
For IOC's running EPICS 7 on CentOS 8 and clients on older OS and
EPICS 3.14.12, the readback value for asynchronous 'caput' commands
(caput -c) are often the old value, not new value.  I have an older
gateway running on old hardware, OS and EPICS which also does this
but 100 times LESS frequently for the same type of IOC and client.
The other difference between the two set-ups is that the new
gateway is running on a virtual server (VMWare) and the old gateway
is running on hardware.


Further, when I see these errors, packet captures show the only
differences in behaviour are that:
- on the client side of the gateway, in the bad transaction, the
"update" packet that follows the "write" arrives after the write
is acknowledged whereas normal transactions have the
acknowledgement packet arrive after the "update" packet.
- on the server side of the gateway, in the bad transaction, the
"update" packet carries a data payload of zeros in all bytes,
whereas normal transactions have non-zero data payloads.

It looks as if updates from the IOC are arriving late and
the gateway is returning unusable data.

I haven't ruled out server performance yet, but before I sink
any more time into this, I'd like to ask, has anyone
experienced problems running Gateways on virtual servers?
Or could this be due to running the gateway on newer, higher
performance machines, or due to mismatches in the way channel
access is handled by EPICS 7 versus 3.14.12.4?

ru