Experimental Physics and
Industrial Control System

Ru Igarashi via Tech-talk <tech-talk at aps.anl.gov> · Sat, 17 Apr 2021 19:01:51 +0000

I built and ran ca-gateway R2.0.6.0 and R2.1.2 on SL/RHEL 7, with
EPICS 7, and it is exhibiting the same problem with asynchronous
caput returning with the old value rather than new.

I am going to determine if this problem only presents with caput
by porting my tests to SNL and also porting to a database-only
implementation.  If those are unaffected, I may end up disregarding
this problem as a fringe use case.

ru
________________________________________
From: Ru Igarashi
Sent: Saturday, April 17, 2021 0:37
To: Hugo Slepicka
Cc: tech-talk at aps.anl.gov
Subject: Re: problems with running EPICS Gateway on virtual machines?

Sorry for taking so long to report this...

I built and ran R2.0.6.0 on Scientific Linux/RHEL 7, EPICS 3.14.12.4,
and it is exhibiting the same problem with asynchronous caput returning
with the old value rather than new.

The configuration that isn't giving me trouble is R2.0.2.1 on SL/RHEL 5,
EPICS 3.14.7 (yes, very old), on a Dell server (i.e. not a VM).

And as reported earlier, time stamps are OK.

ru
________________________________________
From: Hugo Slepicka <hhslepicka at gmail.com>
Sent: Tuesday, April 13, 2021 12:56
To: Ru Igarashi
Cc: tech-talk at aps.anl.gov
Subject: External: Re: problems with running EPICS Gateway on virtual machines?

Hi Ru,

Could you check if doing a caget -a shows the timestamp as undefined?

Another thing is: could you try to downgrade R2.0.6 and see if it works?

Cheers,
Hugo

On Tue, Apr 13, 2021 at 10:44 AM Ru Igarashi via Tech-talk <tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>> wrote:

Hello,

I have at least one problem with running EPICS CA Gateway 2.1.2
on Scientific Linux/Redhat Enterprise Linux 7 with EPICS 3.14.12.4.
For IOC's running EPICS 7 on CentOS 8 and clients on older OS and
EPICS 3.14.12, the readback value for asynchronous 'caput' commands
(caput -c) are often the old value, not new value.  I have an older
gateway running on old hardware, OS and EPICS which also does this
but 100 times LESS frequently for the same type of IOC and client.
The other difference between the two set-ups is that the new
gateway is running on a virtual server (VMWare) and the old gateway
is running on hardware.

Further, when I see these errors, packet captures show the only
differences in behaviour are that:
- on the client side of the gateway, in the bad transaction, the
"update" packet that follows the "write" arrives after the write
is acknowledged whereas normal transactions have the
acknowledgement packet arrive after the "update" packet.
- on the server side of the gateway, in the bad transaction, the
"update" packet carries a data payload of zeros in all bytes,
whereas normal transactions have non-zero data payloads.

It looks as if updates from the IOC are arriving late and
the gateway is returning unusable data.

I haven't ruled out server performance yet, but before I sink
any more time into this, I'd like to ask, has anyone
experienced problems running Gateways on virtual servers?
Or could this be due to running the gateway on newer, higher
performance machines, or due to mismatches in the way channel
access is handled by EPICS 7 versus 3.14.12.4?

ru

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System