Experimental Physics and
Industrial Control System

Ru Igarashi via Tech-talk <tech-talk at aps.anl.gov> · Wed, 14 Apr 2021 07:04:26 +0000

Yong,

Thanks for the -no_cache suggestion.  It did stop the problem but is not
a long term solution because I'm relying on caching so that the
gateway can act as a proxy or channel access concentrator (to
reduce traffic).

The -w option doesn't help.  That's very likely because the response
updates for errant transactions are coming in well under 1 s, i.e.
right after the record finishes processing from the caput.  Recall
that I reported a caget right after the caput has the correct value.
That's happening well under 1 second after the caput.  I think
this is some sort of borderline timing issue.

I have to try Hugo's suggestion of running v2.0.6.

ru
________________________________________
From: Hu, Yong <yhu at bnl.gov>
Sent: Tuesday, April 13, 2021 20:09
To: Johnson, Andrew N.; Ru Igarashi
Cc: EPICS tech-talk
Subject: Re: External: Re: problems with running EPICS Gateway on virtual machines?

Hi Ru,

To turn off caching, add the option "-no_cache" in your gateway command.

Since you use "caput -c pvName", I guess "pvName" seems to be an asynchronous record. You can try this to find out how much time the command "caput -c" takes: time(caput -c pvName).

Another try: increase to the default timeout (1 second) to a bigger number (10 seconds), i.e. caput -c -w 10 pvName

HTH,
Yong

On 4/13/21, 5:52 PM, "Tech-talk on behalf of Johnson, Andrew N. via Tech-talk" <tech-talk-bounces at aps.anl.gov on behalf of tech-talk at aps.anl.gov> wrote:

    What happens when you run a camonitor on the PV(s) at the same time as you do the put, does it show the full sequence that you're expecting?

    ISTR there is a configuration setting for the CA gateway which can turn off cacheing, which may be what you want for one that you’re going caputs through; maybe someone else with more knowledge can comment on that.

    - Andrew

    > On Apr 13, 2021, at 4:33 PM, Ru Igarashi via Tech-talk <tech-talk at aps.anl.gov> wrote:
    >
    > Hello Hugo,
    >
    > Neither camonitor nor caget shows bad timestamps.  Also, I forgot
    > to mention that, when I do a caget right after a caput -c it returns
    > with the correct value, and running a parallel camonitor exacerbates
    > the problem.  I plan on running more detailed tests with caget and
    > camonitor, and with debugging turned on.  But I would like to know
    > if this is a familiar problem or that running CA Gateways on VMWare
    > is a bad idea to begin with (I had big latency problems many years
    > ago, so I went to a hardware solution at that time).
    >
    > ru
    > ________________________________________
    > From: Hugo Slepicka <hhslepicka at gmail.com>
    > Sent: Tuesday, April 13, 2021 12:56
    > To: Ru Igarashi
    > Cc: tech-talk at aps.anl.gov
    > Subject: External: Re: problems with running EPICS Gateway on virtual machines?
    >
    > Hi Ru,
    >
    > Could you check if doing a caget -a shows the timestamp as undefined?
    >
    > Another thing is: could you try to downgrade R2.0.6 and see if it works?
    >
    > Cheers,
    > Hugo
    >
    > On Tue, Apr 13, 2021 at 10:44 AM Ru Igarashi via Tech-talk <tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>> wrote:
    >
    > Hello,
    > I have at least one problem with running EPICS CA Gateway 2.1.2
    > on Scientific Linux/Redhat Enterprise Linux 7 with EPICS 3.14.12.4.
    > For IOC's running EPICS 7 on CentOS 8 and clients on older OS and
    > EPICS 3.14.12, the readback value for asynchronous 'caput' commands
    > (caput -c) are often the old value, not new value.  I have an older
    > gateway running on old hardware, OS and EPICS which also does this
    > but 100 times LESS frequently for the same type of IOC and client.
    > The other difference between the two set-ups is that the new
    > gateway is running on a virtual server (VMWare) and the old gateway
    > is running on hardware.
    >
    >
    > Further, when I see these errors, packet captures show the only
    > differences in behaviour are that:
    > - on the client side of the gateway, in the bad transaction, the
    > "update" packet that follows the "write" arrives after the write
    > is acknowledged whereas normal transactions have the
    > acknowledgement packet arrive after the "update" packet.
    > - on the server side of the gateway, in the bad transaction, the
    > "update" packet carries a data payload of zeros in all bytes,
    > whereas normal transactions have non-zero data payloads.
    >
    > It looks as if updates from the IOC are arriving late and
    > the gateway is returning unusable data.
    >
    > I haven't ruled out server performance yet, but before I sink
    > any more time into this, I'd like to ask, has anyone
    > experienced problems running Gateways on virtual servers?
    > Or could this be due to running the gateway on newer, higher
    > performance machines, or due to mismatches in the way channel
    > access is handled by EPICS 7 versus 3.14.12.4?
    >
    > ru
    >

    --
    Complexity comes for free, simplicity you have to work for.

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System