Re: pvxput denied via EPICS_PVA_NAME_SERVERS but works via EPICS_PVA_ADDR_LIST --- (pvxs 1.5.1 / p4p 4.1.7)

["Williams Jr., Ernest L. via Tech-talk" <tech-talk at aps.anl.gov>]

Hi everyone,

I wanted to share a quick follow-up and a short debugging story related to the pvxput permission denied issue we were seeing with the PVA gateway.

The original problem we were investigating was that pvxput operations were returning “permission denied” when going through the gateway running p4p 4.1.7. Since there have been improvements in newer versions, I upgraded the gateway to p4p 4.2.2 and expected the issue to be resolved. However, after deploying the new gateway I was still seeing inconsistent behavior depending on how the client discovered the PV, which initially made it look like the upgrade had not fixed the issue.

To investigate further I compared the two PVA discovery modes. In UDP search mode the client was configured with EPICS_PVA_ADDR_LIST=<gateway-host-IP>:5167, and in TCP name-server mode it was configured with EPICS_PVA_NAME_SERVERS=<gateway-host-IP>:5169. In my local laptop test environment, where only a single gateway instance was running (p4p 4.2.2), both discovery modes behaved identically and resolved to the same gateway endpoint, which is the expected behavior.

On the production gateway host, however, I noticed that the endpoint returned by pvxinfo depended on the discovery mode. UDP search mode returned <gateway-host-IP>:40433, while TCP name-server mode returned <gateway-host-IP>:5169. At first this was confusing because the gateway should present a consistent front-door endpoint.

I ran a packet capture on the gateway host to understand what was happening. The UDP search request from the client was sent to port 5167, the gateway replied to the client, and the reply instructed the client to connect to TCP port 40433. That suggested the responder on UDP 5167 was advertising 40433 as the data endpoint.

Looking at the listening sockets on the gateway host revealed the real cause: two different gateway processes were running simultaneously. One gateway instance (running p4p 4.2.2 from my newer test environment) was listening on UDP 5167 and TCP 40433, while another older gateway instance (p4p 4.1.7 from an existing installation) was listening on TCP 5169. As a result, UDP search requests were being handled by the new gateway instance, while TCP name-server requests were going to the older gateway instance. In other words, my comparison was unintentionally testing two different gateway processes rather than two discovery modes of the same gateway.

After stopping the older gateway instance so that only the new 4.2.2 gateway was running, both discovery modes behaved exactly the same and resolved to the same endpoint, matching the behavior I observed in my laptop tests.

So in the end the new gateway version does appear to work correctly. The confusing behavior I observed after the upgrade was simply due to a deployment mistake on my part — the older gateway was still running in parallel, which meant UDP discovery and TCP name-server discovery were reaching different processes.

The main takeaway from this little adventure is that when debugging PVA discovery issues it’s really helpful to double-check which processes are actually listening on the discovery and server ports. A quick look at the socket table or a small tcpdump capture can immediately reveal if multiple responders are present.

Thanks again to Michael Davidsaver and everyone working on the gateway — it was a fun little exercise following the discovery traffic and figuring out what was really happening.

Cheers,
Ernest

---

From: Williams Jr., Ernest L. <ernesto at slac.stanford.edu>
Sent: Friday, March 6, 2026 8:09 PM
To: Michael Davidsaver <mdavidsaver at gmail.com>
Cc: Tech Talk EPICS <tech-talk at aps.anl.gov>; Williams Jr., Ernest L. <ernesto at slac.stanford.edu>
Subject: Re: pvxput denied via EPICS_PVA_NAME_SERVERS but works via EPICS_PVA_ADDR_LIST --- (pvxs 1.5.1 / p4p 4.1.7)

 

Hi Michael,

Responses below

---

From: Williams Jr., Ernest L. <ernesto at slac.stanford.edu>
Sent: Friday, March 6, 2026 1:58 PM
To: Michael Davidsaver <mdavidsaver at gmail.com>
Cc: Tech Talk EPICS <tech-talk at aps.anl.gov>; Williams Jr., Ernest L. <ernesto at slac.stanford.edu>
Subject: Re: pvxput denied via EPICS_PVA_NAME_SERVERS but works via EPICS_PVA_ADDR_LIST --- (pvxs 1.5.1 / p4p 4.1.7)

 

Sent from my iPhone

On Mar 6, 2026, at 13:40, Michael Davidsaver <mdavidsaver at gmail.com> wrote:



BEWARE: This email originated outside of our organization. DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe.

Hi Ernest,

On 3/6/26 11:41 AM, Williams Jr., Ernest L. via Tech-talk wrote:

...Client Environment

Client tool: pvxs 1.5.1
Gateway implementation: p4p 4.1.7

Is this "stock" PVXS, or the SPVA development branch?

This is stock pvxs

Also, the latest P4P is 4.2.2.  Could you test with this version?

I will redo my tests with the latest version

===============================================================================

ELW:  I am now using 4.2.2 and receive the error still

/sdf/group/cds/sw/epics/users/ernesto/sandbox/modules/pvxs/1.5.1/bin/rhel9-x86_64/pvxput SIOC:SYS0:ML06:AO007 12.3
ERROR N4pvxs6client11RemoteErrorE : Put permission denied by gateway

=======================================================================================================

wrt. gateway testing.  What does the "<statusprefix>asTest" RPC call show?  This call should return the identity which p4p.gw would use for ACF decisions.

https ://epics-base.github.io/p4p/gw.html#status-pvs

================================================================================================================

ELW:  When using EPICS_PVA_NAME_SERVERS it just timeouts.

[ernesto@dev-epicsgw config]$ /sdf/group/cds/sw/epics/users/ernesto/sandbox/modules/pvxs/1.5.1/bin/rhel9-x86_64/pvxcall PGWLCLSS3DF:asTest pv=SIOC:SYS0:ML06:AO007
Timeout

[ernesto@dev-epicsgw config]$ /sdf/group/cds/sw/epics/users/ernesto/sandbox/modules/pvxs/1.5.1/bin/rhel9-x86_64/pvxcall -w 10 PGWLCLSS3DF:asTest pv=SIOC:SYS0:ML06:AO007
Timeout

However, when using 

EPICS_PVA_ADDR_LIST=<gateway-ip>:5167
EPICS_PVA_AUTO_ADDR_LIST=NO

pvxcall return with all the goods.

==================================================================================================================================================

...

Question

Is it expected that the p4p gateway behaves differently for put authorization depending on whether the PV was resolved through:

• UDP search (EPICS_PVA_ADDR_LIST)
• TCP name-server mode (EPICS_PVA_NAME_SERVERS) ?

No.

In particular, should both mechanisms ultimately route the client through the same gateway path and authorization logic, or is there a known difference in how these endpoints are handled?

This is not expected...  Unless SPVA comes into the picture, and there are legitimately two different TCP ports.  One for plain PVA, and a second for TLS+PVA.

Our longer-term goal is to standardize on EPICS_PVA_NAME_SERVERS, since it works much better for containerized environments and controlled network paths. Before moving forward with that approach, I wanted to confirm whether the behavior I am seeing is expected or if I might be missing something in the gateway configuration.

Maybe relevant, with SPVA development branch, PVXS will understand an extended syntax to specify protocol as well as port.

export EPICS_PVA_NAME_SERVERS="pva://1.2.3.4 pvas://1.2.3.4"

(Yes, I know it the prefix should now be "spva://", but this schema pre-dates that name change)

If helpful, I can also provide packet traces or additional logs from the gateway side.

Thanks very much for any guidance.

Cheers,

Ernest Williams
Dept. Head, Controls Software Engineering
Instrumentation Division, Technology Innovation Directorate
SLAC National Laboratory, Stanford University