Re: pvlist does not find all servers in network

[Florian Feldbauer via Tech-talk <tech-talk at aps.anl.gov>]

Ok found the issue.

On the MPOD crate controller, nftable is used as firewall, not iptables....

The nftables config was:

>   chain output {
>             type filter hook output priority 0; policy drop;

followed by a list of a few ports/services on UDP and TCP that were allowed.

From the debug output below (e.g. " 2026-01-28T10:33:24.334 Sending 49 bytes 0.0.0.0:38188 -> 192.168.10.1:53857.")
I guess neither the source nor the destination port of the reply are fixed?
Which means, I basically have to change the rules to just accept all outbound UDP traffic (which I do not really see an issue with....it's in a local subnet anyway).

Cheers,
Florian

On 1/28/26 12:03, Florian Feldbauer via Tech-talk wrote:

Hey Andrew and Michael,

to answer the obvious questions:

• Output from `uname -a`:

  root@iseg-iCS2-5230317:~# uname -a
  Linux iseg-iCS2-5230317 4.9.220-2.8.8+g93458a0ecf099 #1 SMP Fri May 16 07:15:45 UTC 2025 armv7l GNU/Linux

• Broadcast address is 192.168.10.255 on all devices in this network.
• On the MPOD is only one IOC running.
• On the MPOD EPICS base 7.0.4.1 is running,
• On my PC base 7.0.9 is installed.

In both cases the corresponding tag was recursively checked out from the git repository and build with the all submodules.

I know about `netstat -p` und normally use `-tulpn`, but:

root@iseg-iCS2-5230317:~# netstat -tulpn
netstat: invalid option -- 'p'
BusyBox v1.24.1 (2025-05-16 07:17:53 UTC) multi-call binary.

Usage: netstat [-ral] [-tuwx] [-en]

Using $EPICS_PVA_DEBUG=3 was a great tip! I thought receiving the UDP broadcast would be the issue but obviously sending a reply is:

2026-01-28T10:33:23.690 UDP Client Rx (37) 192.168.10.255:5076 <- 192.168.10.1:53857
2026-01-28T10:33:23.690 Message [0xUDP Server Rx (37) 192.168.10.255:5076 <- 192.168.10.1:538573, v0x2] received from 192.168.10.1:53857 on 192.168.10.255:5076 : 
Message [0x3, v0x2] received from 192.168.10.1:53857 on Search
192.168.10.255:5076 : Search request0x00 
0x0000000000  0000000081000000 000000000 81000000 00000000 000 000000000000  .... .... 00  ..... ......
0x10 00. .00ffff 00000000... 61d20000 . 00  ...      .. .... .... a..... .

0x10 0000ffff 00000000 61d20000 00        .... .... a... .
2026-01-28T10:33:23.754 Sending 49 bytes 0.0.0.0:38188 -> 192.168.10.1:53857.
2026-01-28T10:33:23.754 Socket sendto to 192.168.10.1:53857 error: Operation not permitted.
2026-01-28T10:33:23.956 UDP Client Rx (37) 192.168.10.255:5076 <- 192.168.10.1:53857
2026-01-28T10:33:23.956 Message [0xUDP Server Rx (37) 192.168.10.255:5076 <- 192.168.10.1:53857
Message [0x3, v0x2] received from 192.168.10.1:53857 on 3, v0x2] received from 192.168.10.1:53857 on 192.168.10.255:5076 : Search request
0x00 000000192.168.10.255:5076 : 00 81000000Search
0x00 00000000  00000000 000000810000  .... ...0000 . .00000... ...00.
00 0x10 0000ff0000ff 0000000000  ...0. ..00 .. ...61d2. 0000.. 00        ...... .... a... .

0x10 0000ffff 00000000 61d20000 00        .... .... a... .
2026-01-28T10:33:24.024 Sending 49 bytes 0.0.0.0:38188 -> 192.168.10.1:53857.
2026-01-28T10:33:24.024 Socket sendto to 192.168.10.1:53857 error: Operation not permitted.
2026-01-28T10:33:24.212 UDP Client Rx (37) 192.168.10.255:5076 <- 192.168.10.1:53857
2026-01-28T10:33:24.212 Message [0xUDP Server Rx (37) 192.168.10.255:5076 <- 192.168.10.1:53857
Message [0x3, v0x3, v0x2] received from 192.168.10.1:53857 on 192.168.10.255:5076 : Search request2] received from 192.168.10.1:53857 on 
0x192.168.10.255:5076 : 00 00000000 810000Search
00 000x00 000000000000000 81000000  000000000  ....00000000 000000 ...00  ... .... ....
0x10 0000ffff.. . 0000..0000 61d20000 . ....0 .00        .....
0x10 0000ff0.. .... a... .
ff 00000000 61d20000 00        .... .... a... .
2026-01-28T10:33:24.334 Sending 49 bytes 0.0.0.0:38188 -> 192.168.10.1:53857.
2026-01-28T10:33:24.334 Socket sendto to 192.168.10.1:53857 error: Operation not permitted.

I don't know why there is an operation not permitted error.
The firewall should be open:

root@iseg-iCS2-5230317:~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination        

Cheers,
Florian

On 1/28/26 02:40, Michael Davidsaver wrote:

Hello Florian,

First, the usual questions.  Which versions of epics-base and associated modules are involved?  Including pvDataCPP, pvAccessCPP, and pva2pva if built as separate modules.  Do you set any of the EPICS_PVA* environment variables?

On this MPOD crate.  Is there only the one IOC process?  fyi. you can add '-p' to 'netstat' and it will tell you which PID each socket is associated with.

As Andrew mentioned.  Setting $EPICS_PVA_DEBUG=3 will turn on a fire hose for logging.

From the output provided, PVXS is not present.  You might also try "pvxlist" and/or "softIocPVX" to see if the behavior differs.

On 1/27/26 4:02 AM, Florian Feldbauer via Tech-talk wrote:

Hey all,

I have a strange problem, that I haven't encountered before.

In our lab we have a small test setup for our pixel sensors.
As HV/LV power supply for these sensors we use a Wiener MPOD crate with an iseg Crate Controller.
The crate controller is running Linux (The Ångström Distribution v2017.12) and an EPICS IOC with PVA.

Now, when I run `pvlist` from another PC I see a list of all the IOCs running in this network except the one on the iseg Crate Controller.
When I explicitly run `pvlist iseg-iCS2-5230317` I get a list of all PVs from that IOC, so PVA communication works in general, but for some reason this IOC is not reacting on broadcasts.

I checked with netstat and for me this looks correct:

root@iseg-iCS2-5230317:~# netstat -tuln | grep 5076
udp        0      0 224.0.0.128:5076        0.0.0.0:*
udp        0      0 192.168.10.255:5076     0.0.0.0:*
udp        0      0 192.168.10.22:5076      0.0.0.0:*
udp        0      0 224.0.0.128:5076        0.0.0.0:*
udp        0      0 192.168.10.255:5076     0.0.0.0:*
udp        0      0 192.168.10.22:5076      0.0.0.0:*
udp        0      0 :::50767                :::*

There is no firewall running on the crate controller.

Any idea what could cause this issue?

Cheers,
Florian

-- 
Ruhr-Universität Bochum
AG der Experimentalphysik I
Dr. Florian Feldbauer
NB 2/131 / Fach 125
Universitätsstr. 150
D-44801 Bochum

Office: NB 2/134
Phone:  (+49)234 / 32-23563
Fax:    (+49)234 / 32-14170
https://paluma.ruhr-uni-bochum.de

-- 
Ruhr-Universität Bochum
AG der Experimentalphysik I
Dr. Florian Feldbauer
NB 2/131 / Fach 125
Universitätsstr. 150
D-44801 Bochum

Office: NB 2/134
Phone:  (+49)234 / 32-23563
Fax:    (+49)234 / 32-14170
https://paluma.ruhr-uni-bochum.de