Experimental Physics and
Industrial Control System

Michael Davidsaver via Tech-talk <tech-talk at aps.anl.gov> · Wed, 13 Apr 2022 14:19:44 -0700

On 4/13/22 10:50, Jure Varlec via Tech-talk wrote:

Hi Michael,

Binding a socket to a specific interface doesn't always work so well
when that interface disappears and reappears.  You mention "wifi" and
"after a reboot or several", this makes me wonder if your network
interface(s) are being deleted and recreated.

I agree that could be a problem if the IOC would be started during boot.
But it is not. This is my development system and I'm starting the IOC
manually, after all the interfaces are already set up. So I don't think this
is it, or at least, I don't see how it could be.

And you also are also not doing a suspend to RAM or hibernate?
Or otherwise (un)plugging hardware?

I brought out the debugger and dug into the code today. The socket is
bound, so I don't see why a "random" source address would be chosen
by the OS. I went a step further and extended cas_send_dg_msg(): I
replaced sendto() with sendmsg(), adding ancillary message data
via IP_PKTINFO socket option in order to override the source address.
It "works", in the sense that if I use a non-existent address, the sending
fails. However, if I use a valid source address, the IP header will contain
the address of the WiFi interface instead.

wrt. IP_PKTINFO (and sendmsg/recvmsg in general) I'd recommend reading
the "ip" man page very closely.  I've been tripped up before by descriptions
of behavior which I only understood after experiencing it.

https://man7.org/linux/man-pages/man7/ip.7.html

As I read the section on IP_PKTINFO and sendto(), ipi_spec_dst and
ipi_ifindex are hints use in routing table lookup.  If you think
about it from the prospective of permission.  It wouldn't be sensible
to allow a non-privileged user to choose/forge the IP source address.

This is totally bizarre. I'm now convinced that the OS is rewriting the
address after the packet leaves the IOC, but I don't know how or why.
I quote the relevant part of systemd-networkd config below if anyone
has experience with a similar setup. I don't see anything special. Is it
possible that a bridge interface with an IP behaves differently than
a physical interface? AFAIK, assigning an IP address to a bridge is
standard practice and recommended, but I could be mistaken.

As I recall, I've gotten myself into trouble while playing with
bridging and tunneling when both ends of the tunnel, or two taps
on the bridge, were on the same host.

Beyond this, wrt. understanding routing in the Linux IP stack, all I can
do is mention "ip route get <dest>", and assure you that however confused
you are now, it would be worse with IPv6.

some random notes after a frustrating afternoon...
https://github.com/mdavidsaver/pvxs/wiki/IPv6-notes

Best,
Jure

==> 20-machines.netdev <==
[Match]
Host=*

[NetDev]
Kind=bridge
Name=machines

==> 20-machines.network <==
[Match]
Name=machines

[Network]
Address=10.1.1.1/24
DHCPServer=true
IPForward=true
IPMasquerade=true

[DHCPServer]
DNS=10.1.1.1
EmitDNS=true
PoolOffset=128
PoolSize=100

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System