Experimental Physics and
Industrial Control System

Michael Davidsaver via Tech-talk <tech-talk at aps.anl.gov> · Tue, 23 Nov 2021 07:28:06 -0800

On 11/22/21 2:25 PM, Wilkins, Stuart via Tech-talk wrote:
> Hi!
> 
>  
> 
> I wanted to post here an idea / test implementation that we are working on at NSLS-II. We have been though some work here to restructure our network (including changing all IP addresses, but that is another story!). We now have ~28 beamlines and would like to move towards a more sustainable network structure., moving to where most of our critical services run in the NSLS-II data center, or even outside the NSLS-II complex. We would also like to where possible remove the requirement to “Dual Home” machines and rely on network routing. The latter has been a huge goal for us and we are almost there!

If you are willing to forego the access control functions of a CA gateway,
and can effect the necessary router configuration, then fyi. IP multicast
is an option for CA since Base 3.16.1.

> export EPICS_CA_ADDR_LIST=224.1.2.3
> export EPICS_CA_MCAST_TTL=2
> camonitor pv:name

TTL=2 assumes only one router hop is necessary.

Also, remember that search of TCP is possible, so eg. a pair of gateways
with a limited tunnel (SSH or VPN) can bridge between logically separated
networks.  (cf. $EPICS_CA_NAME_SERVERS)

> From the network perspective, we currently have the Layer 3 boundary at the beamline aggregation switch with each beamline having their own /21 network space. For many of our central services (for example mongo) they run on the routable network ok. However, with the layer 3 boundary being at the beamline we lose the ability to see EPICS broadcasts at the data center without allowing broadcasts outside the local subnet or doing a “Layer 2 drag” and dual homing the common services which we want to eliminate.
> 
>  
> 
> In trying to solve this problem, we have developed a small client / server program called epics-relay which allows us to “transport” the UDP broadcast packets from one subnet to another over UDP unicast. One half is a collector which listens on the broadcast ports for nameserver and beacons. When a nameserver request comes in, the packet is disassembled and the PVs inspected. The PV names are then REGEX matched and if good, the packet is sent via UDP unicast to the emitter.
> 
>  
> 
> The emitter, listens for inputs from the collector and if the packet is valid sends that request or beacon out on the local subnet on the local broadcast address but with a source IP of the original request. This then causes the IOC to respond to the name request back unicast to the host and the virtual circuit is established.
> 
>  
> 
> If this is run in pairs (one emitter / collector on each host), then beacons also make there way through so we gain the ability to see IOC restarts etc.
> 
>  
> 
> The “alpha” development code is here: https://github.com/NSLS-II/epics-relay <https://github.com/NSLS-II/epics-relay> It needs some refactoring, docs etc which is my work this week.
> 
>  
> 
> Let me say, this is NOT supposed to remove the need of gateways, they till serve very important functions. This is solving a different problem.
> 
>  
> 
> What I like about this approach is that it is very simple code, it contains no state and most importantly it does not get in the way of the virtual circuit. That means that it scales much more easily for larger data such as waveforms. If the program restarts the connections are not affected. I think it can be improved with perhaps rate limiting for name requests etc. Any ideas would be very welcome!
> 
>  
> 
> Finally, I don’t claim any of this is original I think a lot of these ideas have been floating around the community. In fact the idea came from one of the UDP solutions listed on the WIKI. I also want to thank the BNL networking team especially Paul Cianciulli for all the work they did at NSLS-II and helping work on this problem.
> 
>  
> 
> In our testing it works very well (so far!) to run the archiver in our data center on VMs on a different subnet to the beamline. With a adhered to naming convention the regular expressions work well to ensure only certain name requests are tunneled.
> 
>  
> 
> I am posting this to see if this is interesting to anyone and also looking to see if anyone wants to contribute. Please contact me if you are interested!
> 
>  
> 
> Thanks!
> 
> S
> 
>  
> 
> ---
> 
>  
> 
> Stuart Wilkins Ph.D. (he, him, his)
> Program Manager
> Data Science and Systems Integration Program
> National Synchrotron Light Source II
> Office: 631.344.2851
> swilkins at bnl.gov <mailto:swilkins at bnl.gov>
> <https://www.bnl.gov/>
> Twitter <https://twitter.com/BrookhavenLab> | Facebook <https://www.facebook.com/brookhavenlab> | Instagram <https://www.instagram.com/brookhavenlab/> | LinkedIn <https://www.linkedin.com/company/brookhavenlab/>
> 
>  
> 

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System