Thanks Andrew and Mark for the suggestions. The problem has been resolved.

 

The culprits were several network cameras (FLIR Blackfly) that had lost configuration and booted with  local IP addresses. IT was able to locate the network jack which lead us to the camera.

 

We did not have time to look at the specific traffic from the camera that caused the all the VxWorks  (Version 6.9) IOC’s on the subnet to crash. It should be noted that some Linux IOC’s and MEDM clients also issued the same warning messages (but did not crash).

 

Thanks again for the help.

 

Joe

 

Hi Joe,

The following warning messages are appearing periodically (~2hrs) on all the EPICs IOC's (VxWorks and Linux - base-3.14.12.3) on a beamline here at the APS - (base-3.14.12.3).

 

../udpiiu.cpp: Undecipherable (payload too small) UDP msg from 169.254.0.2:3956 ignored

../udpiiu.cpp: Undecipherable (payload too small) UDP msg from 169.254.132.146:3956 ignored

../udpiiu.cpp: Undecipherable (payload too small) UDP msg from 169.254.0.2:3956 ignored

../udpiiu.cpp: Undecipherable (payload too small) UDP msg from 169.254.0.2:3956 ignored

../udpiiu.cpp: Undecipherable (payload too small) UDP msg from 169.254.0.2:3956 ignored

../udpiiu.cpp: Undecipherable (payload too small) UDP msg from 169.254.0.2:3956 ignored

../udpiiu.cpp: Undecipherable (payload too small) UDP msg from 169.254.0.2:3956 ignored

../udpiiu.cpp: Undecipherable (payload too small) UDP msg from 169.254.132.146:3956 ignored

../udpiiu.cpp: Undecipherable (payload too small) UDP msg from 169.254.132.146:3956 ignored

IP addresses in the 169.254 range are self-assigned by a computer that cannot get an address from a DHCP server. You should have IT track down which network port(s) are using these addresses and get those machines configured properly (or have them disconnected).

Eventually the CAS-Event processes fail (after a day or so). A reboot brings everything back and the sequence starts over.

That is unfortunate, we would like IOCs to be immune to bad packets. If you can get a packet capture of the broadcast packets going to port 5064 on this network (before the rogue client gets killed) it would let us try to prevent this particular payload from crashing the IOCs.

This happened after the IOCs where brought up after a shutdown period.  

 

Is this a network issue (someone on the CA ports that should not be) or should I look at the CA Clients (medm)?

I don't think these messages would be caused by a real CA client, even one that doesn't have a proper IP address. You need to get the network police down to that beamline to find the rogue machine(s). It's a pity it's now after 5pm on a holiday weekend; Mary is still here though.

- Andrew

-- 

Complexity comes for free, Simplicity you have to work for.