Experimental Physics and Industrial Control System
|
Recently we had an incident where many of our devices using modbus were knocked offline, or experience intermittent connectivity. This occurred across all subnets, and seemed to affect devices that used modbus to communicate. Below are more details. I am
wondering if anyone has seen something like this before? Automation Direct says there are some firmware updates we are behind on. I suspect Beckhoff will say the same. I will be recording network traffic on one of these devices to try and catch some malformed
packet, or some other thing.
Thanks for your consideration.
-Alex
-
First noticeable symptom was a Beckhoff BK9000 modbus tcp coupler lost communication with EPICS (our interface is based on modbus)
-
A vacuum system (using automation direct koyo plc) showed purple on all permissive indicators. These are the Cn bits in the PLC and when I checked the IOC, the Cn asyn port was throwing errors, specifically modbus exception 6, which means that the PLC was busy
with something else.
-
Another hutch also reported that their PLC was having issues communicating with EPICS. Controls still worked, but the readbacks would go purple everytime a valve was actuated. This means that EPICS was probably encountering timeout, or PLC busy responses.
-
In both cases, the PLC appeared to continue functioning. Communication with EPICS was the only interruption.
-
Checking all Beckhoff couplers (BK9000) in both hutch subnets showed they were able to be pinged, and other application layer services were functioning (eg ADS for the beckhoff devices, ks4000 software could still connect over ethernet).
-
New Beckhoff PLCs were not affected by this issue.
-
Telnet to port 502 was functional for all these devices
-
Modpoll was able to communicate with the PLCs
-
EPICS would have had library loading issues if something in the module changed or was missing
-
The issues appeared without a restart of the IOCs suggesting that it was external to EPICS
-
The issue occurred across multiple subnets
-
ECOM (the interface for the Automation Direct PLCs to be programmed remotely) is currently non-functioning for PLCs that haven't been power-cycled. A hutch PLC was cycled and the ECOM interface recovered.
-
ECOM for the AD PLCs shares the NIC with modbus communication, still don't know what port it uses, 16#7070.
-
Tested robustness of the AD PLC NIC by sending a jumbo frame packet with ICMP, caused modbus communication to completely fail and the NIC to stop responding to ping. ie. crashed the NIC. Beckhoff couplers were more resilient, only showed increased in response
time. This may be some other issue, but is recorded here for consideration. Before sending this packet, the NIC error light is not lit on the AD plcs.
-
Currently attempting to extract log from another hutch, PLC before powercycling, no log was present
|
- Replies:
- Re: Modbus devices lost connectivity Andrew Johnson
- RE: Modbus devices lost connectivity Mark Rivers
- Navigate by Date:
- Prev:
RE: FFT and waveform Mark Rivers
- Next:
Re: Modbus devices lost connectivity Andrew Johnson
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
<2016>
2017
2018
2019
2020
2021
2022
2023
2024
- Navigate by Thread:
- Prev:
Re: FFT and waveform Michael Davidsaver
- Next:
Re: Modbus devices lost connectivity Andrew Johnson
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
<2016>
2017
2018
2019
2020
2021
2022
2023
2024
|
ANJ, 15 Jul 2016 |
·
Home
·
News
·
About
·
Base
·
Modules
·
Extensions
·
Distributions
·
Download
·
·
Search
·
EPICS V4
·
IRMIS
·
Talk
·
Bugs
·
Documents
·
Links
·
Licensing
·
|