RTEMS NFS/RPIOC problems?

[Ricardo Cardenes <rcardenes@gemini.edu>]

Hi,

We've been experiencing some problems on some of our systems (based on MVME 2700/3100; EPICS 3.14.12.7, RTEMS 4.10.2) related to the RPC subsystem, like these:

2018-06-04-tcs.log:Jun  4 12:30:06 E) PORT: tcs_vme, MSG: RPCIO: server '172.17.71.11' not responding - still trying
[...]

\2018-06-07-tcs.log:Jun  7 13:11:37 E) PORT: tcs_vme, MSG: RPCIO: server '172.17.2.10' not responding - still trying
[...]

2018-06-26-tcs.log:Jun 26 16:43:14  E) PORT: tcs_vme, MSG: RPCIO - statistics: already 17000 retries to server 172.17.2.10
2018-06-26-tcs.log:Jun 26 20:57:43  E) PORT: tcs_vme, MSG: RPCIO - statistics: already 18000 retries to server 172.17.2.10

And it seems to me that we're getting too many CA disconnects, too. We hadn't noticed until recently, when we commissioned a system that seems the only one being systematically affected (but it is the one that orchestrates most of the others, too...)

We suspect some hardware or networking problem, possibly outside of our boards, given that the interface stats look quite clear (no corrupt Ethernet frames, no retries, ...), and we've actually seen at least one corrupt UDP header, plus several thousand occurrences of timed out IP fragments (over a period of a few weeks).

In any case, while we investigate the possible hardware issue, has anyone else experienced a similar problem?

Regards,

Ricardo