Experimental Physics and
Industrial Control System

Kevin Meyer <[email protected]> · Mon, 28 Nov 2016 17:17:30 +0100 (CET)

I have a weird situation: On the same host I have one IOC (with all PVs) and about 4 stand-alone SNL sequencer applications (with no databases) that interact with each other (interfaced via the PVs).

In one of the sequencers, I have a call out to to a Python script that calls a web service (to Olog, to create an electronic lab-book entry).

If all the sequencers are running, I can get this one sequencer to lose low-level connection to the network layer and EPICS.
The pvConnectCount() returns 7, for example, but a monitored PV does not get updated. I can terminate the IOC and still the pvConnectCount() shows 7.

If I manually stop the sequencer (seqStop xxxx), the try and restart it (seq xxxx), it then fails to start (even if the IOC and all other sequencers are still functioning normally). 

To repeat: when the "sensitive" sequencer stops working, no other application is affected - all the other sequencers still work, "caget/caput" from the command-line still works, etc.

Sometimes the /var/log/messages shows e1000 network module issues:
Nov 23 14:10:27 localhost kernel: e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
Nov 23 14:11:42 localhost kernel: e1000: eth0: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Nov 23 14:11:52 localhost kernel: e1000: eth0: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
but not always!

To make matters more bizarre, if I stop one of the other sequencers, then this troublesome one functions perfectly!!

Some system information:
hostname 2.6.32-431.20.3.el6.x86_64 #1 SMP Fri Jun 6 18:30:54 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
epics          : Experimental Physics and Industrial Control Syste (3.15.5~rc1)

Has anyone seen anything like this before? Do you have any hints for me, where to look?

Thanks,
Kevin

--
Kevin Meyer
Cosylab, Ljubljana, Slovenia
Telephone: +386-(0)1-320-47-82   Mobile: +386-(0)70-260-321

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System