Re: Fast stream-device IOC suddenly disconnects

[Mark Rivers via Tech-talk <tech-talk@aps.anl.gov>]

Hi Abdalla,

I have been thinking about this some more, and the behavior is very puzzling.

• You suspect that the problem is in the firmware of the serial device.
• You are using a terminal server to connect to the serial device.
• When it hangs up you report that disconnecting the asyn IP port, i.e. closing and reopening the socket to the terminal server restores the asynchronous device updates.
• You are pretty certain that using disconnectOnReadTimeout would also restore the asynchronous device updates.

This behavior would make sense if the device was a directly connected over Ethernet, since something could go wrong on the IP stack on the device, and closing and reopening the socket could fix it.

However, because your are using a terminal server the device itself should have no way of knowing that you have closed and reopened the socket connection to the terminal server.  The terminal server should not be sending any characters to the device when the socket reconnects. So why would it start communicating again after you do that?

There are 2 things I can think of that would explain this:

• The issue is actually with the terminal server, not the device.  I know you said you swapped the terminal server, but did you use a different manufacturer or model?
• If you have enabled flow control between the terminal server and the device then perhaps the device would have a way of knowing when the terminal server connection is reset.  That could either be hardware flow control or XON/XOFF.  Have you enabled flow control on that terminal server channel?

Mark

---

From: Abdalla Ahmad <Abdalla.Ahmad@sesame.org.jo>
Sent: Friday, October 3, 2025 1:11 PM
To: Mark Rivers <rivers@cars.uchicago.edu>
Cc: tech-talk@aps.anl.gov <tech-talk@aps.anl.gov>
Subject: Re: Fast stream-device IOC suddenly disconnects

 

Hello Mark

I wanted to wait for it to disconnect again to investigate further, unfortunately still the ioc shell shows no error messages and the terminal statistics just stops counting. I know disconnectOnReadTimeout will solve it but I really want to troubleshoot further and see if something can be done on the device's firmware side because we changed the terminal server two times with the same scenario. 

Best Regards 

Abdalla

---

From: Mark Rivers <rivers@cars.uchicago.edu>
Sent: Friday, October 3, 2025 5:18:24 PM
To: Abdalla Ahmad <Abdalla.Ahmad@sesame.org.jo>
Cc: tech-talk@aps.anl.gov <tech-talk@aps.anl.gov>
Subject: Re: Fast stream-device IOC suddenly disconnects

Hi Abdalla,

I’ve made 3 suggestions:

- Set disconnectOnReadTimeout

- Log all communication to see if it stops

- Look at the statistics in the terminal server

Have you tried these?

Mark

Sent from my iPhone

On Sep 28, 2025, at 10:08 AM, Abdalla Ahmad <Abdalla.Ahmad@sesame.org.jo> wrote:



Hi Mark

Thanks for the info. I forgot to mention that the device is a serial port connected to a terminal server, and the device has been tested locally streaming data for periods longer than what we have so I don't think it just stopped sending data.

Best Regards 

Abdalla.

TEST
Get Outlook for Android

---

From: Mark Rivers <rivers@cars.uchicago.edu>
Sent: Sunday, September 28, 2025 5:02:36 PM
To: tech-talk@aps.anl.gov <Tech-talk@aps.anl.gov>; Abdalla Ahmad <Abdalla.Ahmad@sesame.org.jo>
Subject: Re: Fast stream-device IOC suddenly disconnects

 

Hi Abdalla,

It looks like the device streams data without needing to send a command, i.e. it is not a write/read operation, correct?

If it freezes then that suggests that the device has stopped sending data for some reason.  That should mean that the asyn read operation timed out, and StreamDevice should be reporting that?

You can try adding this to the startup script after drvAsynIPPortConfigure:

asynSetOption($(PORT), 0, "disconnectOnReadTimeout", "Y")

That will disconnect the port when a read times out, which is equivalent to writing 0 to the CNCT field.

Mark

---

From: Tech-talk <tech-talk-bounces@aps.anl.gov> on behalf of Abdalla Ahmad via Tech-talk <tech-talk@aps.anl.gov>
Sent: Sunday, September 28, 2025 6:08 AM
To: tech-talk@aps.anl.gov <Tech-talk@aps.anl.gov>
Subject: Fast stream-device IOC suddenly disconnects

 

Hi

 

We have a device streaming data at 200 Hz controlled with stream device, parses the streamed data and writes to corresponding PVs. The problem is that every day or two, the readings freeze. At first I had to restart the IOC but I noticed that the IOC does not show any error so I re-connect the asyn port through writing 0 to the CNCT field and it worked. The st.cmd file has the port configured:

 

drvAsynIPPortConfigure(“PORT”, “ip-address:port-number”, 0, 0, 0)

 

What could be the issue here? I am using EPICS base 7.0.9 and latest version of all modules.

 

Best Regards,

Abdalla Al-Dalleh

Control Engineer

SESAME