EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  <20232024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  <20232024 
<== Date ==> <== Thread ==>

Subject: RE: When I use an IOC in a container, streamDevice occasionally reports that protocol has been aborted, which causes the records in the IOC to become inaccessible from the host computer.
From: "Manoussakis, Adamandios via Tech-talk" <tech-talk at aps.anl.gov>
To: Zimoch Dirk <dirk.zimoch at psi.ch>, "Wang, Andrew" <wang126 at llnl.gov>
Cc: EPICS tech-talk <tech-talk at aps.anl.gov>
Date: Fri, 12 May 2023 21:04:37 +0000

Hi Dirk,

 

I will try to look into the possibility the containers are interfering/stalling, but we only seem to have issue with this one containerized IOC.  We have multiple containerized IOCs that speak with hardware and do not crash (which is odd trying to find out why we have this happening).  Also we have a few versions back system containerized IOCs that does not have these partial crashes we currently are seeing (might be a change we made in the IOC).  I doubt the system is overburdened either as its an enterprise server (2.8GHz 64 core 128gb ram) that everything is running on, we barely are touching the resources (1-3%) at the moment.   

 

Thanks for looking at this!

 

Adam

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Zimoch Dirk via Tech-talk
Sent: Friday, May 12, 2023 12:16 PM
To: Wang, Andrew <wang126 at llnl.gov>
Cc: EPICS tech-talk <tech-talk at aps.anl.gov>
Subject: Re: When I use an IOC in a container, streamDevice occasionally reports that protocol has been aborted, which causes the records in the IOC to become inaccessible from the host computer.

 

Hi Andy,

 

I suspect your container stalls from time to time, causing timeouts in the protocol.

I do not think that containers have been designed with real-time performance in mind. Thus I am not really surprised that it does not behave like a pure host. Maybe you can tune how the host schedules the containers? Or reduce the number of containers per host? It may simply be overburdened. Do you have any figures on the system load?

Dirk



Am 12.05.2023 um 20:06 schrieb Wang, Andrew via Tech-talk <tech-talk at aps.anl.gov>:



Hi all,

 

I have created multiple IOCs for the project in which I am involved. They are all running in their own Docker container in a host computer running Ubuntu 20.04. In each Docker container, the following EPICs and support module versions are used.

 

  1. EPICS: 7.0.4
  2. StreamDevice: 2.8.15
  3. Asyn: 4.41

 

In one of the IOCs, I have a SSEQ record that is used to push a scalar value to multiple records that set four parameters for the target instrument. There is an instance where streamDevice is unable to push the value to the second parameter, causing the protocol to abort. Then, a few minutes later, my colleagues and I have observed that no records from the IOC in question can be accessed through Channel Access. This is the error message that we receive.

 

Read operation timed out: some PV data was not read.

<RECORD_NAME> 0

CA.Client.Exception……………………………………………………..

Warning: “Virtual circuit disconnect”

Context: “op=0, channel=<RECORD_NAME>, type=DBR_TIME_DOUBLE, count=1, ctx=”<IP ADDRESS:PORT>”

Source File: ../getCopy.cpp line 91

Current Time: <TIME>

 

This also meant that I was unable to check the STAT field to see what the cause of the abortion was.

 

Thank you and I look forward to hearing back from everyone.

 

Andy

 

 

Purple ribbon awareness

 


References:
When I use an IOC in a container, streamDevice occasionally reports that protocol has been aborted, which causes the records in the IOC to become inaccessible from the host computer. Wang, Andrew via Tech-talk
Re: When I use an IOC in a container, streamDevice occasionally reports that protocol has been aborted, which causes the records in the IOC to become inaccessible from the host computer. Zimoch Dirk via Tech-talk

Navigate by Date:
Prev: Re: When I use an IOC in a container, streamDevice occasionally reports that protocol has been aborted, which causes the records in the IOC to become inaccessible from the host computer. Zimoch Dirk via Tech-talk
Next: RE: When I use an IOC in a container, streamDevice occasionally reports that protocol has been aborted, which causes the records in the IOC to become inaccessible from the host computer. Mark Rivers via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  <20232024 
Navigate by Thread:
Prev: Re: When I use an IOC in a container, streamDevice occasionally reports that protocol has been aborted, which causes the records in the IOC to become inaccessible from the host computer. Zimoch Dirk via Tech-talk
Next: RE: When I use an IOC in a container, streamDevice occasionally reports that protocol has been aborted, which causes the records in the IOC to become inaccessible from the host computer. Mark Rivers via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  <20232024 
ANJ, 12 May 2023 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·