Re: When I use an IOC in a container, streamDevice occasionally reports that protocol has been aborted, which causes the records in the IOC to become inaccessible from the host computer.

["Knap, Giles \(DLSLtd,RAL,LSCI\) via Tech-talk" <tech-talk at aps.anl.gov>]

I'd just like to add that we have several IOCs running in containers including AreaDetector (Aravis) and motion controllers (turbo pmac) and have not seen any issue like this.

However, I have yet to try a Stream Device, so I'm interested to see your results, Andrew.

There, challenges with CA discovery and container networking. Our solution is to use the host network in our IOC containers. How have you solved this and are you able to verify that network routing to/between containers is working?

---

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Wang, Andrew via Tech-talk <tech-talk at aps.anl.gov>
Sent: 13 May 2023 00:27
To: Mark Rivers <rivers at cars.uchicago.edu>; Zimoch Dirk <dirk.zimoch at psi.ch>
Cc: EPICS tech-talk <tech-talk at aps.anl.gov>
Subject: RE: When I use an IOC in a container, streamDevice occasionally reports that protocol has been aborted, which causes the records in the IOC to become inaccessible from the host computer.

 

Hi Mark and Dirk,

 

Thank you for your responses. I will do my best to identify if it is a container or EPICS issue by separating out the IOC from the container as soon as possible. It won’t be until later next week unfortunately.

 

From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Friday, May 12, 2023 3:29 PM
To: Zimoch Dirk <dirk.zimoch at psi.ch>; Wang, Andrew <wang126 at llnl.gov>
Cc: EPICS tech-talk <tech-talk at aps.anl.gov>
Subject: RE: When I use an IOC in a container, streamDevice occasionally reports that protocol has been aborted, which causes the records in the IOC to become inaccessible from the host computer.

 

Timeouts in the protocol should be recoverable.  From what I understand once this happens all CA access to the IOC is lost.  If so, this is a more serious issue.

 

Mark

 

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Zimoch Dirk via Tech-talk
Sent: Friday, May 12, 2023 2:16 PM
To: Wang, Andrew <wang126 at llnl.gov>
Cc: EPICS tech-talk <tech-talk at aps.anl.gov>
Subject: Re: When I use an IOC in a container, streamDevice occasionally reports that protocol has been aborted, which causes the records in the IOC to become inaccessible from the host computer.

 

Hi Andy,

 

I suspect your container stalls from time to time, causing timeouts in the protocol.

I do not think that containers have been designed with real-time performance in mind. Thus I am not really surprised that it does not behave like a pure host. Maybe you can tune how the host schedules the containers? Or reduce the number of containers per host? It may simply be overburdened. Do you have any figures on the system load?

Dirk

 

Am 12.05.2023 um 20:06 schrieb Wang, Andrew via Tech-talk <tech-talk at aps.anl.gov>:



Hi all,

 

I have created multiple IOCs for the project in which I am involved. They are all running in their own Docker container in a host computer running Ubuntu 20.04. In each Docker container, the following EPICs and support module versions are used.

 

1. EPICS: 7.0.4
2. StreamDevice: 2.8.15
3. Asyn: 4.41

 

In one of the IOCs, I have a SSEQ record that is used to push a scalar value to multiple records that set four parameters for the target instrument. There is an instance where streamDevice is unable to push the value to the second parameter, causing the protocol to abort. Then, a few minutes later, my colleagues and I have observed that no records from the IOC in question can be accessed through Channel Access. This is the error message that we receive.

 

Read operation timed out: some PV data was not read.

<RECORD_NAME> 0

CA.Client.Exception……………………………………………………..

Warning: “Virtual circuit disconnect”

Context: “op=0, channel=<RECORD_NAME>, type=DBR_TIME_DOUBLE, count=1, ctx=”<IP ADDRESS:PORT>”

Source File: ../getCopy.cpp line 91

Current Time: <TIME>

 

This also meant that I was unable to check the STAT field to see what the cause of the abortion was.

 

Thank you and I look forward to hearing back from everyone.

 

Andy

 

 

Purple ribbon awareness

 

 

-- 

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom