Re: alarm handling in EPICS 7

[Pedro Gigoux via Tech-talk <tech-talk at aps.anl.gov>]

Hi Andrew,

Thanks for your prompt reply. I attached the schematic and the test database we used to isolate the problem. We have two IOCs: one that provides the record SYM:HEX01:CONTROLON and a second IOC that has three records:

• test:ai: Reads data from the first IOC (INP : CA_LINK SYM:HEX01:CONTROLON NPP NMS)
• test:calc: Reads data from test:ai (INPA: DB_LINK test:ai.VAL PP NMS) and increments a counter. INPA is set to PP to trigger reading.
  
• test:ao: Receives the value of the counter.

If the two IOC are up, we see the following:

test:ai.SEVR                   2025-03-28 14:43:05.111621 NO_ALARM  
test:ai.STAT                   2025-03-28 14:43:05.111621 NO_ALARM  
test:calc.SEVR                 2025-03-28 14:43:05.111623 NO_ALARM  
test:calc.STAT                 2025-03-28 14:43:05.111623 NO_ALARM  
test:ao.SEVR                   2025-03-28 14:43:05.111624 NO_ALARM  
test:ao.STAT                   2025-03-28 14:43:05.111624 NO_ALARM  
test:ao.VAL                    2025-03-28 14:43:05.111624 3  
test:ao.VAL                    2025-03-28 14:43:06.111855 4  
test:ao.VAL                    2025-03-28 14:43:07.111749 5  
test:ao.VAL                    2025-03-28 14:43:08.111765 6  
test:ao.VAL                    2025-03-28 14:43:09.111681 7

If I stop the IOC that has SYM:HEX01:CONTROLON, then the alarm severity changes and the counter stops updating, i.e. test:calc stops processing:

test:ao.VAL                    2025-03-28 14:43:38.111802 36  
test:ao.VAL                    2025-03-28 14:43:39.111805 37  
test:ao.VAL                    2025-03-28 14:43:40.111659 38  
test:ao.VAL                    2025-03-28 14:43:41.111668 39  
test:ai.SEVR                   2025-03-28 14:43:42.111818 INVALID LINK INVALID
test:ai.STAT                   2025-03-28 14:43:42.111818 LINK LINK INVALID
test:calc.SEVR                 2025-03-28 14:43:42.111826 INVALID LINK INVALID
test:calc.STAT                 2025-03-28 14:43:42.111826 LINK LINK INVALID

If I keep the IOC down and set test:ai.SIMM=1 then the alarm is cleared and the counter starts updating:

test:ai.SEVR                   2025-03-28 14:44:18.111641 NO_ALARM  
test:ai.STAT                   2025-03-28 14:44:18.111641 NO_ALARM  
test:calc.SEVR                 2025-03-28 14:44:18.111657 NO_ALARM  
test:calc.STAT                 2025-03-28 14:44:18.111657 NO_ALARM  
test:ao.VAL                    2025-03-28 14:44:18.111662 40  
test:ao.VAL                    2025-03-28 14:44:19.111728 41  
test:ao.VAL                    2025-03-28 14:44:20.111719 42  
test:ao.VAL                    2025-03-28 14:44:21.111760 43  
test:ao.VAL                    2025-03-28 14:44:22.111726 44

In EPICS 3.14, the records keep processing if the IOC goes down.

Thank you,

Pedro.

On Fri, 28 Mar 2025 at 14:19, Johnson, Andrew N. <anj at anl.gov> wrote:

Hi Pedro,

 

Can you please post a concrete example of a record configuration that used to work in EPICS 3.14.x or 3.15.x and no longer does in EPICS 7.0.x? If you can simplify that to a small number of soft records in each of 2 IOCs that would help us understand and replicate your specific issue. I don’t immediately recognize it as anything that we’ve explicitly changed, but we might have broken your use-case by mistake. Once we see the specific problem we may be able to suggest alternative configurations.

 

Thanks,

 

- Andrew

 

-- 

Complexity comes for free, Simplicity you have to work for.

 

 

On 3/28/25, 3:51 PM, "Tech-talk" <tech-talk at aps.anl.gov> wrote:

 

Hello,

 

I am writing to get your advice on managing system unavailability within EPICS 7. In our current operational model we can switch between different instruments seamlessly if one encounters an issue and becomes unavailable. This strategy was effective in previous EPICS versions. However, after migrating to EPICS 7, records reading data from systems that are no longer available stop processing, even if we don't rely on the data from that particular system to continue observing. The issue arises because broken CA links set the alarm severity (SEVR) to INVALID, the alarm status (STAT) to LINK, and halt the record processing of the downstream records. We want a mechanism to override the alarm severity when an instrument becomes unresponsive, ideally with minimal operator intervention.

 

We have identified three potential ways of achieving this:

·         Maximize Severity Attribute: The idea was to use this attribute to prevent the alarm propagation, but it seems that it does not provide what we need.

·         SIMM Field: Setting the SIMM field to YES enables the record to continue processing without being affected by the INVALID alarm status. The SVAL field can be used to define a simulation value and the SIMS field specifies the simulation mode alarm severity (NO_ALARM in our case). We have tested this approach and it seems to work well. STAT is set to SIMM and the downstream records process without problems.

·         DISA Field: Making DISA=DISV disables the record. The DISS field defines the record's disable severity (e.g. NO_ALARM). This approach also seems to work. STAT is set to DISABLE and the downstream records process without problems as well.

The SIMM field seems to be the most promising option. I would greatly appreciate your insights on this, as well as any alternative approaches that you might suggest.

Thank you,
Pedro

Attachment: schematic.png
Description: PNG image

Attachment: test.db
Description: Binary data