Understanding alarms, alarm server and notification usage in other facilities

[Érico Nogueira Rolim via Tech-talk <tech-talk at aps.anl.gov>]

Hi folks!

As part of evaluating a unified solution for alarms and notifications at
Sirius, we are interested in hearing about the community's experiences
with alarm servers and related services.

What we want is a system to receive abstract "notifications", and turn
those into some kind of message, for example by SMS (for areas with bad
internet coverage), e-mail, MS Teams messages, and sound alarms in
control rooms.

The Phoebus alarm server, from our understanding, depends entirely on
alarms implemented in the EPICS layer. As explored in
https://urldefense.us/v3/__https://indico.global/event/14049/contributions/135608/__;!!G_uCfscf7eWS!axho_P0N7Qmsrf6LR1H9vQCbrykegZWwW1MhKVasQTSh6a8XvDnELShqUQeZ-2B4AY6_iOAsErQVNsZB_0Zho-cHIO4$  , there are some
limitations to these kinds of alarms, mainly that their conditions tend
to be static, even though there are situations that change the desired
thresholds and severity for their conditions: different shifts
(maintenance, commissioning, beam for users), different operation modes
(top-up, accumulation, single-bunch), etc. There's also the matter that
some limits might not be known when a hard IOC is being developed (by an
"expert") and will only be determined as part of commissioning or
ongoing operation experience, requiring that the expert be involved in
this step as well.

Some of the above can be worked around by adding a layer of soft IOCs on
top of the existing IOCs. These IOCs can be made up of records which
include additional logic for enabling alarms and might be simpler to
change and control. This requires developing and deploying an IOC
whenever such logic is necessary, can involve a lot of database
duplication, and requires someone knowledgeable in EPICS databases (or
some of the "soft IOC" alternatives) to implement it.

These solutions seem to address daily operation, with reasonably well
understood thresholds and conditions. However, we have observed a
variety of other needs from our operation and support staff: it's
possible that thresholds which are relevant to operation are not the
same as what subsystem experts want to be notified for; support staff
investigating malfunctioning devices might want warnings when the device
is within a certain operating range; someone running a procedure (be it
an experiment or part of commissioning) would like to be notified when
PVs reach some desired value.

These cases don't seem to fit in the "EPICS PV" model of alarm handling,
due to the amount of desired thresholds (more than 2 upper and 2 lower),
or the dynamic nature of what they want to check (the people involved
might not be familiar enough with EPICS development to write a soft IOC
on their own, and we don't want people deploying new IOCs with no
oversight). They seem to require some PV monitoring solution of their
own, possibly able to combine information from multiple PVs.

The way we see it, the implementations for turning "notifications" into
messages could be reused for both EPICS PVs and this other solution;
Phoebus Alarm Server's usage of Kakfa to decouple clients from the
server should simplify this, too.

For specific questions:

- Do the circumstances and requirements I described make sense? Are they
similar at your facility?

- What alarm server solution are you using at your facility? How is it
integrated with your communication tools to send notifications?

- How dynamic are your alarm conditions?

- How do you deal with alarms which can be ignored in certain situations?

- If you have implemented some kind of solution for dynamic alarms, what
does it look like?

Thank you very much,

Érico


Aviso Legal: Esta mensagem e seus anexos podem conter informações confidenciais e/ou de uso restrito. Observe atentamente seu conteúdo e considere eventual consulta ao remetente antes de copiá-la, divulgá-la ou distribuí-la. Se você recebeu esta mensagem por engano, por favor avise o remetente e apague-a imediatamente.

Disclaimer: This email and its attachments may contain confidential and/or privileged information. Observe its content carefully and consider possible querying to the sender before copying, disclosing or distributing it. If you have received this email by mistake, please notify the sender and delete it immediately.