Diagnosing and overcoming GigE camera freezes

["Henrique F. Simoes via Tech-talk" <tech-talk at aps.anl.gov>]

Dear tech-talkers,

In recent years, we at LNLS have faced some issues with Basler GigE cameras
suddenly freezing, which are used for electron and photon beam visualization.
In a typical scenario, the camera suddenly won't answer anything in its network
interface, including ICMP (ping) and GVCP discovery packets (sent by the Aravis
tool). This has shown to occur both in the accelerator and beamlines. The only
way we know to recover from this situation is to power cycle the device.

In the accelerator, where this happens more often, we power them with PoE Aruba
switches, which enables us to power cycle them remotely through an SMNP-based
IOC. In the beamlines, the situation is more uncomfortable: we power them with
PoE injectors because the switches aren't PoE. This means a human must
interfere, which makes the experience even worse.

I don't think that's something only happening to us, as I've seen some mentions
of cameras freezing in the accelerator community. In a response to a question
related to the aforementioned Aruba switch IOC [1], John Priller mentioned that

> We use the package to communicate with ISEG power supply crates, and remote
> power-distribution units providing power on/off switching for things like
> cameras that sometimes hang up due to magnetic fields or neutrons or etc in
> our accelerator

Evan Daykin also reported this in ADAravis [2].

In the last EPICS Collab Meeting, Chris Gregory presented us their challenges
at the STFC Central Laser Facility [3], which included a mention to

> The camera IOC needs re-start because the IOC gets frozen after 4/5 days of
> acquiring the image.

So here goes my questions to you. If you faced similar issues in the past, what
did you do to diagnose it and overcome it? If you still face them, how do you
deal with it? If you use GigE cameras and never had such issues, which camera
model do you use and in which environment?

We are still investigating whether halts correlate with the camera temperature
right before it happens, which is one of our current hypotheses. Another is the
possibly high incoming traffic in its network interface (e.g. EPICS
broadcasts), which we've seen lead to problems on Galil controllers in the
past.

Changing the manufacturer is another ongoing discussion internally, which aims
to change the situation somehow. Any recommendation in this direction is very
welcome as well. I've seen an older thread on this topic [4], but some models
are no longer available, and I expect things to have changed after 7 years.

Thanks in advance for any comments.

[1] epics.anl.gov/tech-talk/2024/msg00890.php
[2] github.com/areaDetector/ADAravis/issues/21
[3] indico.stfc.ac.uk/event/1233/contributions/8412/
[4] epics.anl.gov/tech-talk/2018/msg01918.php

--
Henrique F. Simões
Control Software Group (SwC)
Brazilian Synchrotron Light Laboratory (LNLS)
Brazilian Center for Research in Energy and Materials (CNPEM)

Aviso Legal: Esta mensagem e seus anexos podem conter informações confidenciais e/ou de uso restrito. Observe atentamente seu conteúdo e considere eventual consulta ao remetente antes de copiá-la, divulgá-la ou distribuí-la. Se você recebeu esta mensagem por engano, por favor avise o remetente e apague-a imediatamente.

Disclaimer: This email and its attachments may contain confidential and/or privileged information. Observe its content carefully and consider possible querying to the sender before copying, disclosing or distributing it. If you have received this email by mistake, please notify the sender and delete it immediately.