RE: Diagnosing and overcoming GigE camera freezes

[Abdalla Ahmad via Tech-talk <tech-talk at aps.anl.gov>]

Hi

I would like to share our experience with Basler GigE cameras (acA1300). We use them to visualize the beam from the injector to the storage-ring, we use them at some beamlines but controlled via Pylon Viewer. Right now, they are controlled with ADPylon. The issue was there was a lot of disconnection with the cameras, it was a network setup issue:
* Jumbo frames were enabled (especially on intermediate switches)
* A certain camera was moved to newer network switch for testing.

Right now, on this camera, we can achieve between 1.5M and 2M frames continuously at 10 Hz frame rate with much less disconnection. We used the ace2 model for some period and it used to freeze due to overheating, we removed the camera so we never got the chance to troubleshoot.

Best Regards,
Abdalla Al-Dalleh
Control Engineer
SESAME

-----Original Message-----
From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Henrique F. Simoes via Tech-talk
Sent: Thursday, May 22, 2025 10:04 PM
To: Tech-talk at aps.anl.gov
Cc: Eduardo Pereira Coelho <eduardo.coelho at lnls.br>; christopher.gregory at stfc.ac.uk
Subject: Diagnosing and overcoming GigE camera freezes

Dear tech-talkers,

In recent years, we at LNLS have faced some issues with Basler GigE cameras suddenly freezing, which are used for electron and photon beam visualization.
In a typical scenario, the camera suddenly won't answer anything in its network interface, including ICMP (ping) and GVCP discovery packets (sent by the Aravis tool). This has shown to occur both in the accelerator and beamlines. The only way we know to recover from this situation is to power cycle the device.

In the accelerator, where this happens more often, we power them with PoE Aruba switches, which enables us to power cycle them remotely through an SMNP-based IOC. In the beamlines, the situation is more uncomfortable: we power them with PoE injectors because the switches aren't PoE. This means a human must interfere, which makes the experience even worse.

I don't think that's something only happening to us, as I've seen some mentions of cameras freezing in the accelerator community. In a response to a question related to the aforementioned Aruba switch IOC [1], John Priller mentioned that

> We use the package to communicate with ISEG power supply crates, and 
> remote power-distribution units providing power on/off switching for 
> things like cameras that sometimes hang up due to magnetic fields or 
> neutrons or etc in our accelerator

Evan Daykin also reported this in ADAravis [2].

In the last EPICS Collab Meeting, Chris Gregory presented us their challenges at the STFC Central Laser Facility [3], which included a mention to

> The camera IOC needs re-start because the IOC gets frozen after 4/5 
> days of acquiring the image.

So here goes my questions to you. If you faced similar issues in the past, what did you do to diagnose it and overcome it? If you still face them, how do you deal with it? If you use GigE cameras and never had such issues, which camera model do you use and in which environment?

We are still investigating whether halts correlate with the camera temperature right before it happens, which is one of our current hypotheses. Another is the possibly high incoming traffic in its network interface (e.g. EPICS broadcasts), which we've seen lead to problems on Galil controllers in the past.

Changing the manufacturer is another ongoing discussion internally, which aims to change the situation somehow. Any recommendation in this direction is very welcome as well. I've seen an older thread on this topic [4], but some models are no longer available, and I expect things to have changed after 7 years.

Thanks in advance for any comments.

[1] epics.anl.gov/tech-talk/2024/msg00890.php
[2] github.com/areaDetector/ADAravis/issues/21
[3] indico.stfc.ac.uk/event/1233/contributions/8412/
[4] epics.anl.gov/tech-talk/2018/msg01918.php

--
Henrique F. Simões
Control Software Group (SwC)
Brazilian Synchrotron Light Laboratory (LNLS) Brazilian Center for Research in Energy and Materials (CNPEM)

Aviso Legal: Esta mensagem e seus anexos podem conter informações confidenciais e/ou de uso restrito. Observe atentamente seu conteúdo e considere eventual consulta ao remetente antes de copiá-la, divulgá-la ou distribuí-la. Se você recebeu esta mensagem por engano, por favor avise o remetente e apague-a imediatamente.

Disclaimer: This email and its attachments may contain confidential and/or privileged information. Observe its content carefully and consider possible querying to the sender before copying, disclosing or distributing it. If you have received this email by mistake, please notify the sender and delete it immediately.