Re: Diagnosing and overcoming GigE camera freezes

[Michael Davidsaver via Tech-talk <tech-talk at aps.anl.gov>]

Do you monitor the temperature of these cameras?  Maybe they are overheating?


On 5/22/25 12:04, Henrique F. Simoes via Tech-talk wrote:
> Dear tech-talkers,
>
> In recent years, we at LNLS have faced some issues with Basler GigE cameras
> suddenly freezing, which are used for electron and photon beam visualization.
> In a typical scenario, the camera suddenly won't answer anything in its network
> interface, including ICMP (ping) and GVCP discovery packets (sent by the Aravis
> tool). This has shown to occur both in the accelerator and beamlines. The only
> way we know to recover from this situation is to power cycle the device.
>
> In the accelerator, where this happens more often, we power them with PoE Aruba
> switches, which enables us to power cycle them remotely through an SMNP-based
> IOC. In the beamlines, the situation is more uncomfortable: we power them with
> PoE injectors because the switches aren't PoE. This means a human must
> interfere, which makes the experience even worse.
>
> I don't think that's something only happening to us, as I've seen some mentions
> of cameras freezing in the accelerator community. In a response to a question
> related to the aforementioned Aruba switch IOC [1], John Priller mentioned that
>
> > We use the package to communicate with ISEG power supply crates, and remote
> > power-distribution units providing power on/off switching for things like
> > cameras that sometimes hang up due to magnetic fields or neutrons or etc in
> > our accelerator
> Evan Daykin also reported this in ADAravis [2].
>
> In the last EPICS Collab Meeting, Chris Gregory presented us their challenges
> at the STFC Central Laser Facility [3], which included a mention to
>
> > The camera IOC needs re-start because the IOC gets frozen after 4/5 days of
> > acquiring the image.
> So here goes my questions to you. If you faced similar issues in the past, what
> did you do to diagnose it and overcome it? If you still face them, how do you
> deal with it? If you use GigE cameras and never had such issues, which camera
> model do you use and in which environment?
>
> We are still investigating whether halts correlate with the camera temperature
> right before it happens, which is one of our current hypotheses. Another is the
> possibly high incoming traffic in its network interface (e.g. EPICS
> broadcasts), which we've seen lead to problems on Galil controllers in the
> past.
>
> Changing the manufacturer is another ongoing discussion internally, which aims
> to change the situation somehow. Any recommendation in this direction is very
> welcome as well. I've seen an older thread on this topic [4], but some models
> are no longer available, and I expect things to have changed after 7 years.
>
> Thanks in advance for any comments.
>
> [1] epics.anl.gov/tech-talk/2024/msg00890.php
> [2] github.com/areaDetector/ADAravis/issues/21
> [3] indico.stfc.ac.uk/event/1233/contributions/8412/
> [4] epics.anl.gov/tech-talk/2018/msg01918.php
>
> --
> Henrique F. Simões
> Control Software Group (SwC)
> Brazilian Synchrotron Light Laboratory (LNLS)
> Brazilian Center for Research in Energy and Materials (CNPEM)
>
> Aviso Legal: Esta mensagem e seus anexos podem conter informações confidenciais e/ou de uso restrito. Observe atentamente seu conteúdo e considere eventual consulta ao remetente antes de copiá-la, divulgá-la ou distribuí-la. Se você recebeu esta mensagem por engano, por favor avise o remetente e apague-a imediatamente.
>
> Disclaimer: This email and its attachments may contain confidential and/or privileged information. Observe its content carefully and consider possible querying to the sender before copying, disclosing or distributing it. If you have received this email by mistake, please notify the sender and delete it immediately.