|
At CHESS we use GigE cameras from Allied Vision. These also stop on occasion. We have been struggling to figure out if it is temperature or radiation.
They also seem significantly more likely to fail on power cycle, i.e. they are working and then there is a power outage (scheduled or unscheduled) and we discover multiple cameras dead.
We happen to use some of these cameras on computer which has multiple IP addresses assigned to one NIC. This causes a significant memory leak in the Allied Vision driver which requires us to restart those IOCs weekly, but this is a pretty unique case.
For the few cameras that are critical to operations we have a watchdog base on image count. This at least lets us handle the interruption more gracefully.
John Dobbins
Research Support Specialist
Cornell High Energy Synchrotron Source
Cornell University
From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Henrique F. Simoes via Tech-talk <tech-talk at aps.anl.gov>
Sent: Thursday, May 22, 2025 3:04 PM
To: Tech-talk at aps.anl.gov <Tech-talk at aps.anl.gov>
Cc: Eduardo Pereira Coelho <eduardo.coelho at lnls.br>; christopher.gregory at stfc.ac.uk <christopher.gregory at stfc.ac.uk>
Subject: Diagnosing and overcoming GigE camera freezes
Dear tech-talkers,
In recent years, we at LNLS have faced some issues with Basler GigE cameras
suddenly freezing, which are used for electron and photon beam visualization.
In a typical scenario, the camera suddenly won't answer anything in its network
interface, including ICMP (ping) and GVCP discovery packets (sent by the Aravis
tool). This has shown to occur both in the accelerator and beamlines. The only
way we know to recover from this situation is to power cycle the device.
In the accelerator, where this happens more often, we power them with PoE Aruba
switches, which enables us to power cycle them remotely through an SMNP-based
IOC. In the beamlines, the situation is more uncomfortable: we power them with
PoE injectors because the switches aren't PoE. This means a human must
interfere, which makes the experience even worse.
I don't think that's something only happening to us, as I've seen some mentions
of cameras freezing in the accelerator community. In a response to a question
related to the aforementioned Aruba switch IOC [1], John Priller mentioned that
> We use the package to communicate with ISEG power supply crates, and remote
> power-distribution units providing power on/off switching for things like
> cameras that sometimes hang up due to magnetic fields or neutrons or etc in
> our accelerator
Evan Daykin also reported this in ADAravis [2].
In the last EPICS Collab Meeting, Chris Gregory presented us their challenges
at the STFC Central Laser Facility [3], which included a mention to
> The camera IOC needs re-start because the IOC gets frozen after 4/5 days of
> acquiring the image.
So here goes my questions to you. If you faced similar issues in the past, what
did you do to diagnose it and overcome it? If you still face them, how do you
deal with it? If you use GigE cameras and never had such issues, which camera
model do you use and in which environment?
We are still investigating whether halts correlate with the camera temperature
right before it happens, which is one of our current hypotheses. Another is the
possibly high incoming traffic in its network interface (e.g. EPICS
broadcasts), which we've seen lead to problems on Galil controllers in the
past.
Changing the manufacturer is another ongoing discussion internally, which aims
to change the situation somehow. Any recommendation in this direction is very
welcome as well. I've seen an older thread on this topic [4], but some models
are no longer available, and I expect things to have changed after 7 years.
Thanks in advance for any comments.
[1] epics.anl.gov/tech-talk/2024/msg00890.php
[2] github.com/areaDetector/ADAravis/issues/21
[3] indico.stfc.ac.uk/event/1233/contributions/8412/
[4] epics.anl.gov/tech-talk/2018/msg01918.php
--
Henrique F. Simões
Control Software Group (SwC)
Brazilian Synchrotron Light Laboratory (LNLS)
Brazilian Center for Research in Energy and Materials (CNPEM)
Aviso Legal: Esta mensagem e seus anexos podem conter informações confidenciais e/ou de uso restrito. Observe atentamente seu conteúdo e considere eventual consulta ao remetente antes de copiá-la, divulgá-la ou distribuí-la. Se você recebeu esta mensagem por
engano, por favor avise o remetente e apague-a imediatamente.
Disclaimer: This email and its attachments may contain confidential and/or privileged information. Observe its content carefully and consider possible querying to the sender before copying, disclosing or distributing it. If you have received this email by mistake,
please notify the sender and delete it immediately.
W
|
