RE: Diagnosing and overcoming GigE camera freezes

[Abdalla Ahmad via Tech-talk <tech-talk at aps.anl.gov>]

Hi Henrique

> "That's a bit surprising for me. I've seen jumbo frames help the IOC keep up with the high input traffic on a modest machine, but I've never considered that it could cause occasional disconnections."
Sorry I may have used the wrong term here because I got confused with another camera, what I meant was very frequent freezes of acquisition, the camera just stops acquiring frames. Unfortunately, I could not remember if the IOC had any error messages.

> "Have you ever disabled jumbo frames on the intermediate switches in the first setup, and validated that this was the only difference in both scenarios?"
The cameras were working without jumbo frames, they would just freeze very frequently. When enabling jumbo frames, acquisition improved but with some freezes from time to time. The solution was to connect the camera to a much better switch.

> "Do you recall what temperature approximately the camera would reach before freezing?"
I remember the camera would reach a temperature of around 55 C in normal operation, but it would not last much time before shutdown. I tried looking up any info on thermal shutdown but could not find it.

I don't know if anyone would agree with this or not, but I think the best solution would be to isolate the cameras completely in their separate network, we proposed a setup were the cameras are connected to their own gigabit and PoE switch and a separate workstation running the IOC with two NICs one for the cameras and one for the main network. We did not proceed because the aforementioned solution is working for us right now.

Best Regards,
Abdalla Al-Dalleh
Control Engineer
SESAME

-----Original Message-----
From: Henrique F. Simoes <henrique.simoes at lnls.br> 
Sent: Wednesday, May 28, 2025 9:51 PM
To: Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo>
Cc: Eduardo Pereira Coelho <eduardo.coelho at lnls.br>; christopher.gregory at stfc.ac.uk; tech-talk at aps.anl.gov
Subject: Re: Diagnosing and overcoming GigE camera freezes

On Monday, May 26, 2025 3:44 AM, Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo> wrote:
> I would like to share our experience with Basler GigE cameras 
> (acA1300). We use them to visualize the beam from the injector to the 
> storage-ring, we use them at some beamlines but controlled via Pylon 
> Viewer. Right now, they are controlled with ADPylon. The issue was 
> there was a lot of disconnection with the cameras, it was a network 
> setup issue:
> * Jumbo frames were enabled (especially on intermediate switches)

That's a bit surprising for me. I've seen jumbo frames help the IOC keep up with the high input traffic on a modest machine, but I've never considered that it could cause occasional disconnections.

Our intermediate switches do have them enabled, while most IOC hosts don't.

> * A certain camera was moved to newer network switch for testing.
>
> Right now, on this camera, we can achieve between 1.5M and 2M frames 
> continuously at 10 Hz frame rate with much less disconnection.

Have you ever disabled jumbo frames on the intermediate switches in the first setup, and validated that this was the only difference in both scenarios?

> We used the ace2 model for some period and it used to freeze due to 
> overheating, we removed the camera so we never got the chance to 
> troubleshoot.

Do you recall what temperature approximately the camera would reach before freezing?

> Best Regards,
> Abdalla Al-Dalleh
> Control Engineer
> SESAME

Thanks for sharing your experiences, Abdalla!

--
Henrique F. Simões
Control Software Group (SwC)
Brazilian Synchrotron Light Laboratory (LNLS) Brazilian Center for Research in Energy and Materials (CNPEM)

Aviso Legal: Esta mensagem e seus anexos podem conter informações confidenciais e/ou de uso restrito. Observe atentamente seu conteúdo e considere eventual consulta ao remetente antes de copiá-la, divulgá-la ou distribuí-la. Se você recebeu esta mensagem por engano, por favor avise o remetente e apague-a imediatamente.

Disclaimer: This email and its attachments may contain confidential and/or privileged information. Observe its content carefully and consider possible querying to the sender before copying, disclosing or distributing it. If you have received this email by mistake, please notify the sender and delete it immediately.