Re: GNU/Linux and GigE camera settings for high-throughput with ADAravis

["O'Hea, James \(DLSLtd, RAL, LSCI\) via Tech-talk" <tech-talk at aps.anl.gov>]

Hi Henrique,

On the Linux server side you can increase your received packet buffers:

/proc/sys/net/core/rmem_default

and

/proc/sys/net/core/rmem_max

Kind regards,

James

Diamond Light Source

---

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Henrique F. Simoes via Tech-talk <tech-talk at aps.anl.gov>
Sent: 10 January 2024 12:53
To: tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: GNU/Linux and GigE camera settings for high-throughput with ADAravis

 

Hi tech-talkers,

I've been setting up a GigE camera from Allied Vision (Alvium G1-500c) with
ADAravis (areaDetector R3-12-1) in a GNU/Linux machine recently and had
struggled to have the machine and camera setup for its maximum throughput (2952
x 1944 x 3 @ 8fps). A typical problematic acquisition would be full of failed
frames due to timeouts.

For a local test setup, I've been able to manage achieve a high throughput by:

- Setting the GevSCPSPacketSize feature to 1500B (defaults to 576B), matching
  MTU [1]. This was not updated by Aravis auto packet size procedure (I haven't
  investigated why yet).
- Using Packet Socket (by enabling NET_RAW Linux capability for the process)
  [3, 4]
- Increasing receive buffer size (through sysctl net.core.rmem_max) [1]
- Increasing the number of RX ring buffers through ethtool --set-ring

For instance, this made me achieve 7.4Hz lossless traffic at full resolution
(~380k images acquired). I haven't yet checked how each of these influence
individually, though. Some of them might not be required for achieving this
result.

At even higher rates, specification limit (8.12Hz, in this case), I start to
see some timeouts until all frames become corrupted with something like this:

2024/01/09 20:22:00.427 ADAravis:lookupColorMode: Could not find a match for pixel format: 0
2024/01/09 20:22:00.427 ADAravis:processBuffer: unknown pixel format 0
2024/01/09 20:22:00.683 ADAravis::newBufferCallback bad frame status: Timeout

** (process:2): CRITICAL **: 20:22:00.683: arv_buffer_get_image_pixel_format: assertion 'arv_buffer_payload_type_has_aoi (buffer->priv->payload_type)' failed
** (process:2): CRITICAL **: 20:22:00.683: arv_buffer_get_image_width: assertion 'arv_buffer_payload_type_has_aoi (buffer->priv->payload_type)' failed
** (process:2): CRITICAL **: 20:22:00.683: arv_buffer_get_image_height: assertion 'arv_buffer_payload_type_has_aoi (buffer->priv->payload_type)' failed
** (process:2): CRITICAL **: 20:22:00.683: arv_buffer_get_image_x: assertion 'arv_buffer_payload_type_has_aoi (buffer->priv->payload_type)' failed
** (process:2): CRITICAL **: 20:22:00.683: arv_buffer_get_image_y: assertion 'arv_buffer_payload_type_has_aoi (buffer->priv->payload_type)' failed

I've also seen recommendations for setting the maximum possible MTU (jumbo
frame) [1, 3], but my NIC (a Broadcom NetXtreme BCM5764M) does not support more
than 1500B. Another recommendation is to enable link-layer flow control at the
NIC [1] (and gateway switch), but I couldn't capture any pause frame packet
sent in my setting (a misconfiguration, perhaps?).

Decreasing DeviceLinkThroughputLimit (and turning DeviceLinkThroughputLimitMode
on) showed to be helpful in some scenarios [2], but it actually restricts
transfer rate, making the maximum possible frame rate below the camera's
specification. Hence, its traffic becomes less prone to failures.

After replicating this configuration (as close as I could) in our deployment
machine, the failure rate raised to about 2.8% (of 940k continuously acquired
images) with a lower rate of 6.4Hz (with throughput limit set up to its
minimum). The machine has another NIC (an Intel 82583V), thus another driver
and configuration options. We also use another switch model (possibly with
other settings).

Is there anything else you do to tune GNU/Linux machines for supporting these
GigE cameras?

My intention here is twofold: increase our knowledge of configuration options
and document this for future reference.

Best regards,
Henrique F. Simões

[1]: Tips and tricks to connect 1000BASE-T in Alvium G1 User Guide. Available
at
<https://eur03.safelinks.protection.outlook.com/?url="">>
[2]: https://eur03.safelinks.protection.outlook.com/?url="">
[3]: https://eur03.safelinks.protection.outlook.com/?url="">
[4]: https://eur03.safelinks.protection.outlook.com/?url="">


Aviso Legal: Esta mensagem e seus anexos podem conter informações confidenciais e/ou de uso restrito. Observe atentamente seu conteúdo e considere eventual consulta ao remetente antes de copiá-la, divulgá-la ou distribuí-la. Se você recebeu esta mensagem por engano, por favor avise o remetente e apague-a imediatamente.

Disclaimer: This email and its attachments may contain confidential and/or privileged information. Observe its content carefully and consider possible querying to the sender before copying, disclosing or distributing it. If you have received this email by mistake, please notify the sender and delete it immediately.

 

-- 

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom