Re: GNU/Linux and GigE camera settings for high-throughput with ADAravis

[Mark Rivers via Tech-talk <tech-talk at aps.anl.gov>]

Hi Enrique,

As James said, you should increase net.core.rmem_default, not just net.core.rmem_max.  This is documented in the ADGenICam documentation

https://areadetector.github.io/master/ADGenICam/ADGenICam.html#linux-usb-and-gige-system-settings

The Alvium G1-500C shows a maximum frame rate of 23.0 frames/s in mono8 mode.  That means that in RGB mode the limit is 23.0/3 = 7.66 fames/s.

Here are some suggestions:

• Is the camera on a network with other devices, or a dedicated NIC on the Linux machine?  If on a shared network how many switches between the camera and IOC machine?  If there are shared links other network traffic could be the problem.  A dedicated NIC is often a good solution.
• You can try using ADVimba rather than ADAravis.  It may have more optimized network handling.
• If you use ADVimba you can make the camera run in color mode at the full 23 frames/s.  You do that by using Bayer mode to read the camera, and do the conversion to RGB in the IOC, rather than on the camera.  This reduces the network traffic by a factor of 3.  ADAravis does not currently support this option.

Mark

---

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Henrique F. Simoes via Tech-talk <tech-talk at aps.anl.gov>
Sent: Wednesday, January 10, 2024 6:53 AM
To: tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: GNU/Linux and GigE camera settings for high-throughput with ADAravis

 

Hi tech-talkers,

I've been setting up a GigE camera from Allied Vision (Alvium G1-500c) with
ADAravis (areaDetector R3-12-1) in a GNU/Linux machine recently and had
struggled to have the machine and camera setup for its maximum throughput (2952
x 1944 x 3 @ 8fps). A typical problematic acquisition would be full of failed
frames due to timeouts.

For a local test setup, I've been able to manage achieve a high throughput by:

- Setting the GevSCPSPacketSize feature to 1500B (defaults to 576B), matching
  MTU [1]. This was not updated by Aravis auto packet size procedure (I haven't
  investigated why yet).
- Using Packet Socket (by enabling NET_RAW Linux capability for the process)
  [3, 4]
- Increasing receive buffer size (through sysctl net.core.rmem_max) [1]
- Increasing the number of RX ring buffers through ethtool --set-ring

For instance, this made me achieve 7.4Hz lossless traffic at full resolution
(~380k images acquired). I haven't yet checked how each of these influence
individually, though. Some of them might not be required for achieving this
result.

At even higher rates, specification limit (8.12Hz, in this case), I start to
see some timeouts until all frames become corrupted with something like this:

2024/01/09 20:22:00.427 ADAravis:lookupColorMode: Could not find a match for pixel format: 0
2024/01/09 20:22:00.427 ADAravis:processBuffer: unknown pixel format 0
2024/01/09 20:22:00.683 ADAravis::newBufferCallback bad frame status: Timeout

** (process:2): CRITICAL **: 20:22:00.683: arv_buffer_get_image_pixel_format: assertion 'arv_buffer_payload_type_has_aoi (buffer->priv->payload_type)' failed
** (process:2): CRITICAL **: 20:22:00.683: arv_buffer_get_image_width: assertion 'arv_buffer_payload_type_has_aoi (buffer->priv->payload_type)' failed
** (process:2): CRITICAL **: 20:22:00.683: arv_buffer_get_image_height: assertion 'arv_buffer_payload_type_has_aoi (buffer->priv->payload_type)' failed
** (process:2): CRITICAL **: 20:22:00.683: arv_buffer_get_image_x: assertion 'arv_buffer_payload_type_has_aoi (buffer->priv->payload_type)' failed
** (process:2): CRITICAL **: 20:22:00.683: arv_buffer_get_image_y: assertion 'arv_buffer_payload_type_has_aoi (buffer->priv->payload_type)' failed

I've also seen recommendations for setting the maximum possible MTU (jumbo
frame) [1, 3], but my NIC (a Broadcom NetXtreme BCM5764M) does not support more
than 1500B. Another recommendation is to enable link-layer flow control at the
NIC [1] (and gateway switch), but I couldn't capture any pause frame packet
sent in my setting (a misconfiguration, perhaps?).

Decreasing DeviceLinkThroughputLimit (and turning DeviceLinkThroughputLimitMode
on) showed to be helpful in some scenarios [2], but it actually restricts
transfer rate, making the maximum possible frame rate below the camera's
specification. Hence, its traffic becomes less prone to failures.

After replicating this configuration (as close as I could) in our deployment
machine, the failure rate raised to about 2.8% (of 940k continuously acquired
images) with a lower rate of 6.4Hz (with throughput limit set up to its
minimum). The machine has another NIC (an Intel 82583V), thus another driver
and configuration options. We also use another switch model (possibly with
other settings).

Is there anything else you do to tune GNU/Linux machines for supporting these
GigE cameras?

My intention here is twofold: increase our knowledge of configuration options
and document this for future reference.

Best regards,
Henrique F. Simões

[1]: Tips and tricks to connect 1000BASE-T in Alvium G1 User Guide. Available
at
<https://cdn.alliedvision.com/fileadmin/content/documents/products/cameras/Alvium_G1/techman/Alvium-G1_User-Guide.pdf>
[2]: https://github.com/AravisProject/aravis/issues/314#issuecomment-559459813
[3]: https://github.com/AravisProject/aravis/issues/434#issuecomment-718418648
[4]: https://www.kernel.org/doc/html/latest/networking/packet_mmap.html


Aviso Legal: Esta mensagem e seus anexos podem conter informações confidenciais e/ou de uso restrito. Observe atentamente seu conteúdo e considere eventual consulta ao remetente antes de copiá-la, divulgá-la ou distribuí-la. Se você recebeu esta mensagem por engano, por favor avise o remetente e apague-a imediatamente.

Disclaimer: This email and its attachments may contain confidential and/or privileged information. Observe its content carefully and consider possible querying to the sender before copying, disclosing or distributing it. If you have received this email by mistake, please notify the sender and delete it immediately.