RE: line scan camera with ADAravis driver skips frames

[Mark Rivers via Tech-talk <tech-talk at aps.anl.gov>]

Hi Katie,

 

Ø  This camera model should be capable of 8 kHz, but instead I get maybe half that frame rate with both the Basler Pylon software as well as the IOC.

Ø  We plan to update the network card to handle 10 Gb and I will let you know if that fixes this problem, since now this seems like a bandwidth issue.

 

You said the camera was 12288 x 1 pixels, running in 8-bit mode.  At 8 kHz that would be  12288*8000/1024/1024 = 93.75 MB/s.  That is close to the limit of GigE, but many GigE cameras can use that much bandwidth OK.  If you are only getting half of that then I would check for CPU utilization.  You can do that for the IOC with the “top –H” command.  That will show you the CPU utilization of every thread in the areaDetector IOC.  If any thread is close to 100% then that could be limiting you.  I would be surprised if using a 10 Gbit card would help, because the network interface on the camera is only 1 Gbit, right?

 

Ø  My only other question comes from grabbing my BASLER1:image1:ArrayData PV;

Ø  I have my EPICS_CA_MAX_ARRAY_BYTES set to 150,000,000 (very large for testing) on both the host as well as inside the IOC (verified with epicsEnvShow).

Ø  When I caget BASLER1:image1:ArrayData, it outputs the following error,

Ø  "BASLER1:image1:ArrayData *** CA error The requested data transfer is greater than available memory or EPICS_CA_MAX_ARRAY_BYTES".

Ø  I can return the array using caget up to  ~8100 elements, after which I receive the above error.

 

When you say “8100 elements” how big is each element, i.e. what is the FTVL of your waveform record?

 

First, I would decrease EPICS_CA_MAX_ARRAY_BYTES to a value like 20,000,000.  My understanding is that the IOC and the client will actually allocate arrays of EPICS_CA_MAX_ARRAY_BYTES if an array larger than the default size is needed, so it will be using a lot of memory unnecessarily. 

 

Please send the output of this command in the IOC

epicsPrtEnvParams

 

and this command at the shell running caget

echo $EPICS_CA_MAX_ARRAY_BYTES

 

Something is wrong if you are getting that message.

 

Mark

 

 

 

 

From: Katie Matusik <kmatusik at sigray.com>
Sent: Thursday, April 8, 2021 5:24 PM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: EPICS Tech-Talk <tech-talk at aps.anl.gov>
Subject: Re: line scan camera with ADAravis driver skips frames

 

Hi Mark et al.,

I updated to the latest Aravis release and master ADAravis driver. I also changed my net.core.rmem_default and net.core.rmem_max  to 8 MB each. The biggest difference I saw was getting rid of the virtualization layer and installing EPICS locally. I'm not sure why Docker has such an impact on the network bandwidth considering that I am using the host network, but removing it now allows me to get to higher frame rates without missing frames. Now I also have the camera directly plugged into an ethernet card that should be able to handle 1000 Mb/s on each port, and I configure the port such that it should only see the traffic from this one Basler camera. This camera model should be capable of 8 kHz, but instead I get maybe half that frame rate with both the Basler Pylon software as well as the IOC. We plan to update the network card to handle 10 Gb and I will let you know if that fixes this problem, since now this seems like a bandwidth issue. My only other question comes from grabbing my BASLER1:image1:ArrayData PV; I have my EPICS_CA_MAX_ARRAY_BYTES set to 150,000,000 (very large for testing) on both the host as well as inside the IOC (verified with epicsEnvShow). When I caget BASLER1:image1:ArrayData, it outputs the following error, "BASLER1:image1:ArrayData *** CA error The requested data transfer is greater than available memory or EPICS_CA_MAX_ARRAY_BYTES". I can return the array using caget up to  ~8100 elements, after which I receive the above error. Using pyepics, I can output the full array. Thanks so much in advance and please let me know what further info you need.

 

Best,
Katie 

 

On Tue, Apr 6, 2021 at 11:46 AM Mark Rivers <rivers at cars.uchicago.edu> wrote:

Hi Katie,

 

Ø  I am using the latest areaDetector-master with ADAravis R1-3 and ADGenICam-master.

 

The fact that you are using ADAravis R1-3 means you are using an old aravis 0_7 release.  It could be that the problem you are seeing has already been fixed in aravis.

 

I suggest updating to the latest aravis 0_8 release and ADAravis R2-2 or master.

 

Mark

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Katie Matusik via Tech-talk
Sent: Tuesday, April 6, 2021 1:07 PM
To: tech-talk <tech-talk at aps.anl.gov>
Subject: line scan camera with ADAravis driver skips frames

 

Hi everyone,

The camera that I am connecting to via GigE is a Basler Racer line scan camera (12288 x 1 px) model raL12288-8gm. When I run the camera with binning 1x1 and 8-bit pixel format at 1kHz (slowest frame rate), the driver keeps up until about 30K frames, at which point I start accumulating frame failures and underruns. With a higher frame rate, it happens almost immediately. 

 

I am using the latest areaDetector-master with ADAravis R1-3 and ADGenICam-master. I am running this IOC in a Docker using the host network which has MTU set to 9000. I made a slight adjustment to the ADAravis.cpp driver regarding setting the packet size to jumbo packets rather than determining optimum packet size, which was interfering with my network (lines 379-381 in ADAravis.cpp):

377     if (ARV_IS_GV_DEVICE(this->device)) {
378         // Automatically determine optimum packet size
379 //        arv_gv_device_auto_packet_size(ARV_GV_DEVICE(this->device));
380         // Uncomment this line to set jumbo packets
381         arv_gv_device_set_packet_size(ARV_GV_DEVICE(this->device), 9000);
382     }

I'm attaching snapshots of my camera features settings as well as my st.cmd and st.cmd.base files. Please let me know what other information you need. When I run this using the Basler pylonViewer software, I do not see these skipped frames.

 

Best,

Katie

 

 

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited

 

--

Katie Matusik

Systems Engineer

Sigray, Inc.

5750 Imhoff Drive, Suite I

Concord, CA 94520 USA

Office: 1 (925) 326 - 7926

 

 

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited