Hi Katie,
Ø
This camera model should be capable of 8 kHz, but instead I get maybe half that frame rate with both the Basler Pylon software as well as the IOC.
Ø
We plan to update the network card to handle 10 Gb and I will let you know if that fixes this problem, since now this seems like a bandwidth issue.
You said the camera was 12288 x 1 pixels, running in 8-bit mode. At 8 kHz that would be 12288*8000/1024/1024 = 93.75 MB/s. That is close to the limit of GigE, but many GigE cameras can use that much bandwidth OK. If you are only getting
half of that then I would check for CPU utilization. You can do that for the IOC with the “top –H” command. That will show you the CPU utilization of every thread in the areaDetector IOC. If any thread is close to 100% then that could be limiting you.
I would be surprised if using a 10 Gbit card would help, because the network interface on the camera is only 1 Gbit, right?
Ø
My only other question comes from grabbing my BASLER1:image1:ArrayData PV;
Ø
I have my EPICS_CA_MAX_ARRAY_BYTES set to 150,000,000 (very large for testing) on both the host as well as inside the IOC (verified with epicsEnvShow).
Ø
When I caget BASLER1:image1:ArrayData, it outputs the following error,
Ø
"BASLER1:image1:ArrayData *** CA error The requested data transfer is greater than available memory or EPICS_CA_MAX_ARRAY_BYTES".
Ø
I can return the array using caget up to ~8100 elements, after which I receive the above error.
When you say “8100 elements” how big is each element, i.e. what is the FTVL of your waveform record?
First, I would decrease EPICS_CA_MAX_ARRAY_BYTES to a value like 20,000,000. My understanding is that the IOC and the client will actually allocate arrays of EPICS_CA_MAX_ARRAY_BYTES if an array larger than the default size is needed,
so it will be using a lot of memory unnecessarily.
Please send the output of this command in the IOC
epicsPrtEnvParams
and this command at the shell running caget
echo $EPICS_CA_MAX_ARRAY_BYTES
Something is wrong if you are getting that message.
Mark
From: Katie Matusik <kmatusik at sigray.com>
Sent: Thursday, April 8, 2021 5:24 PM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: EPICS Tech-Talk <tech-talk at aps.anl.gov>
Subject: Re: line scan camera with ADAravis driver skips frames
Hi Mark et al.,
I updated to the latest Aravis release and master ADAravis driver. I also changed my net.core.rmem_default and net.core.rmem_max to 8 MB each. The biggest difference I saw was getting rid of the virtualization layer and installing EPICS
locally. I'm not sure why Docker has such an impact on the network bandwidth considering that I am using the host network, but removing it now allows me to get to higher frame rates without missing frames. Now I also have the camera directly plugged into an
ethernet card that should be able to handle 1000 Mb/s on each port, and I configure the port such that it should only see the traffic from this one Basler camera. This camera model should be capable of 8 kHz, but instead I get maybe half that frame rate with
both the Basler Pylon software as well as the IOC. We plan to update the network card to handle 10 Gb and I will let you know if that fixes this problem, since now this seems like a bandwidth issue. My only other question comes from grabbing my BASLER1:image1:ArrayData
PV; I have my EPICS_CA_MAX_ARRAY_BYTES set to 150,000,000 (very large for testing) on both the host as well as inside the IOC (verified with epicsEnvShow). When I caget BASLER1:image1:ArrayData, it outputs the following error, "BASLER1:image1:ArrayData ***
CA error The requested data transfer is greater than available memory or EPICS_CA_MAX_ARRAY_BYTES". I can return the array using caget up to ~8100 elements, after which I receive the above error. Using pyepics, I can output the full array. Thanks so much
in advance and please let me know what further info you need.
Hi Katie,
Ø
I am using the latest areaDetector-master with ADAravis R1-3 and ADGenICam-master.
The fact that you are using ADAravis R1-3 means you are using an old aravis 0_7 release. It could be that the problem you are seeing has already been fixed in aravis.
I suggest updating to the latest aravis 0_8 release and ADAravis R2-2 or master.
Mark
From: Tech-talk <tech-talk-bounces at aps.anl.gov>
On Behalf Of Katie Matusik via Tech-talk
Sent: Tuesday, April 6, 2021 1:07 PM
To: tech-talk <tech-talk at aps.anl.gov>
Subject: line scan camera with ADAravis driver skips frames
Hi everyone,
The camera that I am connecting to via GigE is a Basler Racer line scan camera (12288 x 1 px) model raL12288-8gm. When I run the camera with binning 1x1 and 8-bit pixel format at
1kHz (slowest frame rate), the driver keeps up until about 30K frames, at which point I start accumulating frame failures and underruns. With a higher frame rate, it happens almost immediately.
I am using the latest areaDetector-master with ADAravis R1-3 and ADGenICam-master. I am running this IOC in a Docker using the host network which has MTU set to 9000. I made a slight
adjustment to the ADAravis.cpp driver regarding setting the packet size to jumbo packets rather than determining optimum packet size, which was interfering with my network (lines 379-381 in ADAravis.cpp):
377 if (ARV_IS_GV_DEVICE(this->device)) {
378 // Automatically determine optimum packet size
379 // arv_gv_device_auto_packet_size(ARV_GV_DEVICE(this->device));
380 // Uncomment this line to set jumbo packets
381 arv_gv_device_set_packet_size(ARV_GV_DEVICE(this->device), 9000);
382 }
I'm attaching snapshots of my camera features settings as well as my st.cmd and st.cmd.base files. Please let me know what other information you need. When I run this using the
Basler pylonViewer software, I do not see these skipped frames.
This email and any files transmitted with it are confidential and intended solely for the use of the
individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you
should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing,
copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited
--
Katie Matusik
5750 Imhoff Drive, Suite I
Office: 1 (925) 326 - 7926
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have
received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please
notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the
contents of this information is strictly prohibited