Re: line scan camera with ADAravis driver skips frames

[Mark Rivers via Tech-talk <tech-talk at aps.anl.gov>]

Hi Katie,


"top'" shows that medm and CAC-TCP-recv are using the most CPU.  I suspect that  CAC-TCP-recv is the channel access client receiving thread in medm.  Do you have a waveform plot that is updating at full frame rate in medm?  If so, I would suggest disabling or throttling the NDStdArrays (Image1) plugin and seeing if that increases the frame rate at which the camera data can be received.


The EPICS_CA_MAX_ARRAY_BYTES values look OK (20,000,000) on both the IOC and client.  Does the client still report an error when you do caget using that value?  If so, something is strange.


Mark



________________________________
From: Katie Matusik <kmatusik at sigray.com>
Sent: Thursday, April 8, 2021 7:34 PM
To: Mark Rivers
Cc: EPICS Tech-Talk
Subject: Re: line scan camera with ADAravis driver skips frames

Hi Mark,
I'm attaching the output of top -H when running max frame rate in 8-bit mode. In 8-bit, the max frame rate I can get is closer to ~6K, and with 12-bit, it is around ~4K. You're right that the camera is 1Gbit, so a 10Gbit card won't fix the problem.

I'm also attaching my st.cmd.base file; each element is 16-bit because I want to be able to run the camera in 12-bit mode, FTVL is set to short.
output of epics> epicsPrtEnvParams
EPICS_CA_ADDR_LIST is undefined
EPICS_CA_CONN_TMO: 30.0
EPICS_CA_AUTO_ADDR_LIST: YES
EPICS_CA_REPEATER_PORT: 5065
EPICS_CA_SERVER_PORT: 5064
EPICS_CA_MAX_ARRAY_BYTES: 20000000
EPICS_CA_MAX_SEARCH_PERIOD: 300.0
EPICS_CA_NAME_SERVERS is undefined
EPICS_CAS_INTF_ADDR_LIST is undefined
EPICS_CAS_IGNORE_ADDR_LIST is undefined
EPICS_CAS_AUTO_BEACON_ADDR_LIST is undefined
EPICS_CAS_BEACON_ADDR_LIST is undefined
EPICS_CAS_SERVER_PORT is undefined
EPICS_CA_BEACON_PERIOD: 15.0
EPICS_CAS_BEACON_PERIOD is undefined
EPICS_CAS_BEACON_PORT is undefined
EPICS_BUILD_COMPILER_CLASS: gcc
EPICS_BUILD_OS_CLASS: Linux
EPICS_BUILD_TARGET_ARCH: linux-x86_64
EPICS_TIMEZONE: CUS::360:031102:110402
EPICS_TS_NTP_INET is undefined
EPICS_IOC_LOG_PORT: 7004
EPICS_IOC_LOG_INET is undefined
EPICS_IOC_LOG_FILE_LIMIT: 1000000
EPICS_IOC_LOG_FILE_NAME is undefined
EPICS_IOC_LOG_FILE_COMMAND is undefined
EPICS_CMD_PROTO_PORT is undefined
EPICS_AR_PORT: 7002
IOCSH_PS1: epics>
IOCSH_HISTSIZE: 50

Output of of MAX_ARRAY_BYTES in terminal running caget:
echo $EPICS_CA_MAX_ARRAY_BYTES
20000000

Thanks!
Katie

On Thu, Apr 8, 2021 at 4:35 PM Mark Rivers <rivers at cars.uchicago.edu<mailto:rivers at cars.uchicago.edu>> wrote:
Hi Katie,


>  This camera model should be capable of 8 kHz, but instead I get maybe half that frame rate with both the Basler Pylon software as well as the IOC.

>  We plan to update the network card to handle 10 Gb and I will let you know if that fixes this problem, since now this seems like a bandwidth issue.

You said the camera was 12288 x 1 pixels, running in 8-bit mode.  At 8 kHz that would be  12288*8000/1024/1024 = 93.75 MB/s.  That is close to the limit of GigE, but many GigE cameras can use that much bandwidth OK.  If you are only getting half of that then I would check for CPU utilization.  You can do that for the IOC with the “top –H” command.  That will show you the CPU utilization of every thread in the areaDetector IOC.  If any thread is close to 100% then that could be limiting you.  I would be surprised if using a 10 Gbit card would help, because the network interface on the camera is only 1 Gbit, right?


>  My only other question comes from grabbing my BASLER1:image1:ArrayData PV;

>  I have my EPICS_CA_MAX_ARRAY_BYTES set to 150,000,000 (very large for testing) on both the host as well as inside the IOC (verified with epicsEnvShow).

>  When I caget BASLER1:image1:ArrayData, it outputs the following error,

>  "BASLER1:image1:ArrayData *** CA error The requested data transfer is greater than available memory or EPICS_CA_MAX_ARRAY_BYTES".

>  I can return the array using caget up to  ~8100 elements, after which I receive the above error.

When you say “8100 elements” how big is each element, i.e. what is the FTVL of your waveform record?

First, I would decrease EPICS_CA_MAX_ARRAY_BYTES to a value like 20,000,000.  My understanding is that the IOC and the client will actually allocate arrays of EPICS_CA_MAX_ARRAY_BYTES if an array larger than the default size is needed, so it will be using a lot of memory unnecessarily.

Please send the output of this command in the IOC
epicsPrtEnvParams

and this command at the shell running caget
echo $EPICS_CA_MAX_ARRAY_BYTES

Something is wrong if you are getting that message.

Mark




From: Katie Matusik <kmatusik at sigray.com<mailto:kmatusik at sigray.com>>
Sent: Thursday, April 8, 2021 5:24 PM
To: Mark Rivers <rivers at cars.uchicago.edu<mailto:rivers at cars.uchicago.edu>>
Cc: EPICS Tech-Talk <tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>>
Subject: Re: line scan camera with ADAravis driver skips frames

Hi Mark et al.,
I updated to the latest Aravis release and master ADAravis driver. I also changed my net.core.rmem_default and net.core.rmem_max  to 8 MB each. The biggest difference I saw was getting rid of the virtualization layer and installing EPICS locally. I'm not sure why Docker has such an impact on the network bandwidth considering that I am using the host network, but removing it now allows me to get to higher frame rates without missing frames. Now I also have the camera directly plugged into an ethernet card that should be able to handle 1000 Mb/s on each port, and I configure the port such that it should only see the traffic from this one Basler camera. This camera model should be capable of 8 kHz, but instead I get maybe half that frame rate with both the Basler Pylon software as well as the IOC. We plan to update the network card to handle 10 Gb and I will let you know if that fixes this problem, since now this seems like a bandwidth issue. My only other question comes from grabbing my BASLER1:image1:ArrayData PV; I have my EPICS_CA_MAX_ARRAY_BYTES set to 150,000,000 (very large for testing) on both the host as well as inside the IOC (verified with epicsEnvShow). When I caget BASLER1:image1:ArrayData, it outputs the following error, "BASLER1:image1:ArrayData *** CA error The requested data transfer is greater than available memory or EPICS_CA_MAX_ARRAY_BYTES". I can return the array using caget up to  ~8100 elements, after which I receive the above error. Using pyepics, I can output the full array. Thanks so much in advance and please let me know what further info you need.

Best,
Katie

On Tue, Apr 6, 2021 at 11:46 AM Mark Rivers <rivers at cars.uchicago.edu<mailto:rivers at cars.uchicago.edu>> wrote:
Hi Katie,


>  I am using the latest areaDetector-master with ADAravis R1-3 and ADGenICam-master.

The fact that you are using ADAravis R1-3 means you are using an old aravis 0_7 release.  It could be that the problem you are seeing has already been fixed in aravis.

I suggest updating to the latest aravis 0_8 release and ADAravis R2-2 or master.

Mark

From: Tech-talk <tech-talk-bounces at aps.anl.gov<mailto:tech-talk-bounces at aps.anl.gov>> On Behalf Of Katie Matusik via Tech-talk
Sent: Tuesday, April 6, 2021 1:07 PM
To: tech-talk <tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>>
Subject: line scan camera with ADAravis driver skips frames

Hi everyone,
The camera that I am connecting to via GigE is a Basler Racer line scan camera (12288 x 1 px) model raL12288-8gm. When I run the camera with binning 1x1 and 8-bit pixel format at 1kHz (slowest frame rate), the driver keeps up until about 30K frames, at which point I start accumulating frame failures and underruns. With a higher frame rate, it happens almost immediately.

I am using the latest areaDetector-master with ADAravis R1-3 and ADGenICam-master. I am running this IOC in a Docker using the host network which has MTU set to 9000. I made a slight adjustment to the ADAravis.cpp driver regarding setting the packet size to jumbo packets rather than determining optimum packet size, which was interfering with my network (lines 379-381 in ADAravis.cpp):
377     if (ARV_IS_GV_DEVICE(this->device)) {
378         // Automatically determine optimum packet size
379 //        arv_gv_device_auto_packet_size(ARV_GV_DEVICE(this->device));
380         // Uncomment this line to set jumbo packets
381         arv_gv_device_set_packet_size(ARV_GV_DEVICE(this->device), 9000);
382     }
I'm attaching snapshots of my camera features settings as well as my st.cmd and st.cmd.base files. Please let me know what other information you need. When I run this using the Basler pylonViewer software, I do not see these skipped frames.

Best,
Katie

[Image removed by sender.]

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited


--
Katie Matusik
Systems Engineer
Sigray, Inc.<http://www.sigray.com/>
5750 Imhoff Drive, Suite I
Concord, CA 94520 USA
Office: 1 (925) 326 - 7926

[Image removed by sender.]

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited


--
Katie Matusik
Systems Engineer
Sigray, Inc.<http://www.sigray.com/>
5750 Imhoff Drive, Suite I
Concord, CA 94520 USA
Office: 1 (925) 326 - 7926

[https://static1.squarespace.com/static/56de1f4a2fe1318cf0aacc98/t/5d9e0fa57996446a078a8e1a/1570639781104/Logo_gmail_small.png?format=300w]

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited