Re: Can't enable custom frame rates for a basler camera in area detector

[Mark Rivers via Tech-talk <tech-talk at aps.anl.gov>]

Hi Abdalla,


> It had a value of 100, now I could not figure out how to choose a value so I put an arbitrary value of 1000 and the acquisition is working good (230K frames so far).


I really don't think that changing that value is what caused the camera to start working without dropping frames.  Your screen shot of commonPlugins.adl shows that the number of free queue elements is 10000, which I assume is the size you set.  That means it had not had to queue any frames, so the queue size is irrelevant.  10000 is also a bad choice, because if the plugin were to slow down for some reason, then the queue would fill up and it would use 10000*1.25MB = 10.25 GB, which is more RAM than you have in your system.  You would bring the Linux machine to a crawl, because it would be swapping memory.


> How do I choose a value for the frame queue size?


For NDPluginStdArrays there is really no need to use a large queue, you don't care if it drops a few frames if it is just for the ImageJ viewer.  I would set it to 10.  Note that you don't need to change the startup script, you can change it at run-time, and it is in autosave.


Mark



________________________________
From: Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo>
Sent: Sunday, January 17, 2021 7:30 AM
To: Mark Rivers; tech-talk at aps.anl.gov
Subject: RE: Can't enable custom frame rates for a basler camera in area detector


Hello Mark



I think the problem is the frame queue size value passed to “NDStdArraysConfigure” in the IOC startup script. It had a value of 100, now I could not figure out how to choose a value so I put an arbitrary value of 1000 and the acquisition is working good (230K frames so far). How do I choose a value for the frame queue size?



Best Regards,

Abdalla.



From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Abdalla Ahmad via Tech-talk
Sent: Sunday, January 17, 2021 1:24 PM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: tech-talk at aps.anl.gov
Subject: RE: Can't enable custom frame rates for a basler camera in area detector



Hello Mark



Attached is the Genicam features GUIs. I noticed that I was loading the NDStdArrays plugin using “NDStdArraysConfigure” and a bunch of other templates from ADCore and NDDriverStdArrays, so I unloaded the plugin and the IOC seems to acquire images so far smoothly, I did a quick run today and It did like 350K frames and more at 32 fps with 11 frame error and few thousands frame underruns. Seems like an issue with the ND plugin.



Best Regards,

Abdalla.



From: Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo<mailto:Abdalla.Ahmad at sesame.org.jo>>
Sent: Saturday, January 16, 2021 7:56 PM
To: Mark Rivers <rivers at cars.uchicago.edu<mailto:rivers at cars.uchicago.edu>>
Cc: tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>
Subject: Re: Can't enable custom frame rates for a basler camera in area detector



Hello Mark



I discussed this briefly with the IT group and there should not be any problem. Regarding the network topology there are gigabit switches spread around the machine connected to a core switch. I don't think there is any kind of interference because cameras control is off as well as higher bandwidth stuff like BPMs fast data.



However, I tried the aravis viewer, from the same aravis version used to build the IOC, to connect to the camera on the same PC and it seems it's working fine. I just did a quick run and it did near 50K frames at 32 fps without any errors. Tomorrow I will run it for a longer period and see how it will perform.



Best Regards,

Abdalla.



________________________________

From: Mark Rivers <rivers at cars.uchicago.edu<mailto:rivers at cars.uchicago.edu>>
Sent: Wednesday, January 13, 2021 4:08 PM
To: Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo<mailto:Abdalla.Ahmad at sesame.org.jo>>
Cc: tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov> <tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>>
Subject: Re: Can't enable custom frame rates for a basler camera in area detector



Hi Abdalla,


Note that your camera does not really push the 1 Gbit Ethernet speed, at 32 frames/s it is only using 320 Mbits/s, about 1/3 of the GigE bandwidth:


 >>> 1296*966.*32*8
320495616.0

What is your network topology, i.e. how many switches between the IOC computer and the camera?  Could the camera traffic be competing with other large network traffic over a single GigE link somewhere?


Mark



________________________________
From: Mark Rivers
Sent: Wednesday, January 13, 2021 7:54 AM
To: Abdalla Ahmad
Cc: tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>
Subject: Re: Can't enable custom frame rates for a basler camera in area detector


Hi Abdalla,


Those performance problems can be caused by some default Linux buffer sizes not being large enough.  Did you increase them as described in the ADGenICam documentation?


https://areadetector.github.io/master/ADGenICam/ADGenICam.html#linux-usb-and-gige-system-settings


Mark



________________________________
From: Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo<mailto:Abdalla.Ahmad at sesame.org.jo>>
Sent: Wednesday, January 13, 2021 5:34 AM
To: Mark Rivers
Cc: tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>
Subject: RE: Can't enable custom frame rates for a basler camera in area detector

Hi Mark

I started to get occasional freezes, i.e. no acquisition on any framerate. When frozen, "Frame underruns" shows a huge number sometimes in millions, and sometimes "Frame failures" shows some value.

I noticed that I was using the master branch for all my support modules so I checked-out to the latest tag for everything. I tried lowering the framerate to like 10. Sometimes, not always, I get this message on the IOC shell

ADARavis::newBufferCallback bad frame status: Timeout

But yesterday I left the camera running on 32 FPS and this morning the IOC shell was filled with:

CRITICAL **: arv_gc_Feature_node_is_available: assertion 'ARV_IS_GC_FEATURE_NODE (gc_feature_node)' failed
Glib-GObject-Warning **: invalied unclassed pointer in cast to 'ArvGcFeatureNode'

Even pylon viewer software at first started with 32 FPS then after few minutes the framerate dropped to 15 (Could this be causing the frame underruns?). I am using the latest version of all modules along with aravis 0.8.5. I'm not sure whether the problem with the camera or the driver.

Best Regards,
Abdalla.

-----Original Message-----
From: Mark Rivers <rivers at cars.uchicago.edu<mailto:rivers at cars.uchicago.edu>>
Sent: Monday, January 11, 2021 3:55 PM
To: Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo<mailto:Abdalla.Ahmad at sesame.org.jo>>
Cc: tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>
Subject: Re: Can't enable custom frame rates for a basler camera in area detector

Hi Abdalla,


I see that you are running ADAravis R1-2, not the most recent version which is R2-2.  In R2-0 there was a bug fix for boolean features.  This is from the RELEASE.md file for R2-0.


  *   Fixed a problem with GenICam boolean features. Previously the code was calling the aravis functions for integer features, which did not work. It was changed to call the functions for boolean features, and it now works correctly.

AcquisitionFrameRateEnable is a boolean feature, so this is probably your problem.


You should update to ADAravis R2-2.  That means you will need to update your aravis installation from 0.7.2 to 0.8.1.  You should also use the most recent release of ADGenICam, R1-7.


Mark



________________________________
From: Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo<mailto:Abdalla.Ahmad at sesame.org.jo>>
Sent: Monday, January 11, 2021 1:44 AM
To: Mark Rivers
Cc: tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>
Subject: RE: Can't enable custom frame rates for a basler camera in area detector

Hello Mark

One messed up sentence, I meant "from EPICS" of course :)

The thing is, FrameRateEnable_RBV is always "NO" and I can't set it through its drop down menu. AcquireTime (Or the exposure time) is set to default value of 0.009, increasing it will give slower rate as expected (0.1 will give array rate of 10) but still can't set the frame rate. The camera normally is running with array rate of 32 but ImageJ is still seeing only 1 FPS.

I am using NDPluginStdArrays and callback time, array rate and byte rates are all set to 0. Attached are the required snapshots.

Thanks.
Abdalla.

-----Original Message-----
From: Mark Rivers <rivers at cars.uchicago.edu<mailto:rivers at cars.uchicago.edu>>
Sent: Sunday, January 10, 2021 4:13 PM
To: Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo<mailto:Abdalla.Ahmad at sesame.org.jo>>
Cc: tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>
Subject: Re: Can't enable custom frame rates for a basler camera in area detector

Hi Abdalla.


> We tried area detector previously to control Basler cameras from Pylon and it was working.


Can you explain what you mean by that sentence?  My understanding is that Pylon is a software suite from Basler that includes both an SDK and a viewer.  How was areaDetector being used "from Pylon"?


> We revisited the setup again but with one problem which is we can't set specific frame rate from the ADAravis.adl GUI, frame rate enable PV value is No and does not change.


This sounds like you are using ADAravis?  If the FrameRateEnable is No then the frame rate will be controlled by the AcquireTime and the maximum readout rate of the camera.  What AcquireTime are you using?  FrameRateEnable is designed to allow you to use a slower frame rate than the AcquireTime would generate, for example if AcquireTime=0.1 you could set FrameRateEnable=Yes and FrameRate=2 to only get 2 frames/s rather than 10 frames/s.


FrameRateEnable is mapped to one of the following GenICam feature names:

AcquisitionFrameRateEnable

AcquisitionFrameRateEnabled


Some cameras use one name, some use the other, and some cameras may provide this feature at all.


> The camera is running with maximum acquisition rate of 32 FPS but ImageJ viewer is seeing only 1 FPS


How are you determining what actual frame rate camera is running?  What is the ArrayRate_RBV PV on the main ADAravis.adl window?  If that is 32 then indeed ADAravis is receiving 32 frames/s from the camera.



> but If I go to NDPluginBaseFull.adl and set the max array rate it will give me the correct "frame rate".


Which plugin are you doing that for?  NDPluginStdArrays, NDPluginPva, etc.?  I suggest you set MinCallbackTime, MaxArrayRate and MaxByteRate all to 0 so that the plugin itself is not throttling the rate at all.


> Seems the camera is running at 32 FPS but only getting one of them each second. Pylon software is working perfectly with the camera, frame rate can be enabled and set with no problems.

There could be several issues that would cause the ADAravis to only get 1 frame/s.  It could be dropping frames for some reason, a plugin could be throttling the frames, etc.

Please send screen shots of the following when you believe the camera is acquiring at 32 frames/s.
- Your main ADAravis.adl screen
- The screen with the plugin you are using for the viewer, i.e. NDStdArrays.adl or NDPva.adl
- The camera-specific *-features_*.adl screen that contains the AcquisitionFrameRateEnable or  AcquisitionFrameRateEnabled PV

Mark


________________________________
From: Tech-talk <tech-talk-bounces at aps.anl.gov<mailto:tech-talk-bounces at aps.anl.gov>> on behalf of Abdalla Ahmad via Tech-talk <tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>>
Sent: Sunday, January 10, 2021 4:36 AM
To: tech-talk at aps.anl.gov<mailto:tech-talk at aps.anl.gov>
Subject: Can't enable custom frame rates for a basler camera in area detector


Hi



We tried area detector previously to control Basler cameras from Pylon and it was working. We revisited the setup again but with one problem which is we can't set specific frame rate from the ADAravis.adl GUI, frame rate enable PV value is No and does not change. The camera is running with maximum acquisition rate of 32 FPS but ImageJ viewer is seeing only 1 FPS, but If I go to NDPluginBaseFull.adl and set the max array rate it will give me the correct "frame rate". Seems the camera is running at 32 FPS but only getting one of them each second. Pylon software is working perfectly with the camera, frame rate can be enabled and set with no problems.



Best Regards,

Abdalla.