Re: Handling Polling and CPU Bottleneck for Large Image Capture in areaDetector

["Kim, Kuktae via Tech-talk" <tech-talk at aps.anl.gov>]

Hi Mark,

Thank you for your response and advice.

You should be setting the SCAN field of GC_SensorTemperature_RBV and GC_DeviceTemperature_RBV.  Those are the input records.

Here are some snippets of the EPICS records for SensorTemperature PVs:

record(ai, "$(P)$(R)GC_SensorTemperature_RBV") {

  field(DTYP, "asynFloat64")

  field(INP,  "@asyn($(PORT),$(ADDR=0),$(TIMEOUT=1))GC_D_SensorTemperature")

  field(PREC, "3")

  field(SCAN, "I/O Intr")

  field(DISA, "0")

}

record(ao, "$(P)$(R)GC_SensorTemperature") {

  field(DTYP, "asynFloat64")

  field(OUT,  "@asyn($(PORT),$(ADDR=0),$(TIMEOUT=1))GC_D_SensorTemperature")

  field(PREC, "3")

  field(DISA, "0")

}

# Added EGU and SCAN fields for GC_SensorTemperature_RBV

record(ai, "$(P)$(R)GC_SensorTemperature_RBV") {

  field(EGU, "C")

  field(SCAN, "1 second")

}

I can confirm the SCAN field by:

$ caget RIX:SVLS:CAM:01:GC_SensorTemperature_RBV.SCAN

RIX:SVLS:CAM:01:GC_SensorTemperature_RBV.SCAN 1 second

However, the value does not update every second.
I suspect this might be because the GC_SensorTemperature PV is set to SCAN Passive and GC_D_SensorTemperature is not updated:

$ caget RIX:SVLS:CAM:01:GC_SensorTemperature.SCAN

RIX:SVLS:CAM:01:GC_SensorTemperature.SCAN Passive

It seems that the GC_SensorTemperature_RBV record is pulling data every second, but the driver itself does not appear to refresh the value automatically.

You should first disable all the plugins and see how fast you can acquire with no plugins running.  Or you can simply disable ArrayCallbacks in the driver.

Then you should determine which plugins really need to run at the full frame rate, and which can run at reduced rate.  For the ones that are OK to run at 5 Hz, for example, you can set the MinCallbackTime to 0.2 seconds.

For plugins that need to run at the full rate you can run most of them with multiple threads.  In the startup script you set MaxThreads and at runtime you can set NumThreads.  If NumThreads > 1 then you need to think about the fact the the NDArrays can be processed out of order.  You can then enable sorting if that is an issue.  This is all documented here: https://areadetector.github.io/areaDetector/ADCore/NDPluginDriver.html

The release notes for ADCore R3-0 lists the plugins that support multiple threads and those that do not. https://github.com/areaDetector/ADCore/blob/master/RELEASE.md

Thank you for your advice. I will follow your recommendations and study the references you shared.

What is the camera model?

Can you send screenshots of the OPI screen for the detector and for the ROI1 and CC1 plugins?

The camera model is the AXIS-SXRF-60 from Axis Photonique, which uses the same sensor as the Dhyana 6060BSI from Tucsen. https://www.axis-photon.com/streak-camera/axis-sxrf-60-very-large-area-soft-x-ray-scmos-camera-on-a-flange/

We are also using the Euresys Coaxlink Quad CXP-12 frame grabber.

I will attached a few OPI screens in a separate email due to the email body size limitations.

Best regards,

Kuktae

---

From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Monday, April 28, 2025 9:37 AM
To: Kim, Kuktae <ktkim at slac.stanford.edu>; tech-talk <tech-talk at aps.anl.gov>
Subject: RE: Handling Polling and CPU Bottleneck for Large Image Capture in areaDetector

 

BEWARE: This email originated outside of our organization. DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe.

Hi Kuktae,

 

• We set up Polling features (ReadStatus.SCAN) so users can monitor sensor and device temperatures for interlocks. However, polling seems to block other tasks and lowers the image capture rate from ~12 Hz to ~7 Hz.

The asyn port is locked during the poll cycle, so that will block the read thread and slow down acquisition.  ReadStatus.SCAN reads all the GenICam features which you probably don’t need to do.

• We added the SCAN field of GC_SensorTemperature and GC_DeviceTemperature records as SCAN = "1 second", but now get warnings like "GC_SensorTemperature is not writable" in the IOC shell.
  Could you advise us on how to properly handle this situation?

You should be setting the SCAN field of GC_SensorTemperature_RBV and GC_DeviceTemperature_RBV.  Those are the input records.

 

• According to the manual, the camera can capture ~11 Hz (standard mode) and ~26 Hz (high-speed mode). We currently reach up to ~13 Hz, but plugin threads are hitting 99.9% CPU usage.
  Is there a way to use multithreading for plugins to reduce CPU load and increase the capture rate?

 

You should first disable all the plugins and see how fast you can acquire with no plugins running.  Or you can simply disable ArrayCallbacks in the driver.

 

Then you should determine which plugins really need to run at the full frame rate, and which can run at reduced rate.  For the ones that are OK to run at 5 Hz, for example, you can set the MinCallbackTime to 0.2 seconds.

 

For plugins that need to run at the full rate you can run most of them with multiple threads.  In the startup script you set MaxThreads and at runtime you can set NumThreads.  If NumThreads > 1 then you need to think about the fact the the NDArrays can be processed out of order.  You can then enable sorting if that is an issue.  This is all documented here:

https://areadetector.github.io/areaDetector/ADCore/NDPluginDriver.html

 

The release notes for ADCore R3-0 lists the plugins that support multiple threads and those that do not.

https://github.com/areaDetector/ADCore/blob/master/RELEASE.md

 

What is the camera model?

 

Can you send screenshots of the OPI screen for the detector and for the ROI1 and CC1 plugins?

 

Mark

 

 

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Kim, Kuktae via Tech-talk
Sent: Monday, April 28, 2025 10:00 AM
To: tech-talk <tech-talk at aps.anl.gov>
Subject: Handling Polling and CPU Bottleneck for Large Image Capture in areaDetector

 

Hello,

 

We are developing an areaDetector IOC for a large sensor camera (6144 x 6144, 16-bit depth, ~75 MB per image) and would like some advice on a few issues:

 

1. Polling Impact:
   We set up Polling features (ReadStatus.SCAN) so users can monitor sensor and device temperatures for interlocks. However, polling seems to block other tasks and lowers the image capture rate from ~12 Hz to ~7 Hz.
   We added the SCAN field of GC_SensorTemperature and GC_DeviceTemperature records as SCAN = "1 second", but now get warnings like "GC_SensorTemperature is not writable" in the IOC shell.
   Could you advise us on how to properly handle this situation?

2. CPU Bottleneck:
   According to the manual, the camera can capture ~11 Hz (standard mode) and ~26 Hz (high-speed mode). We currently reach up to ~13 Hz, but plugin threads are hitting 99.9% CPU usage.
   Is there a way to use multithreading for plugins to reduce CPU load and increase the capture rate?

 

Photos are attached for reference:

 

< CPU Bottleneck: Some plugin threads reach up to 99.9% CPU usage, limiting the capture rate >

 

< Polling Impact: Polling causes overall CPU usage to decrease and lowers image capture performance >

Thank you for your help!

 

Best regards,

Kuktae

 

 

3.