Re: Understanding ASYN_CANBLOCK

[Marco Filho via Tech-talk <tech-talk at aps.anl.gov>]

Hi, Mark and thanks for the help

Thanks for the documentation suggestion, I think it makes a bit more of sense for me now. So ASYN_CANBLOCK allows the queueRequest to place the request on the work queue, if ASYN_CANBLOCK is not set, then the port is locked and we have to wait...

"Can you describe the driver and the number of I/O operations per second in more detail?"

So, we have two test setups:
one with about 1.796 parameters being read and not freezing the IOC and;
one with about 26.900 and freezing the IOC no matter if they are being scanned each 1 or 5 seconds.

Unfortunately, the second test setup is not available today and will be on monday, when I plan to test and see the CPU usage. The first one is using 1.3% of the CPU power and about 1 Mbps from the network interface.

Each parameter read operation sends 8 bytes and receives 16, so if I'm correct: RX bandwidth = 26.900*16*8 ~=3.4 Mbps.

However, I found something important: for some reason, I can only read about 14.285 times per second from the hardware even though we have more bandwidth than that. Code profiling show that it's the actual socket readfrom and sendto functions that take more time, so it's not the code that is not optimized (at least not to justify this order of magnitude in the reading numbers).

So I might recreate a silly example to see if that mimics the behavior: an asynPortDriver that has several "sleep(0.1)" inside it's readInt32, WriteInt32, etc. functions and some records being scanned quicker than that.

The expected behavior is that it freezes the IOC shell or that it should simply timeout for several record scan procedures with the ioc shell and other functionalities remaining unaffected? I will try and report the results.

About the driver: it's a simple asynPortDriver with a bunch of parameters. Some code fragments:

RMM::RMM(const char *portName, const char *addressIP, int addressPort, const char *configFile)
  : asynPortDriver(portName, 1,
      asynInt8ArrayMask | asynInt32Mask | asynInt32ArrayMask | asynInt64Mask | asynFloat64Mask | asynOctetMask | asynDrvUserMask,          // Interfaces that we implement
      asynInt8ArrayMask | asynInt32Mask | asynInt32ArrayMask | asynInt64Mask | asynFloat64Mask | asynOctetMask,       // Interfaces that do callbacks
      ASYN_MULTIDEVICE | ASYN_CANBLOCK, 1, /* ASYN_CANBLOCK=1, ASYN_MULTIDEVICE=1, autoConnect=1 */
      0, 0),  /* Default priority and stack size */
      rmmAPI(addressIP, addressPort, configFile)
{
  createEPICSParams(); //Creates the parameters with createParam()...
  updateRMMInfos(); // Updates some important parameters
  fetchRingStatus();
}

(...)

asynStatus RMM::writeInt32(asynUser *pasynUser, epicsInt32 value)
{
  asynStatus  status = asynSuccess;
  const char *functionName = "writeInt32";
  const char *paramName;
  int addr;

  int function = pasynUser->reason;
  getAddress(pasynUser, &addr);
  getParamName(addr, function, &paramName);

  if ((value) && (function == rmmResetPacketCounts_)) {
    rmmAPI.resetPacketCounters();
  }

(...)

Etc.

---

From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: 30 January 2025 23:23:51
To: tech-talk at aps.anl.gov; Marco Filho
Subject: Re: Understanding ASYN_CANBLOCK

 

Hi Marco,

If you have not already done so you should read the documentation here:

https://epics-modules.github.io/asyn/asynDriver.html#overview-of-queuing

You should always use ASYN_CANBLOCK for any driver which takes more than a few 10s of microseconds to complete an I/O request from device support.  If you do not use ASYN_CANBLOCK then the thread that calls you driver will be blocked until the I/O operation is complete.  This includes the scan tasks (1 second, 2 second, etc.), channel access server threads, etc.

If you are seeing freezing when you use ASYN_CANBLOCK then either you are starved for CPU (what does "top" show?) or something is wrong.  EPICS areaDetector drivers and plugins all use ASYN_CANBLOCK and they do thousands of callbacks to device support per second with no problem with performance.

Can you describe the driver and the number of I/O operations per second in more detail?

One way to reduce CPU is to write a thread in your driver that reads all of the parameters.  The records associated with those parameters have SCAN=I/O Intr.  When your driver does callParamCallbacks() only the parameters that have changed will have their records process.  If most parameters only change infrequently that can drastically reduce the amount of record processing.

Mark

---

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Marco Filho via Tech-talk <tech-talk at aps.anl.gov>
Sent: Thursday, January 30, 2025 4:02 PM
To: tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Understanding ASYN_CANBLOCK

 

Hi all. I'm writing this e-mail to ask for help understanding ASYN_CANBLOCK.

I have an asyn driver that communicates via a thread-safe API with the hardware. It tries to read thousands of parameters and I noticed that the IOC was freezing periodically as I added more parameters to be scanned.

I also noticed that reducing the SCAN time from 1 second to 5 seems to make the IOC shell freeze every couple of seconds or so (instead of freezing constantly), and guessed - just a guess - that this could be due to asyn trying to scan all parameters at once, no matter the period of 5 seconds between each reading.

Searching for a way to distribute the readings in time and reading the documentation, I found ASYN_CANBLOCK, which seems to create a separate thread for managing the queue requests:

"If queueRequest is called for a port that can block the request is queued to a thread dedicated to the port."

I expected then that putting the ASYN_CANBLOCK flag in the driver constructor would solve my problem, but to my surprise the behavior was the opposite: I found out that the constructor was already using this flag:

asynPortDriver(portName, 1,
      asynInt8ArrayMask | asynInt32Mask | asynInt32ArrayMask | asynInt64Mask | asynFloat64Mask | asynOctetMask | asynDrvUserMask,          // Interfaces that we implement
      asynInt8ArrayMask | asynInt32Mask | asynInt32ArrayMask | asynInt64Mask | asynFloat64Mask | asynOctetMask,       // Interfaces that do callbacks
      ASYN_MULTIDEVICE | ASYN_CANBLOCK, 1, /* ASYN_CANBLOCK=1, ASYN_MULTIDEVICE=1, autoConnect=1 */
      0, 0),  /* Default priority and stack size */

And removing the flag caused the OPI and shell to not freeze. By visually and manually inspecting I also have the feeling that the parameters are being properly scanned and that I'm not leaving anything behind.

The question is: by removing ASYN_CANBLOCK am I actually distributing the readings in time or am I doing something else? Are the requests actually being executed in parallel?

Thanks for any elucidations,

Marco