Experimental Physics and
Industrial Control System

Mark Rivers via Tech-talk <[email protected]> · Thu, 30 May 2019 17:31:39 +0000

Hi Klemn,

I think it would be difficult to implement queuing threads in asynPortDriver.  The queuing is actually done at a higher level in asynManager.  If a port has ASYN_CANBLOCK then each time a record processes it queues a request with asynManager, and the thread that asynManager created for that port pulls requests off that queue and processes them, calling the device support callback function.  That callback function is what calls the asynPortDriver.

For your application I understand the reluctance to have 500 asynchronous asyn ports, because each will have its own thread.

It doesn't seem like the input records are a problem.  You are sending a trigger message to the driver, which reads all the status and does callbacks to I/O Intr scanned records.  The driver can return immediately when it receives this trigger message, it can just use one of the general purpose callbacks tasks to do the actual I/O and callbacks.  Thus, for input records the driver does not need to set ASYN_CANBLOCK.

For output records you could also not set ASYN_CANBLOCK.  When a write command is received it can use a general purpose callbacks task to do the actual I/O, and return immediately.  The downside is that the status returned by the write command will not reflect the actual I/O status.  However, you can have another status input record with SCAN=I/O Intr to reflect the actual status of the I/O operation when it does complete.

You could thus have 500 synchronous drivers which have no extra threads beyond general purpose callback theads. 

Alternatively you could have a single asyn port driver for all 500 detectors, and use the asyn ADDR field to specify which detector is being talked to.  In this case you could use ASYN_CANBLOCK because asynManager is only creating 1 thread, not 500 threads.

Mark

-----Original Message-----
From: Klemen Vodopivec <[email protected]> 
Sent: Thursday, May 30, 2019 11:49 AM
To: Mark Rivers <[email protected]>
Cc: [email protected]
Subject: Re: [EXTERNAL] RE: How to share asyn's queue thread among ports?

Hi Mark,

I was hoping you would respond. Thanks for looking into this.

On 5/30/19 12:23 PM, Mark Rivers wrote:
> Hi Klemen,
> 
> I have some ideas on this, but I need to have a little more information.
> 
> About how many detectors do you have?  You said 500 ports, so I assume that means 500 detectors?  You would have one asyn port per detector?

Yes, 1 detector per port, 500 is max number of detectors per one beamline so that would be worst case we have to support.

> 
> Would all ports be running in a single IOC, or multiple IOCs?

It's a single IOC, primarily because there's one dedicated fiber channel that connects all detectors to one PCIe board. We could try to split it up but it's been reliable and performant so far.

> 
> You have about 500 parameters per detector, i.e. per port?

That's right. In database there's potentially more connected records for each parameter. 500 detectors x (500 param PVs + 300 connected PVs) =
400,000 PVs per single IOC. Aside from slow IOC boot, everything else works just fine and performs great.

> 
> How many parameters are being written to, and at about what rate?

Writing parameters is rare. Maybe once a day we will change 2 
parameters. About half of parameters are changing in loop when we are 
calibrating, which lasts about 1 day and happens every few months. 
Performance is not an issue during calibration.

> 
> How many parameters are being read, and about what rate?

Half of parameters are read only for status and are I/O Intr. We have a 
trigger PV that updates them at once every 60 seconds.

> 
> How are your detectors communicating, i.e. TCP/IP or some other mechanism?

It's custom protocol over a custom made PCI express board. We have Linux 
kernel driver talking to PCIe, with asynPortDriver interface to Linux 
driver. There's 2.5Gbps fiber connecting from PCIe to detectors 
aggregator box.

One particularity about the protocol is that we need to exchange all 
configuration parameters at once (similar for status). So in the 
asynPortDriver derived class we pack all parameters into one packet and 
send it to detector. All writable parameters are cached in software and 
there's two additional PVs, one that triggers packing and sending, and 
second one is status of packing/sending operation. If number of changing 
parameters is high, there would be a lot of redundancy but luckily this 
is not the case. There's new detectors on horizon that will support 
exchanging single parameter at a time, which is what is driving this 
initiative.

> 
> Are you currently using the asyn "addr" parameter, or just the "port" to specify the detector and the drvUser field to specify the parameter?

Just the port. Due to the way we manage detector configuration this is 
the easiest. I thought about using addr but haven't tried it out yet.

> 
> Is it important that a write operation actually have completed before the record processing completes?

So far write is asynchronous operation and the client has to use another 
PV to wait for the status. It is desired, however, that writable PVs 
would be synchronous through put callback.

> 
> Would it be OK to have status be periodically polled and use SCAN=I/O Intr?

Yes, that's what we do.

-- Klemen

> 
> Mark
> 
> 
> -----Original Message-----
> From: [email protected] <[email protected]> On Behalf Of Klemen Vodopivec via Tech-talk
> Sent: Thursday, May 30, 2019 9:56 AM
> To: [email protected]
> Subject: How to share asyn's queue thread among ports?
> 
> Hi,
> 
> in a fast data-acquisiting application that also controls hundreds of neutron detectors we are using asynPortDriver interface for detector configuration and status communication which results in many asynPortDriver params. The communication is fast (less than 1ms for request/response round trip) but is probably still considered blocking.
> Especially because there's approx 500 params per port, many are written/read in groups and if every request blocks for 1ms it could block CA thread significantly. Our workaround so far was done in database but is overly complex. The obvious solution is to use asyn's blocking mode, but there's up to 500 ports resulting in as many threads.
> 
> Ideally there would be an asynPortDriver interface to manage queue threading, which we could then overload and do custom thread management.
> Is there any other approach to keep number of threads low while providing blocking mode? Would asyn patch that allows asynPortDriver do queue thread management be of broader interest?
> 
> -- Klemen
> 

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System