Experimental Physics and
Industrial Control System

Matt Newville <[email protected]> · Fri, 23 Dec 2011 08:44:26 -0600

Hi Anton,

On Fri, Dec 23, 2011 at 3:38 AM, Anton Derbenev <[email protected]> wrote:
> Hello all,
>
>
> recently I have tried to make a Python application using Epics Channel Access for Python.
>
> I am using an EPICS areaDetector to operate a Prosilica digital camera, from which I need to rapidly obtain several images.
>
> Getting a single PV waveform data snapshot itself is not an issue and works just fine the way it is described in the pyepics documentation.
>
> Although a call to PV.get(...) for 1280x960x12bit data array blocks the program execution for too long (relatively to camera FPS), I've managed to get rid of any delay from my software by utilizing a multiprocessing Python interface to avoid Global Interpreter Lock.
>
> Thus, my application is capable of running several data handling functions simultaneously without blocking.
>
> That's where the bad part starts: when the PV that holds the image data is set to be automatically monitored (auto_monitor = True), the callback invocation happens too slow.
>
> For example, when the camera is triggered and makes, like, 5 shots in one second, the callback for corresponding PV only runs three times!
> :(
>
> I tried both "high-level" (PV object) and "low-level" (channel access
> contexts) approaches trying to make the callback calling to be faster, but no
> luck.

Sorry for the trouble.  There have been several conversations over the
past few months about getting waveform records with pyepics.  Some
changes were made fairly recently (to version 3.1.4), and there are
some changes in a branch on github (though I think it won't help your
case).  We can make more changes if they're needed, but at this point,
I don't know what those might be.  It would be nice to understand what
you're doing and why it's not working.

For reference, a waveform PV with count larger than
ca.AUTOMONITOR_MAXLENGTH (default
16384) is *not* automatically monitored, unless you use
   PV(name, ..., auto_monitor=True)

I read your message to mean that you did use 'auto_monitor=True'.  If
that's the case, the data will be sent to the PVs internal callback by
the CA library.   A PV.get() should simply return the value received
from the most recent monitor callback.

If auto_monitor is not set for a large waveform, then a PV.get() (or
ca.get() with the Channel ID) will call the C ca_array_get_callback()
function, and then wait for the callback to complete.

In either case, the callback runs in a separate thread, asynchronous
(assuming pre-emptive callbacks) from the main Python thread and
should not be slowed down by anything to do with Python (for example,
the GIL).  The data sent from the CA callback or
ca_array_get_callback() does have to be converted from the C structure
to a Python list or numpy array, but it's hard to believe that cannot
keep up with 5 frames per second for a 1Mb image.  A few diagnostic
tests:

  a) verify with command-line camonitor that you're getting the
expected frame rate over your network.
  b) write a test callback function for the PV object that simply
records or prints when it gets a new value for the waveform.

Those should help isolate the problem. If the problem is that your
Python callback is missing some events,  you might try profiling the
conversion of data from C to Python (ca._unpack(), especially the
conversion to numpy arrays).  I'd be surprised that was the culprit,
but it's easier to believe than most other options.

Again, I doubt that Python's GIL has anything to do with this, as most
of the processing happens in the CA library.  I am also not sure that
multiprocessing would actually help. By default, pyepics tries to
ensure that a single CA thread context is used.  Depending on how you
use it, multiprocessing might increase the network traffic.

> I assume that every time when the automatically monitored PV changes, its
> status is requested ( like in PV.get(...) ) and then passed as argument to the
> callback function.

Actually, a monitored PV is registered with the CA library, and a
callback function is run when the PV changes.  This happens from the C
library, in a thread separate from and asynchronous to the main python
thread.

> So if a CA (channel access) does not allow rapid subsequent queries to a
> single PV, hence the delay that limits the amount of callbacks that can be
> called per second, this amount being less then my camera's FPS.
>
> If this is so, then setting the auto_monitor to False and simply polling the
> PV with desired frequency will not work either (also this approach causes large
> amounts of unnecessary network traffic).

Using 'auto_monitor=False' will result in the least amount of network
traffic.   You would need to do an
explicit PV.get() (which calls ca_array_get_callback()) to get a new
value for a PV.  If you don't ask for data fast enough, you will miss
some data.

> Thanks to the mighty areaDetector software, I have found the workaround
> (omitting EPICS at all), but it involves the file system of the hardware where
> the IOC runs, thus being not completely satisfactory.

Perhaps I'm not understanding -- how can you be using areaDetector and
omitting EPICS at all?  Anyway, if the data can be written to disk at
a sufficient speed, any program ought to be able to keep up...  Are
you running the pyepics program on the same machine as the IOC or a
different machine?

> Hereby I require aid: what are the strategies for working with fast-changing
> large data arrays using pyepics?  How can I make my camera frame data PV
> object to invoke callbacks fast enough or, if it is not possible, what are
> other ways to save and later access the value of such PVs using EPICS?
>
> Best regards,
> Derbenev Anton.

Hope that helps.

--Matt

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System