Re: Strategies for working with fast-changing large arrays

[Matt Newville <newville@cars.uchicago.edu>]

Hi Michael, Anton, All,

Thanks for the report.   FWIW, pyepics uses preemptive callbacks by
default, but I still see dropped frames, independent of the context
model.   After some further testing, I believe this is due to the
conversion of data from C to Python, which is not too surprising.  The
culprit is, as earlier in this conversation, with the '_unpack'
function which does the C-Python data conversion.

For CHAR waveforms, such as images from a Prosilica camera, the
current code has a test for converting to a CHAR waveform is (ca.py,
line 815)
        # waveform data:
        if ntype == dbr.CHAR:
            if use_numpy:
                data = numpy.array(data)
            return copy(data[:])

where 'data' is the still-ctypes data (an unsigned byte Array:
c_ubyte_Array).  Ignoring the 'use_numpy' option, the 'return
copy(data[:])' does two things:
   a) makes a copy to avoid memory overwrites -- I've seen this on some systems.
   b) takes a slice of the whole array 'data[:]' which converts the
ctypes ubyte Array to a list.  This appears to be the slow part.

Replacing that with
        # waveform data:
        if ntype == dbr.CHAR:
            if use_numpy:
                data = numpy.array(data)
            return copy(data)

allows color images from a 1.4Mb camera (so 4177920 data) to be
received at full speed.  The downside is that the data is still a
ctypes array and has to be converted *somewhere* to something useful
to python, though this is simply 'data=data[:]'.   Doing this inside
the user-level callback still causes frames to be dropped, as that is
still run "inside" the event handler.   It might be possible that
running the user callback in a separate thread or expecting the user
to store the data in the callback and convert in a separate
thread/process would work, but I haven't tried that yet.

I'm not sure how cothread handles such an scenario, but I guess it
runs callbacks in a separate thread.

I'm also not sure whether the above change is the best idea for
default use, but having a 'DONT UNPACK' option for each PV/ChannelID
might be a useful option.  In this case, for example, it might allow a
faster image processing and display program, at least in the sense
that fewer frames would be dropped.

Any suggestions on whether that would be a useful addition?

--Matt