Re: Incompatibility of pyepics and pvaPy 4.0.0 on Windows

["Johnson, Andrew N. via Tech-talk" <tech-talk at aps.anl.gov>]

Hi Matt,

Thanks for working on this; can script be made to fail on Linux too, out of interest? Unfortunately neither Sinisa nor I have local access to a Windows system, although there is one that we can remote into, so debugging this stuff isn’t that easy for us. Having your script to test should help though if it fails for us too.

I had a chat with Sinisa earlier. He’s currently working on a possible way to resolve the problem by building EPICS Base without creating any shared libraries, then linking the loadable library for pvaPy using the non-shared library versions, so the result would be that the pvaPy library would be completely self-contained. It would have copies of the libca and libcom routines embedded inside it, but Python would have no way to get to those symbols as they wouldn’t be exported. It occurs to me now that there might be some issues with name decorations when doing the final linke, we’ll have to see.

One thing you maybe could try would be to force (recompile?) pyepics to use the same copies of libca and libcom that pvaPy was built against. You want to be sure that both sets of bindings are calling the exact same set of DLLs. Mark said yesterday that wasn’t able to get pyepics to work with the DLLs that came with pvaccess, was he using the prepackaged versions? Can you reproduce that?

- Andrew

On Sep 10, 2021, at 3:58 PM, Matt Newville <newville at cars.uchicago.edu> wrote:

Hi Andrew, All, 

I was able to reproduce some of this behavior. I have not fully tracked the problem down. I think it does have something to do with the ca.dll from pvaccess, but unfortunately I am not certain how to get a more detailed, C++ traceback from these errors.

To get a failure, I have to do something that I can believe is dangerous:

     a) create CA channels from the main thread

     b) at nearly the same time (and so likely as connections are still being established) create channels in a sub-thread. 

This seems to not depend on whether I use epics.ca.CAThread or jthreading.Thread -- they all use the initial context.

For testing, I used Python 3.8.11 (from Anaconda Python) on Windows10, pyepics 3.5.0, pvapy 4.0.0.  The pyepics versions of ca.dll/Com.dll for Windows64 are from a nightly build of epics-base-7.0-win64 run in May, 2021. The script can cause these access violations in ca_pend_event().  I've added some notes for what can avoid these violations.   I *think* that points to problems handling connection (or the establishment of initial monitor callbacks).

--Matt

#!/usr/bin/env python

import pvaccess           # note 1
import epics
from epics.ca import CAThread

# maybe force the use of ca.dll from pvaccess, also see note 1:
pvaccess_dir, _init = os.path.split(pvaccess.__file__)
os.environ['PYEPICS_LIBCA'] = os.path.join(pvaccess_dir, 'ca.dll')
print('PYEPICS use CA/Com lib: ', epics.ca.find_libca(), epics.ca.find_libCom())

pvnames = ['PyTest:ao1',  'PyTest:long2k']

def watch_pvs(my_pvnames, runtime=5.0):
    my_pvs = [epics.get_pv(name) for name in my_pvnames]
    endtime = time.time() + runtime
    print("created PVS in sub thread with context: ", epics.ca.current_context())
    time.sleep(0.0025)      # note 3
   
    while time.time() < endtime:
        for pv in my_pvs:
            print(pv.pvname, pv.get())
        time.sleep(0.1)
       
# create and maybe get values in main thread:
print("created PVS in main thread with context: ", epics.ca.current_context())
for name in pvnames:
    pv = epics.PV(name)
    # pv.get()           # note 2

# set up thread to create and get PV values
#cthread = CAThread(target=watch_pvs, args=(pvnames, ))
cthread = Thread(target=watch_pvs, args=(pvnames, ))

cthread.start()
time.sleep(0.5)
cthread.join()
# done

Notes:

     1.  Commenting out "import pvaccess" removes the problem.  This is independent of whether PYEPICS_LIBCA is set to point to the ca.dll from pvaccess.

     2.  It seems important to create PVs in the main thread that do not complete connection before connections are created in the sub-thread.  Commenting out  a get or putting in a long enough "sleep" will avoid the problem.

     3.  As with note 2, increasing the wait time between "create PVs" and "actually get values" can also avoid the problem.

For Note 1, I'm not certain but believe that pvaccess is loading all the dlls at import time.  I *think* that the Python process will then only use 1 version of the DLL, possibly no matter what "PYEPICS_LIBCA" is set to.  That is, by the time pyepics tries to load ca.dll, the Windows process will already have a 'ca.dll' loaded and simply use that one.  That is also backed up by doing

     import epics

     epics.initialize_libca()  # force ca.dll to be loaded.

     import pvacess

that gives an ImportError with DLL load failed while importing pvaccess: The specified procedure could not be found. 

I think perhaps the next thing to try would be to write such a threading code using pvaccess, or unwind some of the automated connection portions of pyepics to get that script down to more like "bare CA calls", that might then be translated into C++.  I'm not sure I would be able to do those very quickly.

Would you recommend trying to force pyepics to use a more recent version of ca.dll?  

--Matt

-- 

Complexity comes for free, simplicity you have to work for.