Experimental Physics and
Industrial Control System

Hi Andrew, All, 

I was able to reproduce some of this behavior. I have not fully tracked the problem down. I think it does have something to do with the ca.dll from pvaccess, but unfortunately I am not certain how to get a more detailed, C++ traceback from these errors.

To get a failure, I have to do something that I can believe is dangerous:

     a) create CA channels from the main thread

     b) at nearly the same time (and so likely as connections are still being established) create channels in a sub-thread. 

This seems to not depend on whether I use epics.ca.CAThread or jthreading.Thread -- they all use the initial context.

For testing, I used Python 3.8.11 (from Anaconda Python) on Windows10, pyepics 3.5.0, pvapy 4.0.0.  The pyepics versions of ca.dll/Com.dll for Windows64 are from a nightly build of epics-base-7.0-win64 run in May, 2021. The script can cause these access violations in ca_pend_event().  I've added some notes for what can avoid these violations.   I *think* that points to problems handling connection (or the establishment of initial monitor callbacks).

--Matt

#!/usr/bin/env python

import pvaccess           # note 1
import epics
from epics.ca import CAThread

# maybe force the use of ca.dll from pvaccess, also see note 1:
pvaccess_dir, _init = os.path.split(pvaccess.__file__)
os.environ['PYEPICS_LIBCA'] = os.path.join(pvaccess_dir, 'ca.dll')
print('PYEPICS use CA/Com lib: ', epics.ca.find_libca(), epics.ca.find_libCom())

pvnames = ['PyTest:ao1',  'PyTest:long2k']

def watch_pvs(my_pvnames, runtime=5.0):
    my_pvs = [epics.get_pv(name) for name in my_pvnames]
    endtime = time.time() + runtime
    print("created PVS in sub thread with context: ", epics.ca.current_context())
    time.sleep(0.0025)      # note 3
   
    while time.time() < endtime:
        for pv in my_pvs:
            print(pv.pvname, pv.get())
        time.sleep(0.1)
       
# create and maybe get values in main thread:
print("created PVS in main thread with context: ", epics.ca.current_context())
for name in pvnames:
    pv = epics.PV(name)
    # pv.get()           # note 2

# set up thread to create and get PV values
#cthread = CAThread(target=watch_pvs, args=(pvnames, ))
cthread = Thread(target=watch_pvs, args=(pvnames, ))

cthread.start()
time.sleep(0.5)
cthread.join()
# done

Notes:

     1.  Commenting out "import pvaccess" removes the problem.  This is independent of whether PYEPICS_LIBCA is set to point to the ca.dll from pvaccess.

     2.  It seems important to create PVs in the main thread that do not complete connection before connections are created in the sub-thread.  Commenting out  a get or putting in a long enough "sleep" will avoid the problem.

     3.  As with note 2, increasing the wait time between "create PVs" and "actually get values" can also avoid the problem.

For Note 1, I'm not certain but believe that pvaccess is loading all the dlls at import time.  I *think* that the Python process will then only use 1 version of the DLL, possibly no matter what "PYEPICS_LIBCA" is set to.  That is, by the time pyepics tries to load ca.dll, the Windows process will already have a 'ca.dll' loaded and simply use that one.  That is also backed up by doing

     import epics

     epics.initialize_libca()  # force ca.dll to be loaded.

     import pvacess

that gives an ImportError with DLL load failed while importing pvaccess: The specified procedure could not be found. 

I think perhaps the next thing to try would be to write such a threading code using pvaccess, or unwind some of the automated connection portions of pyepics to get that script down to more like "bare CA calls", that might then be translated into C++.  I'm not sure I would be able to do those very quickly.

Would you recommend trying to force pyepics to use a more recent version of ca.dll?  

--Matt