RE: Incompatibility of pyepics and pvaPy 4.0.0 on Windows

[Mark Rivers via Tech-talk <tech-talk at aps.anl.gov>]

I tested tomoscan using different versions of ca.dll, as controlled by the PYEPICS_LIBCA environment variable.  The tests were all done as follows:

 

-          tomoscan was built without importing pvaccess at all

-          Python 3.8.8

-          pyepics 3.5.0

-          pvapy 4.0.0.

 

PYEPICS_LIBCA	Result
Undefined (Uses Anaconda3\envs\tomoscan\Lib\site-packages\pyepics\ca.dll)	OK
Anaconda3\envs\tomoscan\Lib\site-packages\pvaccess\ca.dll	Crash
base-7.0.5/bin/windows-x64/ca.dll	OK
base-7.0.6/bin/windows-x64/ca.dll	Crash
base-7.0.6/bin/windows-x64/ca.dll (with Freddie’s patch to libCom)	OK

 

I then copied the patched version of base-7.0.6/bin/windows-x64/Com.dll to Anaconda3\envs\tomoscan\Lib\site-packages\pvaccess\Com.dll and tested again.

 

It worked fine with PYEPICS_LIBCA= Anaconda3\envs\tomoscan\Lib\site-packages\pvaccess\ca.dll, which had failed above.

 

It also worked fine when I rebuilt tomoscan with pvaccess imports enabled and PYEPICS_LIBCA undefined, which was my original configuration that failed.

 

So I conclude that Freddie’s patch has fixed the issue.

 

Thanks Freddie!

 

Mark

 

 

 

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Freddie Akeroyd - STFC UKRI via Tech-talk
Sent: Saturday, September 11, 2021 8:02 AM
To: Veseli, Sinisa <sveseli at anl.gov>; Johnson, Andrew N. <anj at anl.gov>; Matt Newville <newville at cars.uchicago.edu>
Cc: tech-talk at aps.anl.gov; De Carlo, Francesco <decarlo at anl.gov>; Nikitin, Viktor <vnikitin at anl.gov>
Subject: RE: Incompatibility of pyepics and pvaPy 4.0.0 on Windows

 

Hi,

 

Applying the change at https://github.com/epics-base/epics-base/pull/200/files fixes the issue on my machine, does it also work for you?

 

Regards,

 

Freddie

 

From: Veseli, Sinisa <sveseli at anl.gov>
Sent: 11 September 2021 09:30
To: Akeroyd, Freddie (STFC,RAL,ISIS) <freddie.akeroyd at stfc.ac.uk>; Johnson, Andrew N. <anj at anl.gov>; Matt Newville <newville at cars.uchicago.edu>
Cc: De Carlo, Francesco <decarlo at anl.gov>; Nikitin, Viktor <vnikitin at anl.gov>; tech-talk at aps.anl.gov
Subject: Re: Incompatibility of pyepics and pvaPy 4.0.0 on Windows

 

Hi,

 

I have also reproduced the problem with Matt's script. Unlike Matt, however, I see the problem when "import pvaccess" is commented out, and PYEPICS_LIBCA is set to point to ca.dll from pvaccess.

 

Sinisa

 

--

Siniša Veseli

Scientific Software Engineering & Data Management

Advanced Photon Source

Argonne National Laboratory

sveseli at anl.gov

(630)252-9182

---

From: Freddie Akeroyd - STFC UKRI <freddie.akeroyd at stfc.ac.uk>
Sent: Friday, September 10, 2021 8:23 PM
To: Johnson, Andrew N. <anj at anl.gov>; Matt Newville <newville at cars.uchicago.edu>
Cc: Veseli, Sinisa <sveseli at anl.gov>; De Carlo, Francesco <decarlo at anl.gov>; Nikitin, Viktor <vnikitin at anl.gov>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: RE: Incompatibility of pyepics and pvaPy 4.0.0 on Windows

 

Hi,

 

I can reproduce the error on my windows machine with your script. I rebuilt 7.0.6 with debug symbols and I get:

 

Exception thrown: read access violation.

__imp_TlsGetValue(...) returned nullptr.

 

>             [Inline Frame] Com.dll!osdThreadGetTimer(...) Line 814  C

               Com.dll!epicsThreadSleep(double seconds) Line 847        C

               ca.dll!ca_client_context::pendEvent(const double & timeout) Line 586     C++

               ca.dll!ca_pend_event(double timeout) Line 474 C++

               [External Code]

               _ctypes.pyd!_call_function_pointer(int flags, int(*)() pProc, void * * avalues, _ffi_type * * atypes, _ffi_type * restype, void * resmem, int argcount) Line 874    C

               _ctypes.pyd!_ctypes_callproc(int(*)() pProc, _object * argtuple, IUnknown * pIunk, _GUID * iid, int flags, _object * argtypes, _object * restype, _object * checker) Line 1214           C

 

In 7.0.6 the wait functions were changed to use waitable timers for higher accuracy, the timer is allocated at thread creation and stored in thread local storage. It looks like there has been an error retrieving this timer from TLS. I can look at it in more detail.

 

Regards,

 

Freddie

 

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Johnson, Andrew N. via Tech-talk
Sent: 10 September 2021 22:31
To: Matt Newville <newville at cars.uchicago.edu>
Cc: EPICS tech-talk <tech-talk at aps.anl.gov>; Veseli, Sinisa <sveseli at anl.gov>; De Carlo, Francesco <decarlo at anl.gov>; Nikitin, Viktor <vnikitin at anl.gov>
Subject: Re: Incompatibility of pyepics and pvaPy 4.0.0 on Windows

 

Hi Matt,

 

Thanks for working on this; can script be made to fail on Linux too, out of interest? Unfortunately neither Sinisa nor I have local access to a Windows system, although there is one that we can remote into, so debugging this stuff isn’t that easy for us. Having your script to test should help though if it fails for us too.

 

I had a chat with Sinisa earlier. He’s currently working on a possible way to resolve the problem by building EPICS Base without creating any shared libraries, then linking the loadable library for pvaPy using the non-shared library versions, so the result would be that the pvaPy library would be completely self-contained. It would have copies of the libca and libcom routines embedded inside it, but Python would have no way to get to those symbols as they wouldn’t be exported. It occurs to me now that there might be some issues with name decorations when doing the final linke, we’ll have to see.

 

One thing you maybe could try would be to force (recompile?) pyepics to use the same copies of libca and libcom that pvaPy was built against. You want to be sure that both sets of bindings are calling the exact same set of DLLs. Mark said yesterday that wasn’t able to get pyepics to work with the DLLs that came with pvaccess, was he using the prepackaged versions? Can you reproduce that?

 

- Andrew

 

 

On Sep 10, 2021, at 3:58 PM, Matt Newville <newville at cars.uchicago.edu> wrote:

 

Hi Andrew, All, 

 

I was able to reproduce some of this behavior. I have not fully tracked the problem down. I think it does have something to do with the ca.dll from pvaccess, but unfortunately I am not certain how to get a more detailed, C++ traceback from these errors.

 

To get a failure, I have to do something that I can believe is dangerous:

     a) create CA channels from the main thread

     b) at nearly the same time (and so likely as connections are still being established) create channels in a sub-thread. 

 

This seems to not depend on whether I use epics.ca.CAThread or jthreading.Thread -- they all use the initial context.

 

For testing, I used Python 3.8.11 (from Anaconda Python) on Windows10, pyepics 3.5.0, pvapy 4.0.0.  The pyepics versions of ca.dll/Com.dll for Windows64 are from a nightly build of epics-base-7.0-win64 run in May, 2021. The script can cause these access violations in ca_pend_event().  I've added some notes for what can avoid these violations.   I *think* that points to problems handling connection (or the establishment of initial monitor callbacks).

 

--Matt

 

#!/usr/bin/env python

import pvaccess           # note 1
import epics
from epics.ca import CAThread

# maybe force the use of ca.dll from pvaccess, also see note 1:
pvaccess_dir, _init = os.path.split(pvaccess.__file__)
os.environ['PYEPICS_LIBCA'] = os.path.join(pvaccess_dir, 'ca.dll')
print('PYEPICS use CA/Com lib: ', epics.ca.find_libca(), epics.ca.find_libCom())

pvnames = ['PyTest:ao1',  'PyTest:long2k']

def watch_pvs(my_pvnames, runtime=5.0):
    my_pvs = [epics.get_pv(name) for name in my_pvnames]
    endtime = time.time() + runtime
    print("created PVS in sub thread with context: ", epics.ca.current_context())
    time.sleep(0.0025)      # note 3
   
    while time.time() < endtime:
        for pv in my_pvs:
            print(pv.pvname, pv.get())
        time.sleep(0.1)
       
# create and maybe get values in main thread:
print("created PVS in main thread with context: ", epics.ca.current_context())
for name in pvnames:
    pv = epics.PV(name)
    # pv.get()           # note 2

# set up thread to create and get PV values
#cthread = CAThread(target=watch_pvs, args=(pvnames, ))
cthread = Thread(target=watch_pvs, args=(pvnames, ))

cthread.start()
time.sleep(0.5)
cthread.join()
# done

 

Notes:

 

     1.  Commenting out "import pvaccess" removes the problem.  This is independent of whether PYEPICS_LIBCA is set to point to the ca.dll from pvaccess.

     2.  It seems important to create PVs in the main thread that do not complete connection before connections are created in the sub-thread.  Commenting out  a get or putting in a long enough "sleep" will avoid the problem.

     3.  As with note 2, increasing the wait time between "create PVs" and "actually get values" can also avoid the problem.

 

For Note 1, I'm not certain but believe that pvaccess is loading all the dlls at import time.  I *think* that the Python process will then only use 1 version of the DLL, possibly no matter what "PYEPICS_LIBCA" is set to.  That is, by the time pyepics tries to load ca.dll, the Windows process will already have a 'ca.dll' loaded and simply use that one.  That is also backed up by doing

 

     import epics

     epics.initialize_libca()  # force ca.dll to be loaded.

     import pvacess

 

that gives an ImportError with DLL load failed while importing pvaccess: The specified procedure could not be found. 

 

I think perhaps the next thing to try would be to write such a threading code using pvaccess, or unwind some of the automated connection portions of pyepics to get that script down to more like "bare CA calls", that might then be translated into C++.  I'm not sure I would be able to do those very quickly.

 

Would you recommend trying to force pyepics to use a more recent version of ca.dll?  

 

--Matt

 

 

 

 

-- 

Complexity comes for free, simplicity you have to work for.

 

This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses.