Re: Weird channel access problem within Python softioc process

["Zhang, Tong via Tech-talk" <tech-talk at aps.anl.gov>]

Hello Michael and all,

I tested with 'import epics' after the line of softioc.iocInit(), but
still got the same results as the first example what Paul showed.

I also tested Paul's second example, but got the same results as the
first example.

Then I tried put a line of 'epics.caget('prefix:long')' after
softioc.iocInit(), it worked somewhat like below output:

Starting iocInit
#######################################################################
#####
## EPICS 7.0.6.0
## Rev. 7.0.6.99.1.0
#######################################################################
#####
iocRun: All initialization complete
2021-08-11T10:13:28.353 Using dynamically assigned TCP port 53839.
cannot connect to prefix:long
thread1: 1
thread2: 2
thread1: 2
thread2: 3
thread1: 3
thread2: 4
thread1: 4
thread2: 5
thread1: 5
thread2: 6
thread1: 6
thread2: 7
thread1: 7
thread2: 8
thread2: 8
thread1: 8
thread2: 9
thread1: 9
thread2: 10
thread1: 10
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "test6.py", line 41, in thread2
thread1: 11
    val = pv.get()
  File "/home/tong/.local/lib/python3.8/site-packages/epics/pv.py",
line 485, in get
    data = self.get_with_metadata(count=count, as_string=as_string,
  File "/home/tong/.local/lib/python3.8/site-packages/epics/pv.py",
line 55, in wrapped
    return func(self, *args, **kwargs)
  File "/home/tong/.local/lib/python3.8/site-packages/epics/pv.py",
line 589, in get_with_metadata
    elif self.nelm <= 1 or val is None:
TypeError: '<=' not supported between instances of 'NoneType' and 'int'
free(): double free detected in tcache 2
Aborted (core dumped) 

From which, line 'cannot connect to prefix:long' says the pv cannot be
connected, but the threads output seemed good, except some error
messages.

The reason why I put the caget() line after softioc.iocInit() is I came
across a similar problem when working with pythonSoftIoc and pyepics,
but I'm using cothread instead of threadings.Thread, this approach
solved my issue (when the callbacks (where pyepics is involved) are not
executed) to make the cothread routines work as expected.

I'm hoping I could understand this issue a little bit deeper, too.

Thanks,
Tong 

On Tue, 2021-08-10 at 20:37 -0700, Michael Davidsaver via Tech-talk
wrote:
> [EXTERNAL] This email originated from outside of FRIB
> 
> On 8/10/21 5:05 PM, Paul Richards via Tech-talk wrote:
> > All,
> > 
> >  
> > 
> > At Keck, we have been using the EPICS Python softioc with great
> > success.  The recent 3.1 release has made it even more popular and
> > easier to use.
> > 
> >  
> > 
> > That said, I have found a very odd behavior that I can replicate,
> > relating to the consumption of keywords inside the same process
> > that runs the softioc.  Sometimes it makes sense to do it this way,
> > instead of finding a reference to the channel in the IOC and
> > manipulating that.
> > 
> >  
> > 
> > A simplified example of the problem:
> > 
> >  1. Import the pyepics and softioc libraries.
> >  2. Start a softioc with a channel (call it ‘prefix:long’ in my
> > example).
> >  3. Start a thread that periodically updates the channel with a new
> > value.
> >  4. Start another thread that tries to access this channel, either
> > with epics.PV() or epics.caget().
> >  5. Channel access in thread 2 always times out, forever.
> > 
> >  
> > 
> > Today I found a “solution” to this that does not make sense: if I
> > start the softioc (the call to softioc.iotInit()) in the first
> > thread, and not in the main process, everything works as expected.
> > 
> >  
> > 
> > Our engineers are unable to explain this behavior.  I hope there’s
> > someone here on the list that might know why!
> 
> You might try moving the 'import epics' line after softioc.iocInit().
> 
> fyi. libca has some special behavior when run within an IOC process
> to facilitate access to local records.  As I recall, this imposes
> a required initialization order.
> 
> Also, libca uses a thread local variable to hold the client context
> pointer.  It is possible that pyepics is leaving its context pointer
> in place, which will likely cause problems during iocInit().
> 
> 
> > Example code that demonstrates the failure:
> > 
> > --------------------
> > 
> >  
> > 
> > import os
> > 
> > import time
> > 
> > import threading
> > 
> >  
> > 
> > os.environ['EPICS_CA_SERVER_PORT'] = '5100'
> > 
> > os.environ['EPICS_CA_ADDR_LIST'] = 'localhost:5100'
> > 
> > os.environ['EPICS_CA_AUTO_ADDR_LIST'] = 'NO'
> > 
> >  
> > 
> > try:
> > 
> >     import epicscorelibs.path.pyepics
> > 
> > except ImportError:
> > 
> >     pass
> > 
> > import epics
> > 
> >  
> > 
> > from softioc import softioc, builder
> > 
> >  
> > 
> > # Create a long record for this demo
> > 
> > long = builder.longOut('prefix:long', initial_value=1)
> > 
> > builder.LoadDatabase()
> > 
> > softioc.iocInit()
> > 
> >  
> > 
> > def thread1():
> > 
> >     while True:
> > 
> >         val = long.get()
> > 
> >         long.set(val + 1)
> > 
> >         print(f'thread1: {val}')
> > 
> >         time.sleep(1)
> > 
> >  
> > 
> > def thread2():
> > 
> >     pv = epics.PV('prefix:long')
> > 
> >  
> > 
> >     while True:
> > 
> >         val = pv.get()
> > 
> >         print(f'thread2: {val}')
> > 
> >         time.sleep(1)
> > 
> >  
> > 
> > t1 = threading.Thread(target=thread1, daemon=True)
> > 
> > t2 = threading.Thread(target=thread2, daemon=True)
> > 
> >  
> > 
> > t1.start()
> > 
> > t2.start()
> > 
> >  
> > 
> > time.sleep(10)
> > 
> >  
> > 
> >  
> > 
> >  
> > 
> > Example that works; note the only change is the movement of the
> > .iocInit() line down into thread1:
> > 
> > --------------------
> > 
> >  
> > 
> > import os
> > 
> > import time
> > 
> > import threading
> > 
> >  
> > 
> > os.environ['EPICS_CA_SERVER_PORT'] = '5100'
> > 
> > os.environ['EPICS_CA_ADDR_LIST'] = 'localhost:5100'
> > 
> > os.environ['EPICS_CA_AUTO_ADDR_LIST'] = 'NO'
> > 
> >  
> > 
> > try:
> > 
> >     import epicscorelibs.path.pyepics
> > 
> > except ImportError:
> > 
> >     pass
> > 
> > import epics
> > 
> >  
> > 
> > from softioc import softioc, builder
> > 
> >  
> > 
> > # Create a long record for this demo
> > 
> > long = builder.longOut('prefix:long', initial_value=1)
> > 
> > builder.LoadDatabase()
> > 
> >  
> > 
> > def thread1():
> > 
> >     softioc.iocInit()
> > 
> >  
> > 
> >     while True:
> > 
> >         val = long.get()
> > 
> >         long.set(val + 1)
> > 
> >         print(f'thread1: {val}')
> > 
> >         time.sleep(1)
> > 
> >  
> > 
> > def thread2():
> > 
> >    pv = epics.PV('prefix:long')
> > 
> >  
> > 
> >     while True:
> > 
> >         val = pv.get()
> > 
> >         print(f'thread2: {val}')
> > 
> >         time.sleep(1)
> > 
> >  
> > 
> > t1 = threading.Thread(target=thread1, daemon=True)
> > 
> > t2 = threading.Thread(target=thread2, daemon=True)
> > 
> >  
> > 
> > t1.start()
> > 
> > t2.start()
> > 
> >  
> > 
> > time.sleep(10)
> > 
> >  
> > 
> >  
> > 
> > The output when it fails:
> > 
> >  
> > 
> > $ python3 ./demoFailingDaemon.py
> > 
> > Starting iocInit
> > 
> > ###################################################################
> > #########
> > 
> > ## EPICS 7.0.6.0
> > 
> > ## Rev. 7.0.6.99.1.0
> > 
> > ###################################################################
> > #########
> > 
> > iocRun: All initialization complete
> > 
> > thread1: 1
> > 
> > thread1: 2
> > 
> > thread1: 3
> > 
> > thread2: None
> > 
> > thread1: 4
> > 
> > thread1: 5
> > 
> > thread2: None
> > 
> > thread1: 6
> > 
> > thread1: 7
> > 
> > thread1: 8
> > 
> > thread2: None
> > 
> > thread1: 9
> > 
> > thread1: 10
> > 
> > thread1: 11
> > 
> >  
> > 
> >  
> > 
> > The output when it works:
> > 
> >  
> > 
> > $ python3 ./demoWorkingDaemon.py
> > 
> > Starting iocInit
> > 
> > ###################################################################
> > #########
> > 
> > ## EPICS 7.0.6.0
> > 
> > ## Rev. 7.0.6.99.1.0
> > 
> > ###################################################################
> > #########
> > 
> > iocRun: All initialization complete
> > 
> > thread1: 1
> > 
> > thread2: 2
> > 
> > thread1: 2
> > 
> > thread2: 3
> > 
> > thread1: 3
> > 
> > thread2: 4
> > 
> > thread1: 4
> > 
> > thread2: 5
> > 
> > thread1: 5
> > 
> > thread2: 6
> > 
> > thread1: 6
> > 
> > thread2: 7
> > 
> > thread1: 7
> > 
> > thread2: 8
> > 
> > thread1: 8
> > 
> > thread2: 9
> > 
> > thread1: 9
> > 
> > thread2: 10
> > 
> > thread1: 10
> > 
> > thread2: 11
> > 
> >  
> > 
> >  
> > 
> >  
> > 
> >  
> > 
> >  
> > 
> > Paul
> > 
> >  
> > 
> >  
> > 
> >  
> > 
> > Description: Description: cid:image001.jpg@01CB39FD.C1593830
> > 
> > 	
> > 
> > *Paul Richards*
> > 
> > *Software Engineer*
> > 
> > W. M. Keck Observatory
> > 
> > 65-1120 Mamalahoa Hwy.
> > 
> > Kamuela, Hawai'i 96743
> > 
> > 	
> > 
> > Main: (808) 885-7887
> > 
> > Direct: (808) 881-3537
> > 
> > Fax: (808) 885-3535
> > 
> >  
> > 
> >  
> >