Experimental Physics and
Industrial Control System

Michael Davidsaver via Tech-talk <tech-talk at aps.anl.gov> · Thu, 5 Aug 2021 16:51:49 -0700

On 8/5/21 9:25 AM, Daykin, Evan wrote:
> Yes, there are other support modules:
> Calc, Tbuf (time buffer for pico8), PyDevSup, SScan, autosave, iocstats, and caputlog.
> 
> If I set the IOC back to 1 thread with all four cards, the shell is spammed with output:
> 
> DIAG_MTCA17:PICO8_CH0:MIN_ASUB: repeated sample (diff=0.000000) [1 times]

> I assume this is because there is some bottleneck. If I reduce to 1 thread and 1 hardware device, the output is gone. So, with the following in mind:
> 
> -1 thread is too slow for four devices
> -multiple threads is untested
> 
> Is the best workaround to split these into multiple IOCs?

Look at it this way.  It is something you can try which might change the situation.
If simply splitting things up does appear to resolve the issue, that is good evidence
that it is load.  If not, well, this still could be a logic error.

My first suggestion would be to split out, or temporarily disable, the python part.
Since the stack traces you posted doesn't show any python working threads, any significant
work being done by python code is happening on a scan or callback thread where it
can contend with other processing.

>  That way, I assume, there is parallel processing without untested resource contention.
> 
> 
> 
> 
> -----Original Message-----
> From: Michael Davidsaver <mdavidsaver at gmail.com> 
> Sent: Thursday, August 5, 2021 11:41 AM
> To: Johnson, Andrew N. <anj at anl.gov>; Daykin, Evan <daykin at frib.msu.edu>
> Cc: EPICS tech-talk <tech-talk at aps.anl.gov>; Rivers, Mark L. <rivers at cars.uchicago.edu>
> Subject: Re: cbLow consumption randomly stopping
> 
> [EXTERNAL] This email originated from outside of FRIB
> 
> On 8/5/21 8:30 AM, Johnson, Andrew N. wrote:
>> Hi Evan,
>>
>> Does this IOC load /any/ other EPICS drivers or device support at all (other than the standard ones that come with Base)?
> 
>> Your configuration with multiple callback threads is unusual, and a lot of device/driver support might not have been written or tested with the thought that there could more than one such callback in mind, so any such support code might be contributing to this issue.
> 
> I'll second this.  I never tested the pico8 driver (or any of my
> drivers) with parallel callbacks enabled.
> 

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System