Experimental Physics and
Industrial Control System

On 6/27/22 10:24, Mark Rivers via Tech-talk wrote:
> Folks,
>
> I am trying to help Pete Siddons with a problem with an IOC that uses the quadEM module for a BPM he is developing.  It runs EPICS on a Zynq processor.  He says it was working on Debian 7, but is not working on Debian 11.

Which Linux kernel variant/version?

>  Because the IOC talks to /dev/mem it needs to run as root.

ick...

I would very much recommend that Pete look at creating a real Linux driver.
mmap()ing "/dev/mem" into an userspace process is playing with fire.

Alternately, adjust file permissions and/or Linux kernel options to bypass
some /dev/mem safety checks (eg. STRICT_DEVMEM and IO_STRICT_DEVMEM)

https://urldefense.com/v3/__https://elixir.bootlin.com/linux/latest/source/lib/Kconfig.debug*L1804__;Iw!!P4SdNyxKAPE!FUgNGKi4r71uFnnc8bN6dWdAsVE564Lgzf5o-sG6kwN9Sen6WKDXYFqb3EbEpUN7LozrwhsWJrOZ-VqJllx9ig$


>
> The driver can be built in simulation mode, so it does not talk to the hardware at all.  In this mode the IOC runs fine when run as a non-privileged user, and takes a total of 3.5 seconds to start up.  This includes the time to create many areaDetector plugins.
>
> However, when running as root there are timeout errors waiting for each of the plugin threads to be created.  The timeout is 2 seconds, and increasing it to 10 seconds does not fix the problem.

The quickest way to an answer will probably be gdb (or gdbserver).

Run with a break point set where the "timeout" message is printed[1].
When this is reached, dump all of the thread stacks. (run
"thread apply all backtrace")

This will show if the plugin threads are being created, and if so,
what they are doing.

[1] https://github.com/areaDetector/ADCore/blob/bcaced94ad48b9bb5fdbd1a7f6d9d8be5c416003/ADApp/pluginSrc/NDPluginDriver.cpp#L912


If you haven't encountered gdbserver before, there are two parts.
On the target run something like:

> $ gdbserver 0.0.0.0:3333 ./bin/linux-arm/myioc st.cmd

Then on the (Debian) host run:

> $ gdb-multiarch
> (gdb) target extended-remote <target-ip>:3333
> (gdb) b NDPluginDriver.cpp:912
> (gdb) run
> ... wait for breakpoint
> (gdb) thread apply all backtrace
> ... lots of output ...
> (gdb) quit

The TCP port number can be anything.  3333 is an arbitrary choice.


> This the abbreviated IOC startup output when not running as root.
>
> *****************************
>
> date
>
> 2022/06/27 16:03:39.620634
>
> errlogInit(5000)
>
> < envPaths
>
> epicsEnvSet("IOC","iocNSLS2_IC")
>
> epicsEnvSet("TOP","/home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4")
>
> …
>
> dbLoadDatabase("/home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4/dbd/quadEMTestApp.dbd")
>
> quadEMTestApp_registerRecordDeviceDriver(pdbbase)
>
> # The search path for database files
>
> # Note: the separator between the path entries needs to be changed to a semicolon (;) on Windows
>
> date
>
> 2022/06/27 16:03:39.953518
>
> drvNSLS2_ICConfigure("NSLS2_IC", 0, 10000)
>
> Memory mapped
>
> Poller thread created
>
> Creating parameters 0
>
> Poll thread starts
>
> Creating parameters 1
>
> Creating parameters 2
>
> Creating parameters 3
>
> Creating parameters 4
>
> Opening DACS
>
> Getting firmware version
>
> fpgabase=0x569a60
>
> FPGA version=1001
>
> Calling callbacks
>
> date
>
> 2022/06/27 16:03:40.129864
>
> # Create 11 statistics plugins
>
> NDStatsConfigure("NSLS2_IC_STATS1", 20, 0, "NSLS2_IC", 0, 0, 0)
>
> dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDStats.template",     "P=quadEMTest:,R=NSLS2_IC:Current1:, PORT=NSLS2_IC_STATS1, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC,NDARRAY_ADDR=0, HIST_SIZE=256,XSIZE=10000,YSIZE=0,NCHANS=1000,ENABLED=1,")
>
> NDTimeSeriesConfigure("NSLS2_IC_STATS1_TS", 20, 0, "NSLS2_IC_STATS1", 1, 23)
>
> …
>
> NDFFTConfigure("NSLS2_IC_FFT11", 20, 0, "NSLS2_IC_TS1", 10)
>
> dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:PosY:,PORT=NSLS2_IC_FFT11,ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=10,NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=PosY")
>
> #< $(QUADEM)/iocBoot/saveRestore.cmd
>
> date
>
> 2022/06/27 16:03:43.046352
>
> iocInit()
>
> ############################################################################
>
> ## EPICS R7.0.6.1
>
> ## Rev. 2022-03-25T16:17+0000
>
> ###########################
>
> iocRun: All initialization complete
>
> #################################################
>
> Gain: 0
>
> Setting bias voltage to 0.000000
>
> date
>
> 2022/06/27 16:03:46.982984
>
> # save settings every thirty seconds
>
> #create_monitor_set("auto_settings.req",30,"P=$(PREFIX), R=$(RECORD)")
>
> *****************************
>
> The IOC starts up fine.  The total time to start is about 3.5 seconds.  There are many plugin threads created during this time.
>
> This the abbreviated IOC startup output when running as root.
>
> *****************************
>
> date
>
> 2022/06/27 14:20:48.298214
>
> errlogInit(5000)
>
> < envPaths
>
> epicsEnvSet("IOC","iocNSLS2_IC")
>
> epicsEnvSet("TOP","/home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4")
>
> …
>
> dbLoadDatabase("/home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4/dbd/quadEMTestApp.dbd")
>
> quadEMTestApp_registerRecordDeviceDriver(pdbbase)
>
> # The search path for database files
>
> # Note: the separator between the path entries needs to be changed to a semicolon (;) on Windows
>
> epicsEnvSet("EPICS_DB_INCLUDE_PATH", "/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db:/home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4/db")
>
> < /home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4/iocBoot/iocNSLS2_IC/NSLS2_IC.cmd
>
> date
>
> 2022/06/27 14:20:48.626216
>
> …
>
> drvNSLS2_ICConfigure("NSLS2_IC", 0, 10000)
>
> Memory mapped
>
> Poller thread created
>
> Creating parameters 0
>
> Poll thread starts
>
> Creating parameters 1
>
> Creating parameters 2
>
> Creating parameters 3
>
> Creating parameters 4
>
> Opening DACS
>
> Getting firmware version
>
> fpgabase=0x5809b8
>
> FPGA version=1001
>
> Calling callbacks
>
> …
>
> date
>
> 2022/06/27 14:20:48.802463
>
> < /home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4/iocBoot/commonPlugins.cmd
>
> …
>
> # Create 11 statistics plugins
>
> NDStatsConfigure("NSLS2_IC_STATS1", 20, 0, "NSLS2_IC", 0, 0, 0)
>
> dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDStats.template",     "P=quadEMTest:,R=NSLS2_IC:Current1:, PORT=NSLS2_IC_STATS1, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC,NDARRAY_ADDR=0, HIST_SIZE=256,XSIZE=10000,YSIZE=0,NCHANS=1000,ENABLED=1,")
>
> …
>
> # FFT plugins
>
> NDFFTConfigure("NSLS2_IC_FFT1", 20, 0, "NSLS2_IC_TS1", 0)
>
> dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:Current1:, PORT=NSLS2_IC_FFT1, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=0, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=Current1")
>
> NDFFTConfigure("NSLS2_IC_FFT2", 20, 0, "NSLS2_IC_TS1", 1)
>
> dbLoadRecords("/home/peter/epics/extensions/synApps/support2022/06/27 14:21:19.270 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:21:23.270 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:21:26.247 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:21:28.278 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> Starting iocInit
>
> -master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:Current2:, PORT=NSLS2_IC_FFT2, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=1, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=Current2")
>
> NDFFTConfigure("NSLS2_IC_FFT3", 20, 0, "NSLS2_IC_TS1", 2
>
> dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:Current3:, PORT=NSLS2_IC_FFT3, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=2, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=Current3")
>
> NDFFTConfigure("NSLS2_IC_FFT4", 20, 0, "NSLS2_IC_TS1", 3)
>
> dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:Current4:, PORT=NSLS2_IC_FFT4, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=3, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=Current4")
>
> NDFFTConfigure("NSLS2_IC_FFT5", 20, 0, "NSLS2_IC_TS1", 4)
>
> dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:SumX:, PORT=NSLS2_IC_FFT5, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=4, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=SumX")
>
> NDFFTConfigure("NSLS2_IC_FFT6", 20, 0, "NSLS2_IC_TS1", 5)
>
> dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:SumY:, PORT=NSLS2_IC_FFT6, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=5, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=SumY")
>
> NDFFTConfigure("NSLS2_IC_FFT7", 20, 0, "NSLS2_IC_TS1", 6)
>
> dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:SumAll:, PORT=NSLS2_IC_FFT7, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=6, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=SumAll")
>
> NDFFTConfigure("NSLS2_IC_FFT8", 20, 0, "NSLS2_IC_TS1", 7)
>
> dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:DiffX:, PORT=NSLS2_IC_FFT8, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=7, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=DiffX")
>
> NDFFTConfigure("NSLS2_IC_FFT9", 20, 0, "NSLS2_IC_TS1", 8)
>
> dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:DiffY:, PORT=NSLS2_IC_FFT9, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=8, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=DiffY")
>
> NDFFTConfigure("NSLS2_IC_FFT10", 20, 0, "NSLS2_IC_TS1", 9)
>
> dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:PosX:,PORT=NSLS2_IC_FFT10,ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=9, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=PosX")
>
> NDFFTConfigure("NSLS2_IC_FFT11", 20, 0, "NSLS2_IC_TS1", 10)
>
> dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:PosY:,PORT=NSLS2_IC_FFT11,ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=10,NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=PosY")
>
> #< $(QUADEM)/iocBoot/saveRestore.cmd
>
> date
>
> 2022/06/27 14:21:29.253141
>
> iocInit()
>
> ############################################################################
>
> ## EPICS R7.0.6.1
>
> ## Rev. 2022-03-25T16:17+0000
>
> ###########################2022/06/27 14:23:25.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:23:25.241 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
>
> 2022/06/27 14:23:32.243 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:23:32.244 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
>
> 2022/06/27 14:23:34.244 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
>
> epicsThread::~epicsThread(): "_main_" blocking for thread "NSLS2_IC_STATS2_Plugin_1" to exit
>
> was epicsThread object destroyed before thread exit ?
>
> epicsThread::~epicsThread(): "_main_" blocking for thread "NSLS2_IC_STATS2_Plugin_1" to exit
>
> was epicsThread object destroyed before thread exit ?
>
> epicsThread::~epicsThread(): "_main_" blocking for thread "NSLS2_IC_STATS2_Plugin_1" to exit
>
> was epicsThread object destroyed before thread exit ?
>
> 2022/06/27 14:24:05.241 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
>
> 2022/06/27 14:24:08.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:24:08.241 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
>
> 2022/06/27 14:24:08.241 quadEMTest:NSLS2_IC:Current2:TS:QueueSize devAsynInt32::processCallbackOutput process write error NDPluginTimeSeries::writeInt32: status=0, function=64, value=20
>
> 2022/06/27 14:24:10.242 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
>
> epicsThread::~epicsThread(): "_main_" blocking for thread "NSLS2_IC_STATS2_TS_Plugin_1" to exit
>
> was epicsThread object destroyed before thread exit ?
>
> 2022/06/27 14:24:23.241 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
>
> 2022/06/27 14:24:23.242 quadEMTest:NSLS2_IC:Current2:TS:NumThreads devAsynInt32::processCallbackOutput process write error NDPluginTimeSeries::writeInt32: status=0, function=67, value=1
>
> 2022/06/27 14:24:26.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:24:26.241 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
>
> 2022/06/27 14:24:28.242 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
>
> 2022/06/27 14:24:35.242 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
>
> 2022/06/27 14:24:51.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:24:51.241 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
>
> 2022/06/27 14:24:53.242 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
>
> 2022/06/27 14:24:54.240 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
>
> 2022/06/27 14:25:09.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:25:09.241 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
>
> 2022/06/27 14:25:11.241 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
>
> epicsThread::~epicsThread(): "_main_" blocking for thread "NSLS2_IC_STATS7_Plugin_1" to exit
>
> was epicsThread object destroyed before thread exit ?
>
> 2022/06/27 14:25:26.240 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
>
> 2022/06/27 14:25:31.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:25:31.242 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
>
> 2022/06/27 14:25:31.242 quadEMTest:NSLS2_IC:SumAll:TS:NumThreads devAsynInt32::processCallbackOutput process write error NDPluginTimeSeries::writeInt32: status=0, function=67, value=1
>
> 2022/06/27 14:25:34.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:25:34.242 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
>
> 2022/06/27 14:25:36.243 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
>
> 2022/06/27 14:25:40.240 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
>
> 2022/06/27 14:25:43.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:25:43.242 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
>
> 2022/06/27 14:25:43.243 quadEMTest:NSLS2_IC:DiffX:TS:QueueSize devAsynInt32::processCallbackOutput process write error NDPluginTimeSeries::writeInt32: status=0, function=64, value=20
>
> 2022/06/27 14:25:45.244 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
>
> 2022/06/27 14:25:47.240 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
>
> 2022/06/27 14:25:47.242 quadEMTest:NSLS2_IC:DiffX:TS:NumThreads devAsynInt32::processCallbackOutput process write error NDPluginTimeSeries::writeInt32: status=0, function=67, value=1
>
> 2022/06/27 14:25:50.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:25:50.242 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
>
> 2022/06/27 14:25:52.243 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
>
> epicsThread::~epicsThread(): "_main_" blocking for thread "NSLS2_IC_STATS9_Plugin_1" to exit
>
> was epicsThread object destroyed before thread exit ?
>
> 2022/06/27 14:26:04.240 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
>
> 2022/06/27 14:26:07.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:26:07.248 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
>
> 2022/06/27 14:26:07.249 quadEMTest:NSLS2_IC:DiffY:TS:QueueSize devAsynInt32::processCallbackOutput process write error NDPluginTimeSeries::writeInt32: status=0, function=64, value=20
>
> 2022/06/27 14:26:09.249 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
>
> 2022/06/27 14:26:13.240 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
>
> 2022/06/27 14:26:13.248 quadEMTest:NSLS2_IC:DiffY:TS:NumThreads devAsynInt32::processCallbackOutput process write error NDPluginTimeSeries::writeInt32: status=0, function=67, value=1
>
> 2022/06/27 14:26:24.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:26:24.242 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
>
> 2022/06/27 14:26:26.243 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
>
> 2022/06/27 14:26:35.240 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
>
> 2022/06/27 14:26:38.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:26:38.249 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
>
> 2022/06/27 14:26:38.249 quadEMTest:NSLS2_IC:PosY:TS:QueueSize devAsynInt32::processCallbackOutput process write error NDPluginTimeSeries::writeInt32: status=0, function=64, value=20
>
> 2022/06/27 14:26:40.250 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
>
> epicsThread::~epicsThread(): "_main_" blocking for thread "NSLS2_IC_STATS11_TS_Plugin_1" to exit
>
> was epicsThread object destroyed before thread exit ?
>
> 2022/06/27 14:26:52.241 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
>
> 2022/06/27 14:26:52.242 quadEMTest:NSLS2_IC:PosY:TS:NumThreads devAsynInt32::processCallbackOutput process write error NDPluginTimeSeries::writeInt32: status=0, function=67, value=1
>
> 2022/06/27 14:26:55.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:26:55.242 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
>
> 2022/06/27 14:26:55.243 quadEMTest:NSLS2_IC:image1:QueueSize devAsynInt32::processCallbackOutput process write error
>
> 2022/06/27 14:26:57.244 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
>
> 2022/06/27 14:27:01.240 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
>
> 2022/06/27 14:27:01.242 quadEMTest:NSLS2_IC:image1:NumThreads devAsynInt32::processCallbackOutput process write error
>
> 2022/06/27 14:27:15.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:27:15.248 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
>
> 2022/06/27 14:27:15.249 quadEMTest:NSLS2_IC:FFT:Current2:NumThreads devAsynInt32::processCallbackOutput process write error
>
> 2022/06/27 14:27:23.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:27:23.242 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
>
> 2022/06/27 14:27:23.243 quadEMTest:NSLS2_IC:FFT:Current4:QueueSize devAsynInt32::processCallbackOutput process write error
>
> 2022/06/27 14:27:25.244 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
>
> 2022/06/27 14:27:28.241 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
>
> 2022/06/27 14:27:28.249 quadEMTest:NSLS2_IC:FFT:Current4:NumThreads devAsynInt32::processCallbackOutput process write error
>
> 2022/06/27 14:27:33.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:27:33.248 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
>
> 2022/06/27 14:27:33.249 quadEMTest:NSLS2_IC:FFT:SumX:NumThreads devAsynInt32::processCallbackOutput process write error
>
> 2022/06/27 14:27:38.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:27:38.249 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
>
> 2022/06/27 14:27:38.249 quadEMTest:NSLS2_IC:FFT:SumY:NumThreads devAsynInt32::processCallbackOutput process write error
>
> 2022/06/27 14:27:38.249 NDPluginDriver::deleteCallbackThreads incorrect response from plugin thread 0 = 0
>
> 2022/06/27 14:27:39.240 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
>
> 2022/06/27 14:27:39.241 quadEMTest:NSLS2_IC:FFT:SumAll:QueueSize devAsynInt32::processCallbackOutput process write error
>
> 2022/06/27 14:27:41.243 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:27:41.244 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
>
> 2022/06/27 14:27:41.244 quadEMTest:NSLS2_IC:FFT:SumAll:NumThreads devAsynInt32::processCallbackOutput process write error
>
> 2022/06/27 14:27:45.241 NDPluginDriver::deleteCallbackThreads incorrect response from plugin thread 0 = 0
>
> 2022/06/27 14:27:46.241 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
>
> 2022/06/27 14:27:46.242 quadEMTest:NSLS2_IC:FFT:DiffY:QueueSize devAsynInt32::processCallbackOutput process write error
>
> 2022/06/27 14:27:47.241 NDPluginDriver::deleteCallbackThreads incorrect response from plugin thread 0 = 0
>
> 2022/06/27 14:27:48.241 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
>
> 2022/06/27 14:27:48.242 quadEMTest:NSLS2_IC:FFT:PosX:QueueSize devAsynInt32::processCallbackOutput process write error
>
> 2022/06/27 14:27:49.241 NDPluginDriver::deleteCallbackThreads incorrect response from plugin thread 0 = 0
>
> 2022/06/27 14:27:50.241 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
>
> 2022/06/27 14:27:50.244 quadEMTest:NSLS2_IC:FFT:PosY:QueueSize devAsynInt32::processCallbackOutput process write error
>
> 2022/06/27 14:27:53.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:27:56.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:27:59.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:28:02.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:28:05.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:28:08.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> 2022/06/27 14:28:11.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
>
> iocRun: All initialization complete
>
> #################################################
>
> Gain: 0
>
> Setting bias voltage to 0.000000
>
> date
>
> 2022/06/27 14:32:29.293229
>
> # save settings every thirty seconds
>
> #create_monitor_set("auto_settings.req",30,"P=$(PREFIX), R=$(RECORD)")
>
> epics>
>
> *****************************
>
> There are many errors waiting for plugin threads to start  (and then waiting for them to exit).  The total time to start is about 12 minutes compared to 3.5 seconds when not running as root.
>
> Any ideas why there is an issue creating plugin threads when running as root, but not when running as a normal user?
>
> Thanks,
>
> Mark
>