On 6/27/22 10:24, Mark Rivers via Tech-talk wrote:
Folks,
I am trying to help Pete Siddons with a problem with an IOC that uses the quadEM module for a BPM he is developing. It runs EPICS on a Zynq processor. He says it was working on Debian 7, but is not working on Debian 11.
Which Linux kernel variant/version?
Because the IOC talks to /dev/mem it needs to run as root.
ick...
I would very much recommend that Pete look at creating a real Linux driver.
mmap()ing "/dev/mem" into an userspace process is playing with fire.
Alternately, adjust file permissions and/or Linux kernel options to bypass
some /dev/mem safety checks (eg. STRICT_DEVMEM and IO_STRICT_DEVMEM)
https://elixir.bootlin.com/linux/latest/source/lib/Kconfig.debug#L1804
The driver can be built in simulation mode, so it does not talk to the hardware at all. In this mode the IOC runs fine when run as a non-privileged user, and takes a total of 3.5 seconds to start up. This includes the time to create many areaDetector plugins.
However, when running as root there are timeout errors waiting for each of the plugin threads to be created. The timeout is 2 seconds, and increasing it to 10 seconds does not fix the problem.
The quickest way to an answer will probably be gdb (or gdbserver).
Run with a break point set where the "timeout" message is printed[1].
When this is reached, dump all of the thread stacks. (run
"thread apply all backtrace")
This will show if the plugin threads are being created, and if so,
what they are doing.
[1] https://github.com/areaDetector/ADCore/blob/bcaced94ad48b9bb5fdbd1a7f6d9d8be5c416003/ADApp/pluginSrc/NDPluginDriver.cpp#L912
If you haven't encountered gdbserver before, there are two parts.
On the target run something like:
$ gdbserver 0.0.0.0:3333 ./bin/linux-arm/myioc st.cmd
Then on the (Debian) host run:
$ gdb-multiarch
(gdb) target extended-remote <target-ip>:3333
(gdb) b NDPluginDriver.cpp:912
(gdb) run
... wait for breakpoint
(gdb) thread apply all backtrace
... lots of output ...
(gdb) quit
The TCP port number can be anything. 3333 is an arbitrary choice.
This the abbreviated IOC startup output when not running as root.
*****************************
date
2022/06/27 16:03:39.620634
errlogInit(5000)
< envPaths
epicsEnvSet("IOC","iocNSLS2_IC")
epicsEnvSet("TOP","/home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4")
…
dbLoadDatabase("/home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4/dbd/quadEMTestApp.dbd")
quadEMTestApp_registerRecordDeviceDriver(pdbbase)
# The search path for database files
# Note: the separator between the path entries needs to be changed to a semicolon (;) on Windows
date
2022/06/27 16:03:39.953518
drvNSLS2_ICConfigure("NSLS2_IC", 0, 10000)
Memory mapped
Poller thread created
Creating parameters 0
Poll thread starts
Creating parameters 1
Creating parameters 2
Creating parameters 3
Creating parameters 4
Opening DACS
Getting firmware version
fpgabase=0x569a60
FPGA version=1001
Calling callbacks
date
2022/06/27 16:03:40.129864
# Create 11 statistics plugins
NDStatsConfigure("NSLS2_IC_STATS1", 20, 0, "NSLS2_IC", 0, 0, 0)
dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDStats.template", "P=quadEMTest:,R=NSLS2_IC:Current1:, PORT=NSLS2_IC_STATS1, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC,NDARRAY_ADDR=0, HIST_SIZE=256,XSIZE=10000,YSIZE=0,NCHANS=1000,ENABLED=1,")
NDTimeSeriesConfigure("NSLS2_IC_STATS1_TS", 20, 0, "NSLS2_IC_STATS1", 1, 23)
…
NDFFTConfigure("NSLS2_IC_FFT11", 20, 0, "NSLS2_IC_TS1", 10)
dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:PosY:,PORT=NSLS2_IC_FFT11,ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=10,NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=PosY")
#< $(QUADEM)/iocBoot/saveRestore.cmd
date
2022/06/27 16:03:43.046352
iocInit()
############################################################################
## EPICS R7.0.6.1
## Rev. 2022-03-25T16:17+0000
###########################
iocRun: All initialization complete
#################################################
Gain: 0
Setting bias voltage to 0.000000
date
2022/06/27 16:03:46.982984
# save settings every thirty seconds
#create_monitor_set("auto_settings.req",30,"P=$(PREFIX), R=$(RECORD)")
*****************************
The IOC starts up fine. The total time to start is about 3.5 seconds. There are many plugin threads created during this time.
This the abbreviated IOC startup output when running as root.
*****************************
date
2022/06/27 14:20:48.298214
errlogInit(5000)
< envPaths
epicsEnvSet("IOC","iocNSLS2_IC")
epicsEnvSet("TOP","/home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4")
…
dbLoadDatabase("/home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4/dbd/quadEMTestApp.dbd")
quadEMTestApp_registerRecordDeviceDriver(pdbbase)
# The search path for database files
# Note: the separator between the path entries needs to be changed to a semicolon (;) on Windows
epicsEnvSet("EPICS_DB_INCLUDE_PATH", "/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db:/home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4/db")
< /home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4/iocBoot/iocNSLS2_IC/NSLS2_IC.cmd
date
2022/06/27 14:20:48.626216
…
drvNSLS2_ICConfigure("NSLS2_IC", 0, 10000)
Memory mapped
Poller thread created
Creating parameters 0
Poll thread starts
Creating parameters 1
Creating parameters 2
Creating parameters 3
Creating parameters 4
Opening DACS
Getting firmware version
fpgabase=0x5809b8
FPGA version=1001
Calling callbacks
…
date
2022/06/27 14:20:48.802463
< /home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4/iocBoot/commonPlugins.cmd
…
# Create 11 statistics plugins
NDStatsConfigure("NSLS2_IC_STATS1", 20, 0, "NSLS2_IC", 0, 0, 0)
dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDStats.template", "P=quadEMTest:,R=NSLS2_IC:Current1:, PORT=NSLS2_IC_STATS1, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC,NDARRAY_ADDR=0, HIST_SIZE=256,XSIZE=10000,YSIZE=0,NCHANS=1000,ENABLED=1,")
…
# FFT plugins
NDFFTConfigure("NSLS2_IC_FFT1", 20, 0, "NSLS2_IC_TS1", 0)
dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:Current1:, PORT=NSLS2_IC_FFT1, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=0, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=Current1")
NDFFTConfigure("NSLS2_IC_FFT2", 20, 0, "NSLS2_IC_TS1", 1)
dbLoadRecords("/home/peter/epics/extensions/synApps/support2022/06/27 14:21:19.270 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:21:23.270 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:21:26.247 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:21:28.278 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
Starting iocInit
-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:Current2:, PORT=NSLS2_IC_FFT2, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=1, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=Current2")
NDFFTConfigure("NSLS2_IC_FFT3", 20, 0, "NSLS2_IC_TS1", 2
dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:Current3:, PORT=NSLS2_IC_FFT3, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=2, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=Current3")
NDFFTConfigure("NSLS2_IC_FFT4", 20, 0, "NSLS2_IC_TS1", 3)
dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:Current4:, PORT=NSLS2_IC_FFT4, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=3, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=Current4")
NDFFTConfigure("NSLS2_IC_FFT5", 20, 0, "NSLS2_IC_TS1", 4)
dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:SumX:, PORT=NSLS2_IC_FFT5, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=4, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=SumX")
NDFFTConfigure("NSLS2_IC_FFT6", 20, 0, "NSLS2_IC_TS1", 5)
dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:SumY:, PORT=NSLS2_IC_FFT6, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=5, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=SumY")
NDFFTConfigure("NSLS2_IC_FFT7", 20, 0, "NSLS2_IC_TS1", 6)
dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:SumAll:, PORT=NSLS2_IC_FFT7, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=6, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=SumAll")
NDFFTConfigure("NSLS2_IC_FFT8", 20, 0, "NSLS2_IC_TS1", 7)
dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:DiffX:, PORT=NSLS2_IC_FFT8, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=7, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=DiffX")
NDFFTConfigure("NSLS2_IC_FFT9", 20, 0, "NSLS2_IC_TS1", 8)
dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:DiffY:, PORT=NSLS2_IC_FFT9, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=8, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=DiffY")
NDFFTConfigure("NSLS2_IC_FFT10", 20, 0, "NSLS2_IC_TS1", 9)
dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:PosX:,PORT=NSLS2_IC_FFT10,ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=9, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=PosX")
NDFFTConfigure("NSLS2_IC_FFT11", 20, 0, "NSLS2_IC_TS1", 10)
dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:PosY:,PORT=NSLS2_IC_FFT11,ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=10,NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP MS,ENABLED=1,NAME=PosY")
#< $(QUADEM)/iocBoot/saveRestore.cmd
date
2022/06/27 14:21:29.253141
iocInit()
############################################################################
## EPICS R7.0.6.1
## Rev. 2022-03-25T16:17+0000
###########################2022/06/27 14:23:25.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:23:25.241 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
2022/06/27 14:23:32.243 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:23:32.244 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
2022/06/27 14:23:34.244 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
epicsThread::~epicsThread(): "_main_" blocking for thread "NSLS2_IC_STATS2_Plugin_1" to exit
was epicsThread object destroyed before thread exit ?
epicsThread::~epicsThread(): "_main_" blocking for thread "NSLS2_IC_STATS2_Plugin_1" to exit
was epicsThread object destroyed before thread exit ?
epicsThread::~epicsThread(): "_main_" blocking for thread "NSLS2_IC_STATS2_Plugin_1" to exit
was epicsThread object destroyed before thread exit ?
2022/06/27 14:24:05.241 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
2022/06/27 14:24:08.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:24:08.241 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
2022/06/27 14:24:08.241 quadEMTest:NSLS2_IC:Current2:TS:QueueSize devAsynInt32::processCallbackOutput process write error NDPluginTimeSeries::writeInt32: status=0, function=64, value=20
2022/06/27 14:24:10.242 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
epicsThread::~epicsThread(): "_main_" blocking for thread "NSLS2_IC_STATS2_TS_Plugin_1" to exit
was epicsThread object destroyed before thread exit ?
2022/06/27 14:24:23.241 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
2022/06/27 14:24:23.242 quadEMTest:NSLS2_IC:Current2:TS:NumThreads devAsynInt32::processCallbackOutput process write error NDPluginTimeSeries::writeInt32: status=0, function=67, value=1
2022/06/27 14:24:26.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:24:26.241 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
2022/06/27 14:24:28.242 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
2022/06/27 14:24:35.242 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
2022/06/27 14:24:51.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:24:51.241 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
2022/06/27 14:24:53.242 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
2022/06/27 14:24:54.240 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
2022/06/27 14:25:09.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:25:09.241 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
2022/06/27 14:25:11.241 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
epicsThread::~epicsThread(): "_main_" blocking for thread "NSLS2_IC_STATS7_Plugin_1" to exit
was epicsThread object destroyed before thread exit ?
2022/06/27 14:25:26.240 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
2022/06/27 14:25:31.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:25:31.242 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
2022/06/27 14:25:31.242 quadEMTest:NSLS2_IC:SumAll:TS:NumThreads devAsynInt32::processCallbackOutput process write error NDPluginTimeSeries::writeInt32: status=0, function=67, value=1
2022/06/27 14:25:34.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:25:34.242 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
2022/06/27 14:25:36.243 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
2022/06/27 14:25:40.240 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
2022/06/27 14:25:43.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:25:43.242 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
2022/06/27 14:25:43.243 quadEMTest:NSLS2_IC:DiffX:TS:QueueSize devAsynInt32::processCallbackOutput process write error NDPluginTimeSeries::writeInt32: status=0, function=64, value=20
2022/06/27 14:25:45.244 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
2022/06/27 14:25:47.240 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
2022/06/27 14:25:47.242 quadEMTest:NSLS2_IC:DiffX:TS:NumThreads devAsynInt32::processCallbackOutput process write error NDPluginTimeSeries::writeInt32: status=0, function=67, value=1
2022/06/27 14:25:50.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:25:50.242 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
2022/06/27 14:25:52.243 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
epicsThread::~epicsThread(): "_main_" blocking for thread "NSLS2_IC_STATS9_Plugin_1" to exit
was epicsThread object destroyed before thread exit ?
2022/06/27 14:26:04.240 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
2022/06/27 14:26:07.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:26:07.248 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
2022/06/27 14:26:07.249 quadEMTest:NSLS2_IC:DiffY:TS:QueueSize devAsynInt32::processCallbackOutput process write error NDPluginTimeSeries::writeInt32: status=0, function=64, value=20
2022/06/27 14:26:09.249 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
2022/06/27 14:26:13.240 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
2022/06/27 14:26:13.248 quadEMTest:NSLS2_IC:DiffY:TS:NumThreads devAsynInt32::processCallbackOutput process write error NDPluginTimeSeries::writeInt32: status=0, function=67, value=1
2022/06/27 14:26:24.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:26:24.242 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
2022/06/27 14:26:26.243 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
2022/06/27 14:26:35.240 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
2022/06/27 14:26:38.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:26:38.249 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
2022/06/27 14:26:38.249 quadEMTest:NSLS2_IC:PosY:TS:QueueSize devAsynInt32::processCallbackOutput process write error NDPluginTimeSeries::writeInt32: status=0, function=64, value=20
2022/06/27 14:26:40.250 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
epicsThread::~epicsThread(): "_main_" blocking for thread "NSLS2_IC_STATS11_TS_Plugin_1" to exit
was epicsThread object destroyed before thread exit ?
2022/06/27 14:26:52.241 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
2022/06/27 14:26:52.242 quadEMTest:NSLS2_IC:PosY:TS:NumThreads devAsynInt32::processCallbackOutput process write error NDPluginTimeSeries::writeInt32: status=0, function=67, value=1
2022/06/27 14:26:55.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:26:55.242 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
2022/06/27 14:26:55.243 quadEMTest:NSLS2_IC:image1:QueueSize devAsynInt32::processCallbackOutput process write error
2022/06/27 14:26:57.244 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
2022/06/27 14:27:01.240 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
2022/06/27 14:27:01.242 quadEMTest:NSLS2_IC:image1:NumThreads devAsynInt32::processCallbackOutput process write error
2022/06/27 14:27:15.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:27:15.248 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
2022/06/27 14:27:15.249 quadEMTest:NSLS2_IC:FFT:Current2:NumThreads devAsynInt32::processCallbackOutput process write error
2022/06/27 14:27:23.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:27:23.242 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
2022/06/27 14:27:23.243 quadEMTest:NSLS2_IC:FFT:Current4:QueueSize devAsynInt32::processCallbackOutput process write error
2022/06/27 14:27:25.244 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message
2022/06/27 14:27:28.241 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
2022/06/27 14:27:28.249 quadEMTest:NSLS2_IC:FFT:Current4:NumThreads devAsynInt32::processCallbackOutput process write error
2022/06/27 14:27:33.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:27:33.248 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
2022/06/27 14:27:33.249 quadEMTest:NSLS2_IC:FFT:SumX:NumThreads devAsynInt32::processCallbackOutput process write error
2022/06/27 14:27:38.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:27:38.249 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
2022/06/27 14:27:38.249 quadEMTest:NSLS2_IC:FFT:SumY:NumThreads devAsynInt32::processCallbackOutput process write error
2022/06/27 14:27:38.249 NDPluginDriver::deleteCallbackThreads incorrect response from plugin thread 0 = 0
2022/06/27 14:27:39.240 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
2022/06/27 14:27:39.241 quadEMTest:NSLS2_IC:FFT:SumAll:QueueSize devAsynInt32::processCallbackOutput process write error
2022/06/27 14:27:41.243 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:27:41.244 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1
2022/06/27 14:27:41.244 quadEMTest:NSLS2_IC:FFT:SumAll:NumThreads devAsynInt32::processCallbackOutput process write error
2022/06/27 14:27:45.241 NDPluginDriver::deleteCallbackThreads incorrect response from plugin thread 0 = 0
2022/06/27 14:27:46.241 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
2022/06/27 14:27:46.242 quadEMTest:NSLS2_IC:FFT:DiffY:QueueSize devAsynInt32::processCallbackOutput process write error
2022/06/27 14:27:47.241 NDPluginDriver::deleteCallbackThreads incorrect response from plugin thread 0 = 0
2022/06/27 14:27:48.241 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
2022/06/27 14:27:48.242 quadEMTest:NSLS2_IC:FFT:PosX:QueueSize devAsynInt32::processCallbackOutput process write error
2022/06/27 14:27:49.241 NDPluginDriver::deleteCallbackThreads incorrect response from plugin thread 0 = 0
2022/06/27 14:27:50.241 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1
2022/06/27 14:27:50.244 quadEMTest:NSLS2_IC:FFT:PosY:QueueSize devAsynInt32::processCallbackOutput process write error
2022/06/27 14:27:53.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:27:56.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:27:59.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:28:02.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:28:05.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:28:08.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
2022/06/27 14:28:11.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message
iocRun: All initialization complete
#################################################
Gain: 0
Setting bias voltage to 0.000000
date
2022/06/27 14:32:29.293229
# save settings every thirty seconds
#create_monitor_set("auto_settings.req",30,"P=$(PREFIX), R=$(RECORD)")
epics>
*****************************
There are many errors waiting for plugin threads to start (and then waiting for them to exit). The total time to start is about 12 minutes compared to 3.5 seconds when not running as root.
Any ideas why there is an issue creating plugin threads when running as root, but not when running as a normal user?
Thanks,
Mark
- Replies:
- Re: Timeout when running soft IOC as root Siddons, David via Tech-talk
- References:
- Timeout when running soft IOC as root Mark Rivers via Tech-talk
- Navigate by Date:
- Prev:
Re: Timeout when running soft IOC as root Timo Korhonen via Tech-talk
- Next:
Re: EPICS IOC BOOTP timeout at RTEMS on MVME6100 chengsn via Tech-talk
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
<2022>
2023
2024
- Navigate by Thread:
- Prev:
Re: Timeout when running soft IOC as root Timo Korhonen via Tech-talk
- Next:
Re: Timeout when running soft IOC as root Siddons, David via Tech-talk
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
<2022>
2023
2024
|