Hi Sandeep,
The stack trace indicates the problem is in NDPluginStats following a call to its writeInt32 method. That looks like it could be an issue that was fixed in ADCore R3-6. This is from the R3-6 release notes:
NDPluginStats
* Removed calls to processCallbacks() when CursorX, CursorY, or CentroidThreshold are changed. This is no longer thread safe. To see updates when plugin is not receiving NDArrays change the PV and then press ProcessPlugin.
Mark
________________________________
From: Tech-talk <[email protected]> on behalf of Sandeep Kumar Malu - UKRI STFC via Tech-talk <[email protected]>
Sent: Friday, September 20, 2019 5:28 AM
To: [email protected]
Subject: areadetector NDArrayPool issues
Hi,
I am running multiple prosilica cameras. Some camera crashes in 5 min while others run for 5-6 hrs. My guess is something to do with memory stuffs (alloc, dealloc, buffer size, stack size etc). I am also using adPython plugin to do image processing using openCV. The error I get is –
NDArrayPool:reserve ERROR, reference count = 0, should be >= 1, pArray=0x7f1a640010e0
Thread scan-1 (0x562a979bd2c0) can't proceed, suspending.
Dumping a stack trace of thread 'scan-1':
[ 0x7fad8bb6db13]: /usr/local/epics/base/lib/linux-x86_64/libCom.so.3.15.6(epicsStackTrace+0x73)
[ 0x7fad8bb5d4e5]: /usr/local/epics/base/lib/linux-x86_64/libCom.so.3.15.6(cantProceed+0xc5)
[ 0x7fad8b136b6f]: /usr/local/epics/support/areaDetector/current/ADCore/lib/linux-x86_64/libADBase.so(_ZN11NDArrayPool7reserveEP7NDArray+0xdf)
[ 0x7fad8b37f30f]: /usr/local/epics/support/areaDetector/current/ADCore/lib/linux-x86_64/libNDPlugin.so(_ZN14NDPluginDriver21beginProcessCallbacksEP7NDArray+0x36f)
[ 0x7fad8b3c315d]: /usr/local/epics/support/areaDetector/current/ADCore/lib/linux-x86_64/libNDPlugin.so(_ZN13NDPluginStats16processCallbacksEP7NDArray+0x3d)
[ 0x7fad8b3c4164]: /usr/local/epics/support/areaDetector/current/ADCore/lib/linux-x86_64/libNDPlugin.so(_ZN13NDPluginStats10writeInt32EP8asynUseri+0xe4)
[ 0x7fad8a844eab]: /usr/local/epics/support/asyn/asyn4-33/lib/linux-x86_64/libasyn.so(writeInt32+0x3b)
[ 0x7fad8a856085]: /usr/local/epics/support/asyn/asyn4-33/lib/linux-x86_64/libasyn.so(processCallbackOutput+0x25)
[ 0x7fad8a823995]: /usr/local/epics/support/asyn/asyn4-33/lib/linux-x86_64/libasyn.so(queueRequest+0x285)
[ 0x7fad8a8566d9]: /usr/local/epics/support/asyn/asyn4-33/lib/linux-x86_64/libasyn.so(processLo+0x59)
[ 0x7fad8c03fd38]: /usr/local/epics/base/lib/linux-x86_64/libdbRecStd.so.3.15.6(process+0xd8)
[ 0x7fad8bdba34f]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(dbProcess+0x23f)
[ 0x7fad8bdc8efb]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(dbPutLink+0x12b)
[ 0x7fad8c0379d8]: /usr/local/epics/base/lib/linux-x86_64/libdbRecStd.so.3.15.6(execOutput+0x88)
[ 0x7fad8c0380bf]: /usr/local/epics/base/lib/linux-x86_64/libdbRecStd.so.3.15.6(process+0x5ef)
[ 0x7fad8bdba34f]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(dbProcess+0x23f)
[ 0x7fad8bdd5bb0]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(recGblFwdLink+0x10)
[ 0x7fad8c037db8]: /usr/local/epics/base/lib/linux-x86_64/libdbRecStd.so.3.15.6(process+0x2e8)
[ 0x7fad8bdba34f]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(dbProcess+0x23f)
[ 0x7fad8bdcac14]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(scanList+0x64)
[ 0x7fad8bdcafd8]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(periodicTask+0x1a8)
[ 0x7fad8bb67dc8]: /usr/local/epics/base/lib/linux-x86_64/libCom.so.3.15.6(start_routine+0xf8)
[ 0x7fad894086db]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb)
[ 0x7fad89d4988f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)
Thread scan-1 (0x55f358edc150) suspended
The version of EPICS and support modules are
a) base – 3.15.6
b) asyn – 4-33
c) calc – 3-6-1
d) seq – 2.1.21
e) streamDevice – 2-6
f) busy – 1-6
g) areadetector – 3-5
1. ADcore – 3-5
2. ADsupport – 1-9
3. ADprosilica – 2-5
h) Adpython – 1-4 (https://epics.anl.gov/tech-talk/2019/msg00588.php)
The st.cmd file looks like
# Prosilica driver
#########################
prosilicaConfig("$(PREFIX)CAM", $(CAMID), 50, 0)
dbLoadRecords("$(ADPROSILICA)/db/prosilica.template","P=$(PREFIX),R=cam1:,PORT=$(PREFIX)CAM,ADDR=0,TIMEOUT=1")
dbLoadRecords("ADBase.template","P=$(PREFIX),R=cam1:,PORT=$(PREFIX)CAM,ADDR=0,TIMEOUT=1")
dbLoadRecords("NDFile.template","P=$(PREFIX),R=cam1:,PORT=$(PREFIX)CAM,ADDR=0,TIMEOUT=1")
# Create ROI plugins
###################################
NDROIConfigure("$(PREFIX)ROI", $(QSIZE), 0, "$(PREFIX)CAM", 0, 0, 0, 0, 0, $(MAX_THREADS=5))
dbLoadRecords("NDROI.template", "P=$(PREFIX),R=ROI1:, PORT=$(PREFIX)ROI,ADDR=0,TIMEOUT=1,NDARRAY_PORT=$(PREFIX)CAM")
# Create a transform plugin
###################################
NDTransformConfigure("$(PREFIX)TRANS", $(QSIZE), 0, "$(PREFIX)ROI", 0, 0, 0, 0, 0, $(MAX_THREADS=5))
dbLoadRecords("NDTransform.template", "P=$(PREFIX),R=Trans1:, PORT=$(PREFIX)TRANS,ADDR=0,TIMEOUT=1,NDARRAY_PORT=$(PREFIX)ROI")
# adPython plugin for centroid
##############################
adPythonPluginConfigure("$(PREFIX)CENTROID", "$(TOP)/pythonScripts/adPythonCentroid.py", "Template", 10, 0, "$(PREFIX)TRANS", 0, 100, 50,1,200097152)
dbLoadRecords("NDPluginBase.template","P=$(PREFIX),R=Centroid1:, PORT=$(PREFIX)CENTROID,ADDR=0,TIMEOUT=1,NDARRAY_PORT=$(PREFIX)TRANS,NDARRAY_ADDR=0")
dbLoadRecords("$(ADPYTHON)/db/adPythonPlugin.template","P=$(PREFIX),R=Centroid1:, PORT=$(PREFIX)CENTROID, TIMEOUT=1,NDARRAY_PORT=$(PREFIX)TRANS")
# standard arrays plugin
###############################
NDStdArraysConfigure("$(PREFIX)Image", 3, 0, "$(PREFIX)TRANS", 0, 0)
dbLoadRecords("$(ADCORE)/db/NDStdArrays.template","P=$(PREFIX),R=image1:,PORT=$(PREFIX)Image,ADDR=0,TIMEOUT=1,NDARRAY_PORT=$(PREFIX)OVER,TYPE=Int8,FTVL=UCHAR,NELEMENTS=1245488)")
Any way to fix this?
Many Thanks
Sandeep Malu
- References:
- areadetector NDArrayPool issues Sandeep Kumar Malu - UKRI STFC via Tech-talk
- Navigate by Date:
- Prev:
areadetector NDArrayPool issues Sandeep Kumar Malu - UKRI STFC via Tech-talk
- Next:
Re: AW: AW: Correct Settings for PIE816 Controller Peterson, Kevin M. via Tech-talk
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
<2019>
2020
2021
2022
2023
2024
- Navigate by Thread:
- Prev:
areadetector NDArrayPool issues Sandeep Kumar Malu - UKRI STFC via Tech-talk
- Next:
Fw: areadetector NDArrayPool issues Mark Rivers via Tech-talk
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
<2019>
2020
2021
2022
2023
2024
|