Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020 
<== Date ==> <== Thread ==>

Subject: Fw: areadetector NDArrayPool issues
From: Mark Rivers via Tech-talk <tech-talk@aps.anl.gov>
To: tech-talk <tech-talk@aps.anl.gov>
Date: Mon, 23 Sep 2019 12:49:28 +0000
Forwarding to tech-talk, issue resolved.


________________________________
From: Sandeep Kumar Malu - UKRI STFC <sandeep-kumar.malu@stfc.ac.uk>
Sent: Monday, September 23, 2019 3:04 AM
To: Mark Rivers
Subject: RE: areadetector NDArrayPool issues

Hi Mark

Thank you for pointing this out. Indeed, no more error messages and camera is running for a long time without crashing.

Many Thanks
Sandeep Malu


-----Original Message-----
From: Mark Rivers <rivers@cars.uchicago.edu>
Sent: 20 September 2019 13:15
To: Malu, Sandeep Kumar (-,RAL,CLF) <sandeep-kumar.malu@stfc.ac.uk>
Subject: Re: areadetector NDArrayPool issues


Hi Sandeep,

The stack trace indicates the problem is in NDPluginStats following a call to its writeInt32 method.  That looks like it could be an issue that was fixed in ADCore R3-6.  This is from the R3-6 release notes:

NDPluginStats

  *   Removed calls to processCallbacks() when CursorX, CursorY, or CentroidThreshold are changed. This is no longer thread safe. To see updates when plugin is not receiving NDArrays change the PV and then press ProcessPlugin.

Mark


________________________________
From: Tech-talk <tech-talk-bounces@aps.anl.gov> on behalf of Sandeep Kumar Malu - UKRI STFC via Tech-talk <tech-talk@aps.anl.gov>
Sent: Friday, September 20, 2019 5:28 AM
To: tech-talk@aps.anl.gov
Subject: areadetector NDArrayPool issues


Hi,



I am running multiple prosilica cameras. Some camera crashes in 5 min while others run for 5-6 hrs. My guess is something to do with memory stuffs (alloc, dealloc, buffer size, stack size etc).  I am also using adPython plugin to do image processing using openCV. The error I get is -



NDArrayPool:reserve ERROR, reference count = 0, should be >= 1, pArray=0x7f1a640010e0

Thread scan-1 (0x562a979bd2c0) can't proceed, suspending.

Dumping a stack trace of thread 'scan-1':

[    0x7fad8bb6db13]: /usr/local/epics/base/lib/linux-x86_64/libCom.so.3.15.6(epicsStackTrace+0x73)

[    0x7fad8bb5d4e5]: /usr/local/epics/base/lib/linux-x86_64/libCom.so.3.15.6(cantProceed+0xc5)

[    0x7fad8b136b6f]: /usr/local/epics/support/areaDetector/current/ADCore/lib/linux-x86_64/libADBase.so(_ZN11NDArrayPool7reserveEP7NDArray+0xdf)

[    0x7fad8b37f30f]: /usr/local/epics/support/areaDetector/current/ADCore/lib/linux-x86_64/libNDPlugin.so(_ZN14NDPluginDriver21beginProcessCallbacksEP7NDArray+0x36f)

[    0x7fad8b3c315d]: /usr/local/epics/support/areaDetector/current/ADCore/lib/linux-x86_64/libNDPlugin.so(_ZN13NDPluginStats16processCallbacksEP7NDArray+0x3d)

[    0x7fad8b3c4164]: /usr/local/epics/support/areaDetector/current/ADCore/lib/linux-x86_64/libNDPlugin.so(_ZN13NDPluginStats10writeInt32EP8asynUseri+0xe4)

[    0x7fad8a844eab]: /usr/local/epics/support/asyn/asyn4-33/lib/linux-x86_64/libasyn.so(writeInt32+0x3b)

[    0x7fad8a856085]: /usr/local/epics/support/asyn/asyn4-33/lib/linux-x86_64/libasyn.so(processCallbackOutput+0x25)

[    0x7fad8a823995]: /usr/local/epics/support/asyn/asyn4-33/lib/linux-x86_64/libasyn.so(queueRequest+0x285)

[    0x7fad8a8566d9]: /usr/local/epics/support/asyn/asyn4-33/lib/linux-x86_64/libasyn.so(processLo+0x59)

[    0x7fad8c03fd38]: /usr/local/epics/base/lib/linux-x86_64/libdbRecStd.so.3.15.6(process+0xd8)

[    0x7fad8bdba34f]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(dbProcess+0x23f)

[    0x7fad8bdc8efb]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(dbPutLink+0x12b)

[    0x7fad8c0379d8]: /usr/local/epics/base/lib/linux-x86_64/libdbRecStd.so.3.15.6(execOutput+0x88)

[    0x7fad8c0380bf]: /usr/local/epics/base/lib/linux-x86_64/libdbRecStd.so.3.15.6(process+0x5ef)

[    0x7fad8bdba34f]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(dbProcess+0x23f)

[    0x7fad8bdd5bb0]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(recGblFwdLink+0x10)

[    0x7fad8c037db8]: /usr/local/epics/base/lib/linux-x86_64/libdbRecStd.so.3.15.6(process+0x2e8)

[    0x7fad8bdba34f]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(dbProcess+0x23f)

[    0x7fad8bdcac14]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(scanList+0x64)

[    0x7fad8bdcafd8]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(periodicTask+0x1a8)

[    0x7fad8bb67dc8]: /usr/local/epics/base/lib/linux-x86_64/libCom.so.3.15.6(start_routine+0xf8)

[    0x7fad894086db]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb)

[    0x7fad89d4988f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)

Thread scan-1 (0x55f358edc150) suspended



The version of EPICS and support modules are

a)       base - 3.15.6

b)      asyn - 4-33

c)       calc - 3-6-1

d)      seq - 2.1.21

e)      streamDevice - 2-6

f)        busy - 1-6

g)       areadetector - 3-5

1.       ADcore - 3-5

2.       ADsupport - 1-9

3.       ADprosilica - 2-5

h)      Adpython - 1-4 (https://epics.anl.gov/tech-talk/2019/msg00588.php)



The st.cmd file looks like



#   Prosilica driver

#########################

prosilicaConfig("$(PREFIX)CAM", $(CAMID), 50, 0)

dbLoadRecords("$(ADPROSILICA)/db/prosilica.template","P=$(PREFIX),R=cam1:,PORT=$(PREFIX)CAM,ADDR=0,TIMEOUT=1")

dbLoadRecords("ADBase.template","P=$(PREFIX),R=cam1:,PORT=$(PREFIX)CAM,ADDR=0,TIMEOUT=1")

dbLoadRecords("NDFile.template","P=$(PREFIX),R=cam1:,PORT=$(PREFIX)CAM,ADDR=0,TIMEOUT=1")



# Create ROI plugins

###################################

NDROIConfigure("$(PREFIX)ROI", $(QSIZE), 0, "$(PREFIX)CAM", 0, 0, 0, 0, 0, $(MAX_THREADS=5))

dbLoadRecords("NDROI.template", "P=$(PREFIX),R=ROI1:,  PORT=$(PREFIX)ROI,ADDR=0,TIMEOUT=1,NDARRAY_PORT=$(PREFIX)CAM")



# Create a transform plugin

###################################

NDTransformConfigure("$(PREFIX)TRANS", $(QSIZE), 0, "$(PREFIX)ROI", 0, 0, 0, 0, 0, $(MAX_THREADS=5))

dbLoadRecords("NDTransform.template", "P=$(PREFIX),R=Trans1:,  PORT=$(PREFIX)TRANS,ADDR=0,TIMEOUT=1,NDARRAY_PORT=$(PREFIX)ROI")



# adPython plugin for centroid

##############################

adPythonPluginConfigure("$(PREFIX)CENTROID", "$(TOP)/pythonScripts/adPythonCentroid.py", "Template", 10, 0, "$(PREFIX)TRANS", 0, 100, 50,1,200097152)

dbLoadRecords("NDPluginBase.template","P=$(PREFIX),R=Centroid1:, PORT=$(PREFIX)CENTROID,ADDR=0,TIMEOUT=1,NDARRAY_PORT=$(PREFIX)TRANS,NDARRAY_ADDR=0")

dbLoadRecords("$(ADPYTHON)/db/adPythonPlugin.template","P=$(PREFIX),R=Centroid1:, PORT=$(PREFIX)CENTROID, TIMEOUT=1,NDARRAY_PORT=$(PREFIX)TRANS")



# standard arrays plugin

###############################

NDStdArraysConfigure("$(PREFIX)Image", 3, 0, "$(PREFIX)TRANS", 0, 0)

dbLoadRecords("$(ADCORE)/db/NDStdArrays.template","P=$(PREFIX),R=image1:,PORT=$(PREFIX)Image,ADDR=0,TIMEOUT=1,NDARRAY_PORT=$(PREFIX)OVER,TYPE=Int8,FTVL=UCHAR,NELEMENTS=1245488)")



Any way to fix this?



Many Thanks

Sandeep Malu

References:
areadetector NDArrayPool issues Sandeep Kumar Malu - UKRI STFC via Tech-talk

Navigate by Date:
Prev: Base R7.0.2.2 VME_IO Parsing Graham Cox - UKRI STFC via Tech-talk
Next: Re: Base R7.0.2.2 VME_IO Parsing Michael Davidsaver via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020 
Navigate by Thread:
Prev: Re: areadetector NDArrayPool issues Mark Rivers via Tech-talk
Next: Base R7.0.2.2 VME_IO Parsing Graham Cox - UKRI STFC via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020 
ANJ, 23 Sep 2019 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·