Experimental Physics and
Industrial Control System

Mark Rivers via Tech-talk <[email protected]> · Fri, 20 Sep 2019 12:44:06 +0000

Hi Sandeep,

The stack trace indicates the problem is in NDPluginStats following a call to its writeInt32 method.  That looks like it could be an issue that was fixed in ADCore R3-6.  This is from the R3-6 release notes:

NDPluginStats

  *   Removed calls to processCallbacks() when CursorX, CursorY, or CentroidThreshold are changed. This is no longer thread safe. To see updates when plugin is not receiving NDArrays change the PV and then press ProcessPlugin.

Mark

________________________________
From: Tech-talk <[email protected]> on behalf of Sandeep Kumar Malu - UKRI STFC via Tech-talk <[email protected]>
Sent: Friday, September 20, 2019 5:28 AM
To: [email protected]
Subject: areadetector NDArrayPool issues

Hi,

I am running multiple prosilica cameras. Some camera crashes in 5 min while others run for 5-6 hrs. My guess is something to do with memory stuffs (alloc, dealloc, buffer size, stack size etc).  I am also using adPython plugin to do image processing using openCV. The error I get is –

NDArrayPool:reserve ERROR, reference count = 0, should be >= 1, pArray=0x7f1a640010e0

Thread scan-1 (0x562a979bd2c0) can't proceed, suspending.

Dumping a stack trace of thread 'scan-1':

[    0x7fad8bb6db13]: /usr/local/epics/base/lib/linux-x86_64/libCom.so.3.15.6(epicsStackTrace+0x73)

[    0x7fad8bb5d4e5]: /usr/local/epics/base/lib/linux-x86_64/libCom.so.3.15.6(cantProceed+0xc5)

[    0x7fad8b136b6f]: /usr/local/epics/support/areaDetector/current/ADCore/lib/linux-x86_64/libADBase.so(_ZN11NDArrayPool7reserveEP7NDArray+0xdf)

[    0x7fad8b37f30f]: /usr/local/epics/support/areaDetector/current/ADCore/lib/linux-x86_64/libNDPlugin.so(_ZN14NDPluginDriver21beginProcessCallbacksEP7NDArray+0x36f)

[    0x7fad8b3c315d]: /usr/local/epics/support/areaDetector/current/ADCore/lib/linux-x86_64/libNDPlugin.so(_ZN13NDPluginStats16processCallbacksEP7NDArray+0x3d)

[    0x7fad8b3c4164]: /usr/local/epics/support/areaDetector/current/ADCore/lib/linux-x86_64/libNDPlugin.so(_ZN13NDPluginStats10writeInt32EP8asynUseri+0xe4)

[    0x7fad8a844eab]: /usr/local/epics/support/asyn/asyn4-33/lib/linux-x86_64/libasyn.so(writeInt32+0x3b)

[    0x7fad8a856085]: /usr/local/epics/support/asyn/asyn4-33/lib/linux-x86_64/libasyn.so(processCallbackOutput+0x25)

[    0x7fad8a823995]: /usr/local/epics/support/asyn/asyn4-33/lib/linux-x86_64/libasyn.so(queueRequest+0x285)

[    0x7fad8a8566d9]: /usr/local/epics/support/asyn/asyn4-33/lib/linux-x86_64/libasyn.so(processLo+0x59)

[    0x7fad8c03fd38]: /usr/local/epics/base/lib/linux-x86_64/libdbRecStd.so.3.15.6(process+0xd8)

[    0x7fad8bdba34f]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(dbProcess+0x23f)

[    0x7fad8bdc8efb]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(dbPutLink+0x12b)

[    0x7fad8c0379d8]: /usr/local/epics/base/lib/linux-x86_64/libdbRecStd.so.3.15.6(execOutput+0x88)

[    0x7fad8c0380bf]: /usr/local/epics/base/lib/linux-x86_64/libdbRecStd.so.3.15.6(process+0x5ef)

[    0x7fad8bdba34f]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(dbProcess+0x23f)

[    0x7fad8bdd5bb0]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(recGblFwdLink+0x10)

[    0x7fad8c037db8]: /usr/local/epics/base/lib/linux-x86_64/libdbRecStd.so.3.15.6(process+0x2e8)

[    0x7fad8bdba34f]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(dbProcess+0x23f)

[    0x7fad8bdcac14]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(scanList+0x64)

[    0x7fad8bdcafd8]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(periodicTask+0x1a8)

[    0x7fad8bb67dc8]: /usr/local/epics/base/lib/linux-x86_64/libCom.so.3.15.6(start_routine+0xf8)

[    0x7fad894086db]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb)

[    0x7fad89d4988f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)

Thread scan-1 (0x55f358edc150) suspended

The version of EPICS and support modules are

a)       base – 3.15.6

b)      asyn – 4-33

c)       calc – 3-6-1

d)      seq – 2.1.21

e)      streamDevice – 2-6

f)        busy – 1-6

g)       areadetector – 3-5

1.       ADcore – 3-5

2.       ADsupport – 1-9

3.       ADprosilica – 2-5

h)      Adpython – 1-4 (https://epics.anl.gov/tech-talk/2019/msg00588.php)

The st.cmd file looks like

#   Prosilica driver

#########################

prosilicaConfig("$(PREFIX)CAM", $(CAMID), 50, 0)

dbLoadRecords("$(ADPROSILICA)/db/prosilica.template","P=$(PREFIX),R=cam1:,PORT=$(PREFIX)CAM,ADDR=0,TIMEOUT=1")

dbLoadRecords("ADBase.template","P=$(PREFIX),R=cam1:,PORT=$(PREFIX)CAM,ADDR=0,TIMEOUT=1")

dbLoadRecords("NDFile.template","P=$(PREFIX),R=cam1:,PORT=$(PREFIX)CAM,ADDR=0,TIMEOUT=1")

# Create ROI plugins

###################################

NDROIConfigure("$(PREFIX)ROI", $(QSIZE), 0, "$(PREFIX)CAM", 0, 0, 0, 0, 0, $(MAX_THREADS=5))

dbLoadRecords("NDROI.template", "P=$(PREFIX),R=ROI1:,  PORT=$(PREFIX)ROI,ADDR=0,TIMEOUT=1,NDARRAY_PORT=$(PREFIX)CAM")

# Create a transform plugin

###################################

NDTransformConfigure("$(PREFIX)TRANS", $(QSIZE), 0, "$(PREFIX)ROI", 0, 0, 0, 0, 0, $(MAX_THREADS=5))

dbLoadRecords("NDTransform.template", "P=$(PREFIX),R=Trans1:,  PORT=$(PREFIX)TRANS,ADDR=0,TIMEOUT=1,NDARRAY_PORT=$(PREFIX)ROI")

# adPython plugin for centroid

##############################

adPythonPluginConfigure("$(PREFIX)CENTROID", "$(TOP)/pythonScripts/adPythonCentroid.py", "Template", 10, 0, "$(PREFIX)TRANS", 0, 100, 50,1,200097152)

dbLoadRecords("NDPluginBase.template","P=$(PREFIX),R=Centroid1:, PORT=$(PREFIX)CENTROID,ADDR=0,TIMEOUT=1,NDARRAY_PORT=$(PREFIX)TRANS,NDARRAY_ADDR=0")

dbLoadRecords("$(ADPYTHON)/db/adPythonPlugin.template","P=$(PREFIX),R=Centroid1:, PORT=$(PREFIX)CENTROID, TIMEOUT=1,NDARRAY_PORT=$(PREFIX)TRANS")

# standard arrays plugin

###############################

NDStdArraysConfigure("$(PREFIX)Image", 3, 0, "$(PREFIX)TRANS", 0, 0)

dbLoadRecords("$(ADCORE)/db/NDStdArrays.template","P=$(PREFIX),R=image1:,PORT=$(PREFIX)Image,ADDR=0,TIMEOUT=1,NDARRAY_PORT=$(PREFIX)OVER,TYPE=Int8,FTVL=UCHAR,NELEMENTS=1245488)")

Any way to fix this?

Many Thanks

Sandeep Malu

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System