Experimental Physics and
Industrial Control System

Hi,

 

I am running multiple prosilica cameras. Some camera crashes in 5 min while others run for 5-6 hrs. My guess is something to do with memory stuffs (alloc, dealloc, buffer size, stack size etc).  I am also using adPython plugin to do image processing using openCV. The error I get is –

 

NDArrayPool:reserve ERROR, reference count = 0, should be >= 1, pArray=0x7f1a640010e0

Thread scan-1 (0x562a979bd2c0) can't proceed, suspending.

Dumping a stack trace of thread 'scan-1':

[    0x7fad8bb6db13]: /usr/local/epics/base/lib/linux-x86_64/libCom.so.3.15.6(epicsStackTrace+0x73)

[    0x7fad8bb5d4e5]: /usr/local/epics/base/lib/linux-x86_64/libCom.so.3.15.6(cantProceed+0xc5)

[    0x7fad8b136b6f]: /usr/local/epics/support/areaDetector/current/ADCore/lib/linux-x86_64/libADBase.so(_ZN11NDArrayPool7reserveEP7NDArray+0xdf)

[    0x7fad8b37f30f]: /usr/local/epics/support/areaDetector/current/ADCore/lib/linux-x86_64/libNDPlugin.so(_ZN14NDPluginDriver21beginProcessCallbacksEP7NDArray+0x36f)

[    0x7fad8b3c315d]: /usr/local/epics/support/areaDetector/current/ADCore/lib/linux-x86_64/libNDPlugin.so(_ZN13NDPluginStats16processCallbacksEP7NDArray+0x3d)

[    0x7fad8b3c4164]: /usr/local/epics/support/areaDetector/current/ADCore/lib/linux-x86_64/libNDPlugin.so(_ZN13NDPluginStats10writeInt32EP8asynUseri+0xe4)

[    0x7fad8a844eab]: /usr/local/epics/support/asyn/asyn4-33/lib/linux-x86_64/libasyn.so(writeInt32+0x3b)

[    0x7fad8a856085]: /usr/local/epics/support/asyn/asyn4-33/lib/linux-x86_64/libasyn.so(processCallbackOutput+0x25)

[    0x7fad8a823995]: /usr/local/epics/support/asyn/asyn4-33/lib/linux-x86_64/libasyn.so(queueRequest+0x285)

[    0x7fad8a8566d9]: /usr/local/epics/support/asyn/asyn4-33/lib/linux-x86_64/libasyn.so(processLo+0x59)

[    0x7fad8c03fd38]: /usr/local/epics/base/lib/linux-x86_64/libdbRecStd.so.3.15.6(process+0xd8)

[    0x7fad8bdba34f]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(dbProcess+0x23f)

[    0x7fad8bdc8efb]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(dbPutLink+0x12b)

[    0x7fad8c0379d8]: /usr/local/epics/base/lib/linux-x86_64/libdbRecStd.so.3.15.6(execOutput+0x88)

[    0x7fad8c0380bf]: /usr/local/epics/base/lib/linux-x86_64/libdbRecStd.so.3.15.6(process+0x5ef)

[    0x7fad8bdba34f]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(dbProcess+0x23f)

[    0x7fad8bdd5bb0]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(recGblFwdLink+0x10)

[    0x7fad8c037db8]: /usr/local/epics/base/lib/linux-x86_64/libdbRecStd.so.3.15.6(process+0x2e8)

[    0x7fad8bdba34f]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(dbProcess+0x23f)

[    0x7fad8bdcac14]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(scanList+0x64)

[    0x7fad8bdcafd8]: /usr/local/epics/base/lib/linux-x86_64/libdbCore.so.3.15.6(periodicTask+0x1a8)

[    0x7fad8bb67dc8]: /usr/local/epics/base/lib/linux-x86_64/libCom.so.3.15.6(start_routine+0xf8)

[    0x7fad894086db]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb)

[    0x7fad89d4988f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)

Thread scan-1 (0x55f358edc150) suspended

 

The version of EPICS and support modules are

a)       base – 3.15.6

b)      asyn – 4-33

c)       calc – 3-6-1

d)      seq – 2.1.21

e)      streamDevice – 2-6

f)        busy – 1-6

g)       areadetector – 3-5

1.       ADcore – 3-5

2.       ADsupport – 1-9

3.       ADprosilica – 2-5

h)      Adpython – 1-4 (https://epics.anl.gov/tech-talk/2019/msg00588.php)

 

The st.cmd file looks like

 

#   Prosilica driver

#########################

prosilicaConfig("$(PREFIX)CAM", $(CAMID), 50, 0)

dbLoadRecords("$(ADPROSILICA)/db/prosilica.template","P=$(PREFIX),R=cam1:,PORT=$(PREFIX)CAM,ADDR=0,TIMEOUT=1")  

dbLoadRecords("ADBase.template","P=$(PREFIX),R=cam1:,PORT=$(PREFIX)CAM,ADDR=0,TIMEOUT=1")

dbLoadRecords("NDFile.template","P=$(PREFIX),R=cam1:,PORT=$(PREFIX)CAM,ADDR=0,TIMEOUT=1")

 

# Create ROI plugins

###################################

NDROIConfigure("$(PREFIX)ROI", $(QSIZE), 0, "$(PREFIX)CAM", 0, 0, 0, 0, 0, $(MAX_THREADS=5))

dbLoadRecords("NDROI.template", "P=$(PREFIX),R=ROI1:,  PORT=$(PREFIX)ROI,ADDR=0,TIMEOUT=1,NDARRAY_PORT=$(PREFIX)CAM")

 

# Create a transform plugin

###################################

NDTransformConfigure("$(PREFIX)TRANS", $(QSIZE), 0, "$(PREFIX)ROI", 0, 0, 0, 0, 0, $(MAX_THREADS=5))

dbLoadRecords("NDTransform.template", "P=$(PREFIX),R=Trans1:,  PORT=$(PREFIX)TRANS,ADDR=0,TIMEOUT=1,NDARRAY_PORT=$(PREFIX)ROI")

 

# adPython plugin for centroid

##############################

adPythonPluginConfigure("$(PREFIX)CENTROID", "$(TOP)/pythonScripts/adPythonCentroid.py", "Template", 10, 0, "$(PREFIX)TRANS", 0, 100, 50,1,200097152)

dbLoadRecords("NDPluginBase.template","P=$(PREFIX),R=Centroid1:, PORT=$(PREFIX)CENTROID,ADDR=0,TIMEOUT=1,NDARRAY_PORT=$(PREFIX)TRANS,NDARRAY_ADDR=0")

dbLoadRecords("$(ADPYTHON)/db/adPythonPlugin.template","P=$(PREFIX),R=Centroid1:, PORT=$(PREFIX)CENTROID, TIMEOUT=1,NDARRAY_PORT=$(PREFIX)TRANS")

 

# standard arrays plugin

###############################

NDStdArraysConfigure("$(PREFIX)Image", 3, 0, "$(PREFIX)TRANS", 0, 0)

dbLoadRecords("$(ADCORE)/db/NDStdArrays.template","P=$(PREFIX),R=image1:,PORT=$(PREFIX)Image,ADDR=0,TIMEOUT=1,NDARRAY_PORT=$(PREFIX)OVER,TYPE=Int8,FTVL=UCHAR,NELEMENTS=1245488)")

 

Any way to fix this?

 

Many Thanks

Sandeep Malu