Re: ADProsilica crashing on RHEL7 3.10.0-1127

["Dunning, Michael via Tech-talk" <tech-talk at aps.anl.gov>]

I've been testing ADVimba on both rhel6 and rhel7 and it seems to be working well.  Thanks a lot for your work on this module Mark.

I'm trying to use this module with a static IOC build (i.e. setting SHARED_LIBRARIES=NO and STATIC_BUILD=YES).

We build all IOCs in this way and then copy them over to and run them on a production server with the same architecture, which doesn't have access to our development fileserver.

As you know, for linux the vendor support has two parts:
vimbaSupportC  --> No source code, no archive (.a) libraries, only shared object (.so) libraries
vimbaSupportCPP  --> Built by ADVimba from vendor source code

And the libs section of the IOC Makefile looks like this:

...

# Add locally compiled object code

PROD_LIBS += ADVimba

PROD_LIBS += ADGenICam

PROD_LIBS_WIN32 += VimbaCPP VimbaC VimbaImageTransform

PROD_LIBS_Linux += VimbaCPP

PROD_SYS_LIBS_Linux += VimbaC VimbaImageTransform

...

If I build my 'static' IOC, I get a binary which needs the .so libraries:

 $ ldd bin/rhel7-x86_64/vimbaApp
linux-vdso.so.1 =>  (0x00007ffe985dc000)
libVimbaC.so => /afs/slac.stanford.edu/g/spear/vol8/epics/modules/areaDetector/R3.9/ADVimba/lib/rhel7-x86_64/libVimbaC.so (0x00007ff613966000)
libVimbaImageTransform.so => /afs/slac.stanford.edu/g/spear/vol8/epics/modules/areaDetector/R3.9/ADVimba/lib/rhel7-x86_64/libVimbaImageTransform.so (0x00007ff61366c000)
libX11.so.6 => /lib64/libX11.so.6 (0x00007ff61332e000)
libXext.so.6 => /lib64/libXext.so.6 (0x00007ff61311c000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff612f00000)
libreadline.so.6 => /lib64/libreadline.so.6 (0x00007ff612cba000)
libncurses.so.5 => /lib64/libncurses.so.5 (0x00007ff612a93000)
libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007ff612869000)
librt.so.1 => /lib64/librt.so.1 (0x00007ff612661000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007ff61245d000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007ff612156000)
libm.so.6 => /lib64/libm.so.6 (0x00007ff611e54000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ff611c3e000)
libc.so.6 => /lib64/libc.so.6 (0x00007ff611870000)
libxcb.so.1 => /lib64/libxcb.so.1 (0x00007ff611648000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff613c3a000)
libXau.so.6 => /lib64/libXau.so.6 (0x00007ff611444000)

$ objdump -p bin/rhel7-x86_64/vimbaApp | grep NEEDED
  NEEDED               libVimbaC.so
  NEEDED               libVimbaImageTransform.so
  NEEDED               libX11.so.6
  NEEDED               libXext.so.6
  NEEDED               libpthread.so.0
  NEEDED               libreadline.so.6
  NEEDED               libncurses.so.5
  NEEDED               libtinfo.so.5
  NEEDED               librt.so.1
  NEEDED               libdl.so.2
  NEEDED               libstdc++.so.6
  NEEDED               libm.so.6
  NEEDED               libgcc_s.so.1
  NEEDED               libc.so.6

So it's not a static build, but I guess this is expected.  I'm guessing that until the vendor releases source code or archive libraries, that there's just no way around this?

I suppose one option is to drop the .so libraries in a directory whose path exists on both the development and production servers, e.g. /usr/local/lib/ or somewhere like that?

Is there a better way of dealing with this in the EPICS build system, or some other clever trick?

I guess this could be viewed as a more general question of how to do a static IOC build when you only have shared object libraries.

Thanks,

Mike

---

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Pearson, Matthew R. via Tech-talk <tech-talk at aps.anl.gov>
Sent: Tuesday, July 28, 2020 3:06 PM
To: Mark Rivers <rivers at cars.uchicago.edu>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: RE: ADProsilica crashing on RHEL7 3.10.0-1127

 

Hi,

Quick update on this. A power cycle on both cameras fixed both issues.

However, we will plan to move to ADVimba regardless.

Cheers,
Matt

-----Original Message-----
From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Pearson, Matthew R. via Tech-talk
Sent: Friday, July 24, 2020 10:07 AM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: tech-talk at aps.anl.gov
Subject: [EXTERNAL] RE: ADProsilica crashing on RHEL7 3.10.0-1127

Hi Mark & Mike,

The camera is reachable. I tried a non-existent camera, and the IOC starts up normally with the usual message:

prosilicaConfig("AVT.CAM", 10.111.24.99, 100, 0, 0, 0, 10)
2020/07/24 09:59:13.252 prosilica:connectCamera: Cannot find camera 10.111.24.99
prosilica:prosilica: cannot connect to camera 10.111.24.99, manually connect when available.

Also thanks Mike for the info. I also see it affecting a sub-set of our cameras. The two affected cameras are one beamline, and we have a working camera on a different beamline (same Linux versions though). Out of the 2 affected cameras, one crashes with the same seg fault, and the other acts differently:

prosilicaConfig("AVT.CAM", 10.111.24.11, 100, 0, 0, 0, 10)
2020/07/24 10:03:18.670 prosilica:connectCamera: unable to get sensor data on camera 5122144
prosilica:prosilica: cannot connect to camera 10.111.24.11, manually connect when available.

I will be able to head in to the beamline on Monday to power cycle the cameras. We have more than a week before our facility starts up again, so we may try switching to ADVimba.

Cheers,
Matt


-----Original Message-----
From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Thursday, July 23, 2020 7:49 PM
To: Pearson, Matthew R. <pearsonmr at ornl.gov>
Cc: tech-talk at aps.anl.gov
Subject: [EXTERNAL] Re: ADProsilica crashing on RHEL7 3.10.0-1127

Hi Matt,


Is that camera reachable?  Does it crash if you try to connect to a non-existent camera?


I am updating one of our Linux machines to that version of Centos 7.  I can see if I have the problem.  All of our production AVT/Prosilica cameras are now running ADVimba, but I can try a test with ADProsilica.


Mark



________________________________
From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Pearson, Matthew R. via Tech-talk <tech-talk at aps.anl.gov>
Sent: Thursday, July 23, 2020 4:41 PM
To: tech-talk at aps.anl.gov
Subject: ADProsilica crashing on RHEL7 3.10.0-1127


Hi,



We've been using ADProsilica to control our AVT Manta cameras for many years. This uses the legacy PvApi library from AVT. It seems to be failing under our latest RHEL7 update (3.10.0-1127.13.1.el7.x86_64).



I've pasted the stack trace below. The problem seems to be deep inside libPvAPI.a, which is linked into libprosilica.so. libPvAPI.a was provided by AVT, but the last build was in 2015.



Anyone else seen this issue?



I know there is a newer areaDetector driver for AVT cameras, so it may be the case that we just have to switch to using that.



Cheers,

Matt



Program received signal SIGSEGV, Segmentation fault.

0x00007ffff7b61a7a in PGc::TiXmlAttributeSet::Find(char const*) const () from /home/controls/epics/ADProsilica/master/lib/linux-x86_64/libprosilica.so

Missing separate debuginfos, use: debuginfo-install glibc-2.17-307.el7.1.x86_64 jbigkit-libs-2.0-11.el7.x86_64 libgcc-4.8.5-39.el7.x86_64 libjpeg-turbo-1.2.90-8.el7.x86_64 libstdc++-4.8.5-39.el7.x86_64 libtiff-4.0.3-32.el7.x86_64 ncurses-libs-5.9-14.20130511.el7_4.x86_64 readline-6.2-11.el7.x86_64 sssd-client-1.16.4-37.el7_8.3.x86_64 zlib-1.2.7-18.el7.x86_64

(gdb)

(gdb)

(gdb)

(gdb)

(gdb) bt

#0  0x00007ffff7b61a7a in PGc::TiXmlAttributeSet::Find(char const*) const () from /home/controls/epics/ADProsilica/master/lib/linux-x86_64/libprosilica.so

#1  0x00007ffff7b61acd in PGc::TiXmlElement::Attribute(char const*) const () from /home/controls/epics/ADProsilica/master/lib/linux-x86_64/libprosilica.so

#2  0x00007ffff7b683db in SearchForNamedNode(PGc::TiXmlNode*, char const*) () from /home/controls/epics/ADProsilica/master/lib/linux-x86_64/libprosilica.so

#3  0x00007ffff7b4dd55 in cGcContext::GetNode(char const*, bool) () from /home/controls/epics/ADProsilica/master/lib/linux-x86_64/libprosilica.so

#4  0x00007ffff7b60ad2 in cGcBoolNode::SetupFromXML(PGc::TiXmlNode*, PGc::TiXmlNode*) () from /home/controls/epics/ADProsilica/master/lib/linux-x86_64/libprosilica.so

#5  0x00007ffff7b4bef5 in cGcContext::BuildNodeFromXML(char const*, PGc::TiXmlNode*) () from /home/controls/epics/ADProsilica/master/lib/linux-x86_64/libprosilica.so

#6  0x00007ffff7b4fcc2 in cGcContext::ProcessXMLDocument() () from /home/controls/epics/ADProsilica/master/lib/linux-x86_64/libprosilica.so

#7  0x00007ffff7b5020c in cGcContext::LoadXMLString(char const*, bool) () from /home/controls/epics/ADProsilica/master/lib/linux-x86_64/libprosilica.so

#8  0x00007ffff7b4ba12 in cGcInterface::InjectXMLString(char const*, bool) () from /home/controls/epics/ADProsilica/master/lib/linux-x86_64/libprosilica.so

#9  0x00007ffff7b3730f in cPvGigEGenicam::Opening(unsigned int, unsigned int) () from /home/controls/epics/ADProsilica/master/lib/linux-x86_64/libprosilica.so

#10 0x00007ffff7b43071 in pPvRawCamera::Open(unsigned int) () from /home/controls/epics/ADProsilica/master/lib/linux-x86_64/libprosilica.so

#11 0x00007ffff7b303c5 in PvCameraOpenByAddr () from /home/controls/epics/ADProsilica/master/lib/linux-x86_64/libprosilica.so

#12 0x00007ffff7b2b3d7 in prosilica::connectCamera (this=0x6b7030) at ../prosilica.cpp:1317

#13 0x00007ffff7b2d952 in prosilica::prosilica (this=0x6b7030, portName=0x6b6ff0 "AVT.CAM", cameraId=0x6b6ff8 "10.111.24.10", maxBuffers=100, maxMemory=0, priority=0, stackSize=0,

    maxPvAPIFrames=10) at ../prosilica.cpp:1808

#14 0x00007ffff7b2d14b in prosilicaConfig (portName=0x6b6ff0 "AVT.CAM", cameraId=0x6b6ff8 "10.111.24.10", maxBuffers=100, maxMemory=0, priority=0, stackSize=0, maxPvAPIFrames=10)

    at ../prosilica.cpp:1689

#15 0x00007ffff7b2da2b in configprosilicaCallFunc (args=0x61ff60) at ../prosilica.cpp:1839

#16 0x00007ffff5cfc987 in iocshBody (pathname=0x7fffffffde25 "iocBoot/iocbl12-SampleCamera/st.cmd", commandLine=0x0) at ../../../src/libCom/iocsh/iocsh.cpp:771

#17 0x00007ffff5cfccaa in iocsh (pathname=0x7fffffffde25 "iocBoot/iocbl12-SampleCamera/st.cmd") at ../../../src/libCom/iocsh/iocsh.cpp:835

#18 0x0000000000405c0b in main (argc=2, argv=0x7fffffffda88) at ../bl12-SampleCameraMain.cpp:17