Experimental Physics and
Industrial Control System

Florian Feldbauer via Tech-talk <tech-talk at aps.anl.gov> · Thu, 31 Mar 2022 10:50:00 +0200

Hey Michael,

On 3/30/22 20:02, Michael Davidsaver wrote:

On 3/30/22 10:32, Florian Feldbauer wrote:

Hey Michael,

attached is the complete output from make and the readelf command you

mentioned.

From the readelf output:

[Requesting program interpreter: /usr/lib/ld.so.1]

If this file isn't present on target system, this is a sign that you are
trying to use a toolchain for a (slightly) different target.  The best
thing you can do is to sort out this incompatibility.

Instead of doing a multi-arch build with docker buildx on my amd64 host

I tried to build epics base directly on a raspi 3B+ (setting

STATIC_BUILD to YES):

/usr/bin/g++ -o antelope -static -L/home/pi/epics-base/lib/linux-arm

-Wl,-rpath,/home/pi/epics-base/lib/linux-arm -rdynamic        

closure.o error.o lalr.o lr0.o antelope.o mkpar.o output.o reader.o

skeleton.o symtab.o verbose.o warshall.o     -Wl,-Bdynamic 

-lpthread    -lm -lrt -ldl -lgcc

Installing created executable ../../../../bin/linux-arm/antelope
/home/pi/epics-base/bin/linux-arm/antelope -bparse -l -d ../flex/parse.y

make[4]: /home/pi/epics-base/bin/linux-arm/antelope: No such file or

directory

^Cmake[3]: *** [../../../configure/RULES_ARCHS:58: install.linux-arm]

Interrupt

make[2]: *** [../../configure/RULES_DIRS:85: src.install] Interrupt
make[1]: *** [../configure/RULES_DIRS:85: libcom.install] Interrupt
make: *** [configure/RULES_DIRS:85: modules.install] Interrupt

pi@raspberrypi:epics-base $ ldd /home/pi/epics-base/bin/linux-arm/antelope
    linux-vdso.so.1 (0x7ef59000)

    /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so =>

/usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0x76ef1000)

    libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x76d9d000)
    /usr/lib/ld.so.1 => /lib/ld-linux-armhf.so.3 (0x76f06000)
pi@raspberrypi:epics-base $ ll /usr/lib/ld.so.1
ls: cannot access '/usr/lib/ld.so.1': No such file or directory
pi@raspberrypi:epics-base $ ll /lib/ld-linux-armhf.so.3

lrwxrwxrwx 1 root root 30 Dec  1 11:52 /lib/ld-linux-armhf.so.3 ->

arm-linux-gnueabihf/ld-2.31.so

As you can see, the result is the same as within docker and qemu.

antelope gets linked to /usr/lib/ld.so.1 which points to

/lib/ld-linux-armhf.so.3.

The first does not exist on the system, but the latter does.

I doubt that the official Raspbian OS is using the wrong toolchain, also

this only happens when having STATIC_BUILD=YES in configure/CONFIG_SITE

Therefore I compared the command used to build antelope on arm with the

command used on amd64:

/usr/bin/g++ -o antelope -static -L/epics/base/lib/linux-arm

-Wl,-rpath,/epics/base/lib/linux-arm          -rdynamic closure.o

error.o lalr.o lr0.o antelope.o mkpar.o output.o reader.o skeleton.o

symtab.o verbose.o warshall.o -Wl,-Bdynamic  -lpthread   -lreadline

-lm -lrt -ldl -lgcc

/usr/bin/g++ -o antelope -Wl,-Bstatic -L/epics/base/lib/linux-x86_64

-Wl,-rpath,/epics/base/lib/linux-x86_64          -rdynamic

-m64         closure.o error.o lalr.o lr0.o antelope.o mkpar.o

output.o reader.o skeleton.o symtab.o verbose.o warshall.o

-Wl,-Bdynamic  -lpthread   -lreadline -lm -lrt -ldl -lgcc

On arm it uses `-static` while on amd64 it used `-Wl,-Bstatic`...I was

able to pin this issue down to the following difference:

$ cat configure/os/CONFIG.linux-arm.linux-arm
# CONFIG.linux-arm.linux-arm
#
# Definitions for native linux-arm builds
# Sites may override these definitions in CONFIG_SITE.linux-arm.linux-arm
#-------------------------------------------------------

# Include common gnu compiler definitions
include $(CONFIG)/CONFIG.gnuCommon
$ cat configure/os/CONFIG.linux-x86.linux-x86
#  CONFIG.linux-x86.linux-x86
#
# Definitions for linux-x86 host - linux-x86 target builds
# Sites may override these definitions in CONFIG_SITE.linux-x86.linux-x86
#-------------------------------------------------------

# Include common gnu compiler definitions
include $(CONFIG)/CONFIG.gnuCommon

STATIC_LDFLAGS_YES= -Wl,-Bstatic
STATIC_LDFLAGS_NO=
STATIC_LDLIBS_YES= -Wl,-Bdynamic
STATIC_LDLIBS_NO=

Adding those 4 last rows from CONFIG_SITE.linux-x86.linux-x86 to

CONFIG_SITE.linux-arm.linux-arm solved my issue.

Now static build also works on armv7.

Cheers,
Florian

Alternately, you could path Base to remove references to "-Wl,-Bdynamic",
which would result in a fully static executable (with the limitations
which this implies).

/usr/bin/g++ -o antelope -static -L/epics/base/lib/linux-arm

-Wl,-rpath,/epics/base/lib/linux-arm          -rdynamic closure.o

error.o lalr.o lr0.o antelope.o mkpar.o output.o reader.o skeleton.o

symtab.o verbose.o warshall.o -Wl,-Bdynamic  -lpthread   -lreadline

-lm -lrt -ldl -lgcc

However, you may later encounter strange linker errors and/or crashes
if object code from different, somewhat incompatible, toolchains are
being linked together.  So I would encourage you to investigate the
toolchain(s) first.

Cheers,
Florian

Am 30.03.2022 um 17:47 schrieb Michael Davidsaver:

On 3/30/22 07:15, Florian Feldbauer via Tech-talk wrote:

Hey all,

we use docker images to easily deploy our IOCs for our detector

control system.

The images are build for amd64, arm64/aarch64 and armv7.

To reduce the size of those images, I recently switched from

dynamic to static linking by adding the line

 >      && echo "STATIC_BUILD=YES" >> configure/CONFIG_SITE.local \
to our Dockerfile.

When compiling base7 for amd64 and aarch64 this is working just

fine, but I get an error on arm/v7:

#0 189.7 Installing created executable

/epics/base/bin/linux-arm/antelope

#0 190.1 /epics/base/bin/linux-arm/antelope -bparse -l -d

../flex/parse.y

#0 190.1 /usr/lib/ld.so.1: No such file or directory
#0 198.7 Expanding ../libComVersion.h@ to ../O.Common/libComVersion.h

I investigated the issue a bit closer and found the following:

root@195e454a8580:/epics/src/base# ldd

/epics/base/bin/linux-arm/antelope

  linux-vdso.so.1 (0x7ef10000)
  libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x76e71000)
  /usr/lib/ld.so.1 => /lib/ld-linux-armhf.so.3 (0x76f70000)

It looks like "STATIC_BUILD=YES" was ignored.

root@195e454a8580:/epics/src/base# ls /usr/lib/ld.so.1
ls: cannot access '/usr/lib/ld.so.1': No such file or directory

For some reason antelope gets linked against /usr/lib/ld.so.1 on

armv7 but this file does not exist.

Does anyone know what causes this issue?

It would help if you could attach, or post somewhat, the complete
output of a clean 'make'.

Also the output of:

readelf -aW /epics/base/bin/linux-arm/antelope

Cheers,
Florian

--
Ruhr-Universität Bochum
AG der Experimentalphysik I
Dr. Florian Feldbauer
NB 2/131 / Fach 125
Universitätsstr. 150
D-44801 Bochum

Office: NB 2/134
Phone:  (+49)234 / 32-23563
Fax:    (+49)234 / 32-14170
https://paluma.ruhr-uni-bochum.de

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System