Re: Fwd: cross-compiling (kind of) with Buildroot

["Johnson, Andrew N. via Tech-talk" <tech-talk@aps.anl.gov>]

On 7/3/19 3:23 PM, Pierrick M Hanlet wrote:

I would like to keep readline and do have the devel libraries.
hanlet:~/platforms/vmivme7805/buildroot> find . -name "*readline*"
./staging/usr/include/readline/readline.h
./staging/usr/lib/libreadline.a
./staging/usr/lib/libreadline.so
./staging/usr/lib/libreadline.so.6.3
./staging/usr/lib/libreadline.so.6

Note, however, that readline doesn't exist in the path that has the gcc/g++:
hanlet:~/platforms/vmivme7805/buildroot> find . -name gcc
./host/usr/i686-buildroot-linux-uclibc/sysroot/usr/lib/erlang/erts-10.1/include/internal/gcc
./host/usr/i686-buildroot-linux-uclibc/bin/gcc
And this is where I Buildroot has its cross-compile toolchain, which I've defined in
./base/configure/os/CONFIG_SITE.Common.linux-686:
# Location of the cross-compile toolchain
VMIVME=/usr/local/products/elsd/vmivme7805/linux-3.16.7_generic/output
GNU_DIR=$(VMIVME)/host/usr
GNU_TARGET=i686-buildroot-linux-uclibc

where the "buildroot" directory in my "find" commands above is a softlink that points to
the VMIVME path.  As I see it, GNU_DIR as defined points to the Buildroot build tools, but
then the header files and shared libraries for linking can't be found because they reside in
$(VMIVME)/staging/usr/.  Am I understanding this correctly?

Yes, our default is to assume that the build tools, headers and libraries are all aligned, but if not you just have to adjust some variables. You should keep the GNU_DIR setting for finding the compiler tools as you already are.

I could do something like:
GNU_DIR=$(VMIVME)
GNU_TARGET=host/usr/i686-buildroot-linux-uclibc
XXX_LIB=staging/usr/lib          or XXX_LIB=$(VMIVME)/staging/usr/lib
XXX_INC=/staging/usr/include or XXX_INC=$(VMIVME)/staging/usr/include
but I don't know what the XXX_LIB and XXX_INC should be.  I attempted to do this
with the SHRLIB_DEPLIB_DIRS, but it didn't work (I think that it's only for runtime).

To tell your cross-compiler where to find the readline header it needs a -I option that specifies the staging/usr/include directory (these must all be absolute paths BTW). The easiest way to do that is to add a line like
    OP_SYS_INCLUDES += $(VMIVME)/staging/usr/include
which will be translated into a suitable -I option for you.

Does your libreadline also need the application to be linked to ncurses? Newer versions generally don't so try without first if you're not sure. You'll need to set the COMMANDLINE_LIBRARY variable to either READLINE or READLINE_NCURSES depending on the answer to that. Then you also need to give the linker the path to the shared libraries, for which you would set either
    LDFLAGS_READLINE = -L$(VMIVME)/staging/usr/lib
or
    LDFLAGS_READLINE_NCURSES = -L$(VMIVME)/staging/usr/lib
depending on the setting of COMMANDLINE_LIBRARY.

You may also have to set the path to the readline library at runtime, although if the .so appears in /usr/lib on the target (or in some other directory that is searched from /etc/ld.so.conf) you can skip this. If you do have to set it to point elsewhere, that would be something like
    RUNTIME_LDFLAGS_READLINE = -Wl,-rpath,/wherever/lib

If you look in configure/os/CONFIG.linux-x86.linux-arm you'll see it doing stuff with a READLINE_DIR variable which should be similar to what I described above but using the extra variable. That assumes the runtime and compile-time locations are the same

Happy 4th,

- Andrew

On 7/3/19 12:20 PM, Johnson, Andrew N. wrote:

Hi Pierrick,

On 7/3/19 11:51 AM, Pierrick M Hanlet wrote:

The host build was successful, and yes, my host is linux-x86_64.

I reverted the ./base/configure/CONFIG.CrossCommon file and now when attempting
to build the target (I assume that's what I'm doing because my gcc and g++ are from
one of my Buildroot directories rather than /usr/bin), compilation fails at readline/readline.h.

Okay, good. Do you want EPICS to use GNU readline, which is optional and used by the ioc shell to provide command-line history recall and editing?

I suspect your buildroot doesn't have readline installed at all, or it doesn't have the headers for it at least. You can tell the EPICS build to not try to use it by adding
    COMMANDLINE_LIBRARY = EPICS
to your CONFIG_SITE.Common.linux-686 file. It's probably worth doing that to start with, you can always change it later and rebuild after you've installed the library and headers into your buildroot. Once you've made that configuration change try rebuilding again.

HTH,

- Andrew

On 7/2/19 4:45 PM, Johnson, Andrew N. via Tech-talk wrote:

Hi Pierrick,

Sorry for the delay...

On 7/2/19 1:56 PM, Pierrick M Hanlet via Tech-talk wrote:

I'm attempting to build EPICS base for a VMIVME7805 intel based VME card.  The kernel I

have is 3.16.7.  We are using Buildroot on an SL7.5 host machine.  I'm attempting to build
3.14.12.8.

I had found Bruno Martins'  wiki page: https://wiki-ext.aps.anl.gov/epics/index.php/How_To_cross_compile_EPICS_and_a_IOC_to_an_old_x86_Linux_system
which is very helpful.  However, Buildroot segregates the build tools from the lib & bin & include
files.

From Buildroot, my build tools are in:

.../host/usr/bin/i686-buildroot-linux-uclibc/
And, my staged lib & include & bin files are in:
.../host/staging/.

So, here's what I have in ./base/configure/os/CONFIG_SITE.Common.linux-686:
# Location of the cross-compile toolchain
VMIVME=/usr/local/products/elsd/vmivme7805/linux-3.16.7_generic/output
GNU_DIR=$(VMIVME)/host/usr
GNU_TARGET=i686-buildroot-linux-uclibc

SHRLIB_DEPLIB_DIRS += $(VMIVME)/staging/usr/lib
PROD_DEPLIB_DIRS   += $(VMIVME)/staging/usr/lib

In order to get the headers correct, I also modified ./base/configure/CONFIG.CrossCommon:
#GNU_TARGET_INCLUDE_DIR = $(wildcard $(GNU_TARGET:%=$(GNU_DIR)/%/include))
#GNU_TARGET_LIB_DIR = $(wildcard $(GNU_TARGET:%=$(GNU_DIR)/%/lib))
GNU_TARGET_INCLUDE_DIR = $(VMIVME)/staging/usr/include
GNU_TARGET_LIB_DIR     = $(VMIVME)/staging/usr/lib

In many cases instead of modifying a core build file (which in this case will also affect any other cross-build targets) you should be able to override build variable values in your configure/os/CONFIG_SITE.Common.linux-686 file. The CROSS_INCLUDES and CROSS_LDFLAGS macro assignments that use those values don't actually get expanded until they are used to generate a compiler or link line. However this won't affect this issue.

It appears that I can compile everything; for example (the last compile command):
/usr/local/products/elsd/vmivme7805/linux-3.16.7_generic/output/host/usr/bin/i686-buildroot-linux-uclibc-g++ -o libCom.so.3.14 -shared -fPIC -L/usr/local/products/elsd/vmivme7805/linux-3.16.7_generic/output/staging/usr/lib -Wl,-rpath,/usr/local/products/elsd/vmivme7805/linux-3.16.7_generic/output/staging/usr/lib        -m32    -L/usr/local/products/elsd/vmivme7805/linux-3.16.7_generic/output/host/usr/../../staging/usr/lib            bucketLib.o epicsRingPointer.o epicsRingBytes.o postfix.o calcPerform.o cvtFast.o resourceLib.o epicsOnce.o epicsSingletonMutex.o dbmf.o ellLib.o envSubr.o envData.o errlog.o errSymLib.o errSymTbl.o fdmgr.o fdManager.o freeListLib.o gpHashLib.o iocsh.o registry.o libComRegister.o iocLog.o logClient.o macCore.o macEnv.o macUtil.o aToIPAddr.o adjustment.o cantProceed.o epicsConvert.o epicsExit.o epicsStdlib.o epicsString.o truncateFile.o ipAddrToAsciiAsynchronous.o epicsUnitTest.o epicsThread.o epicsMutex.o epicsEvent.o epicsTime.o epicsMessageQueue.o epicsMath.o epicsGeneralTime.o osiClockTime.o osdSock.o osdSockAddrReuse.o osiSock.o systemCallIntMech.o epicsSocketConvertErrnoToString.o osdAssert.o osdFindSymbol.o osdInterrupt.o osdPoolStatus.o osdSignal.o osdEnv.o epicsReadline.o epicsTempFile.o epicsStdio.o osdStdio.o osdThread.o osdMutex.o osdEvent.o osdTime.o osdProcess.o osdNetIntf.o osdMessageQueue.o devLibVME.o devLibVMEOSD.o taskwd.o epicsTimer.o timer.o timerQueue.o timerQueueActive.o timerQueueActiveMgr.o timerQueuePassive.o tsDefs.o      -lpthread   -lreadline -lncurses -lm -lrt -ldl -lgcc

Actually that's a link command, EPICS uses g++ to link programs rather than ld because we have lots go C++ code, and the linking process may require instantiating C++ templates which ld can't do (or it didn't used to be able to). Is the following quote the next line of output (which would be an error from the above g++ command), or the output from a command which you elided?

Yes, the next quotation is the follows immediately from the last one.

But when it comes to loading shared libraries, it fails (next command in make):
/scratch/products/elsd/vmivme7805/linux-3.16.7_generic/output/host/usr/bin/../lib/gcc/i686-buildroot-linux-uclibc/4.8.3/../../../../i686-buildroot-linux-uclibc/bin/ld: cannot find /lib/libubacktrace.so.0
collect2: error: ld returned 1 exit status
make[3]: *** [libCom.so.3.14] Error 1

Firstly, I didn't understand why libubacktrace.so (used by uclibc) appears to have a hard-coded path.

Where is libubacktrace.so actually installed?

This one was resolved.

Can you also do 'ldd /scratch/products/elsd/vmivme7805/linux-3.16.7_generic/output/host/usr/i686-buildroot-linux-uclibc/bin/ld' since the error might be from Linux being unable to run your cross-linker because it needs that .so, and if that's the case you might resolve this by setting LD_LIBRARY_PATH to include the path where the libubacktrace.so is found.

hanlet:~/platforms/vmivme7805/epicsDEV/base> ldd /scratch/products/elsd/vmivme7805/linux-3.16.7_generic/output/host/usr/i686-buildroot-linux-uclibc/bin/ld
    linux-vdso.so.1 =>  (0x00007ffeca51f000)
    libz.so.1 => /home/jdiamond/workspace/ees-buildroot/output/host/usr/lib/libz.so.1 (0x00007f7372fc6000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f7372dc2000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f73729f5000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f73731dc000)

However, when I created a softlink (yes, a cheat) so that it was found, the build failed next on readline;
which tells me, if I understand it, that SHRLIB_DEPLIB_DIRS += $(VMIVME)/staging/usr/lib didn't work
as expected.  From Andrew's response to the 2013 query, the problem appears to be more
subtle; i.e. don't build a linux-x86 system on a linux-x86 system since it will always build for the host first.

In your case you appear to be cross-building for the target architecture linux-686 so it's not the same as your host architecture which I assume (with no evidence though) is probably linux-x86 or linux-x86_64.

I would suggest though that you try running 'make $EPICS_HOST_ARCH' from the top of your Base-3.14.12.7 tree. If this finishes without errors then you're good to look into what's wrong with the target build; if it fails though you do need to fix the host build first.

Success.

I would greatly appreciate a better understanding of the problem and how to implement a remedy?
The application developer's guide does not appear (perhaps I'm missing something) to deal with
the situation presented by Buildroot's segregating build tools from target libraries, header files, etc.

Writing documentation for this level of configurability is pretty much impossible. I'm sorry that the Build chapter of the AppDevGuide is so out of date, but that's one of those things that nobody can easily find the time to do (Janet Anderson was working on some updates but she retired some time back).

Come back with more evidence if the answers to the above don't help you find the issue.

- Andrew

-- 
Complexity comes for free, Simplicity you have to work for.

-- 
Pierrick Hanlet
Fermi National Accelerator
Accelerator Front End Controls
+1-630-840-5555 -- lab
+1-312-687-4980 -- mobile

"Whether you think you can or think you can't, you're right" -- Henry Ford

-- 
Complexity comes for free, Simplicity you have to work for.

-- 
Pierrick Hanlet
Fermi National Accelerator
Accelerator Front End Controls
+1-630-840-5555 -- lab
+1-312-687-4980 -- mobile

"Whether you think you can or think you can't, you're right" -- Henry Ford

-- 
Complexity comes for free, Simplicity you have to work for.