[OpenPower-Firmware] sbe fails to build, parsAndCutElf.py should be objcopy?

Fri Mar 12 22:08:09 AEDT 2021

On 3/12/2021 1:51 AM, Stewart Smith wrote:
>>    define SBE_BUILD_CMDS
>> -       SBE_COMMIT_ID=$(SBE_VERSION) $(MAKE) -C $(@D) \
>> -               LD_LIBRARY_PATH=$(HOST_DIR)/usr/lib \
>> +       SBE_COMMIT_ID=$(SBE_VERSION) $(HOST_MAKE_ENV) $(MAKE) -C $(@D) \
>>                   CROSS_COMPILER_PATH=$(PPE42_GCC_BIN) \
>>                   all
>>    endef
> 
> That seemed to not fix it for me. Building a tree with that fix still elicits the same error, even with the fixed python patch I included originally.

Yeah for some reason it seemed to succeed on my laptop's Fedora33, but failed once I tried to enable the build on Fedora33 containers (that for some reason were disabled on the ozlabs server)

This commit here is another attempt at fixing it: https://github.com/open-power/op-build/pull/4028/commits/c30c4be6a48ad3919b5ae40b35196d31d62a9365

But with that, I'm now hitting an issue with the Hostboot CI (built on top of RHEL 7.x x86-64 with devtoolset-8):

/sbe-5799af7203689edc9590544b520595f05d9ce7ca/src/tools/trace -f Makefile #
g++ -O3 -w -g -I./ ppetracepp.C -o /tmp/hostboot-jenkins.swg-devops/jenkins-slave/workspace/Hostboot/OpenPower/GHPR/build-multi-3872/CONFIG/witherspoon/OS/rhel7/build/op-build/output/build/sbe-5799af7203689edc9590544b520595f05d9ce7ca/obj/power_axone/genfiles/ppetracepp
gcc -w -g -I./ -I/tmp/hostboot-jenkins.swg-devops/jenkins-slave/workspace/Hostboot/OpenPower/GHPR/build-multi-3872/CONFIG/witherspoon/OS/rhel7/build/op-build/output/build/sbe-5799af7203689edc9590544b520595f05d9ce7ca/src/import/chips/p9/procedures/ppe/pk/trace ppe2fsp.c ppe2fsp_cmd.c -o /tmp/hostboot-jenkins.swg-devops/jenkins-slave/workspace/Hostboot/OpenPower/GHPR/build-multi-3872/CONFIG/witherspoon/OS/rhel7/build/op-build/output/build/sbe-5799af7203689edc9590544b520595f05d9ce7ca/obj/power/genfiles/ppe2fsp
python buildInfo.py /tmp/hostboot-jenkins.swg-devops/jenkins-slave/workspace/Hostboot/OpenPower/GHPR/build-multi-3872/CONFIG/witherspoon/OS/rhel7/build/op-build/output/build/sbe-5799af7203689edc9590544b520595f05d9ce7ca/obj/power/genfiles
/tmp/hostboot-jenkins.swg-devops/jenkins-slave/workspace/Hostboot/OpenPower/GHPR/build-multi-3872/CONFIG/witherspoon/OS/rhel7/build/op-build/output/host/powerpc64le-buildroot-linux-gnu/sysroot/ppe42-toolchain/linux/libexec/gcc/powerpc-eabi/4.9.2/cc1plus: error while loading shared libraries: libmpfr.so.6: cannot open shared object file: No such file or directory
make[3]: *** [Makefile:174: linker_seeprom] Error 1
make[2]: *** [Makefile:179: all] Error 2
make[2]: Leaving directory '/tmp/hostboot-jenkins.swg-devops/jenkins-slave/workspace/Hostboot/OpenPower/GHPR/build-multi-3872/CONFIG/witherspoon/OS/rhel7/build/op-build/output/build/sbe-5799af7203689edc9590544b520595f05d9ce7ca/src/build'
make[1]: *** [Makefile:36: DD2] Error 2
make[1]: *** Waiting for unfinished jobs....
gcc -w -g -I./ -I/tmp/hostboot-jenkins.swg-devops/jenkins-slave/workspace/Hostboot/OpenPower/GHPR/build-multi-3872/CONFIG/witherspoon/OS/rhel7/build/op-build/output/build/sbe-5799af7203689edc9590544b520595f05d9ce7ca/src/import/chips/p9/procedures/ppe/pk/trace ppe2fsp.c ppe2fsp_cmd.c -o /tmp/hostboot-jenkins.swg-devops/jenkins-slave/workspace/Hostboot/OpenPower/GHPR/build-multi-3872/CONFIG/witherspoon/OS/rhel7/build/op-build/output/build/sbe-5799af7203689edc9590544b520595f05d9ce7ca/obj/power_axone/genfiles/ppe2fsp
python buildInfo.py /tmp/hostboot-jenkins.swg-devops/jenkins-slave/workspace/Hostboot/OpenPower/GHPR/build-multi-3872/CONFIG/witherspoon/OS/rhel7/build/op-build/output/build/sbe-5799af7203689edc9590544b520595f05d9ce7ca/obj/power_axone/genfiles
/tmp/hostboot-jenkins.swg-devops/jenkins-slave/workspace/Hostboot/OpenPower/GHPR/build-multi-3872/CONFIG/witherspoon/OS/rhel7/build/op-build/output/host/powerpc64le-buildroot-linux-gnu/sysroot/ppe42-toolchain/linux/libexec/gcc/powerpc-eabi/4.9.2/cc1plus: error while loading shared libraries: libmpfr.so.6: cannot open shared object file: No such file or directory
make[3]: *** [Makefile:174: linker_seeprom] Error 1
make[2]: *** [Makefile:179: all] Error 2
make[2]: Leaving directory '/tmp/hostboot-jenkins.swg-devops/jenkins-slave/workspace/Hostboot/OpenPower/GHPR/build-multi-3872/CONFIG/witherspoon/OS/rhel7/build/op-build/output/build/sbe-5799af7203689edc9590544b520595f05d9ce7ca/src/build'
make[1]: *** [Makefile:39: axone] Error 2
make[1]: Leaving directory '/tmp/hostboot-jenkins.swg-devops/jenkins-slave/workspace/Hostboot/OpenPower/GHPR/build-multi-3872/CONFIG/witherspoon/OS/rhel7/build/op-build/output/build/sbe-5799af7203689edc9590544b520595f05d9ce7ca'
make: *** [package/pkg-generic.mk:250: /tmp/hostboot-jenkins.swg-devops/jenkins-slave/workspace/Hostboot/OpenPower/GHPR/build-multi-3872/CONFIG/witherspoon/OS/rhel7/build/op-build/output/build/sbe-5799af7203689edc9590544b520595f05d9ce7ca/.stamp_built] Error 2
make: Leaving directory '/tmp/hostboot-jenkins.swg-devops/jenkins-slave/workspace/Hostboot/OpenPower/GHPR/build-multi-3872/CONFIG/witherspoon/OS/rhel7/build/op-build/buildroot'
+ '[' 2 -ne 0 ']'
+ echo 'witherspoon build failed.'

I tried to debug this, but to be honest I'm hitting ANOTHER issue before the above, apparently Hostboot tracehash stuff doesn't like binutils 2.33.1 that we pulled with the buildroot rebase. And now I'm kinda lost on WHY this is apparently not an issue for CI:

     CXX        decoder.C
     LD         libfapi2_rt.so
/tmp/klausk2/op-build/output/host/bin/powerpc64le-buildroot-linux-gnu-ld: BFD (GNU Binutils) 2.33.1 assertion fail elf64-ppc.c:11041
make[11]: *** [../../../../src/build/mkrules/cc.rules.mk:128: ../../../../img/libfapi2_rt.so] Error 1
make[10]: *** [../../../../src/build/mkrules/passes.rules.mk:105: _BUILD/PASSES/CODE/BODY] Error 2
make[9]: *** [../../../src/build/mkrules/passes.rules.mk:105: _BUILD/SUBDIR/CODE/runtime] Error 2
make[8]: *** [../../../src/build/mkrules/passes.rules.mk:105: _BUILD/PASSES/CODE/POST] Error 2
make[7]: *** [../../src/build/mkrules/passes.rules.mk:105: _BUILD/SUBDIR/CODE/fapi2] Error 2
make[7]: *** Waiting for unfinished jobs....

>> Bad news is that SBE is (as you're well aware) an 'interesting' mix of
>> using host and ppe-specific
>> toolchains, and to make things worse there's apparently no consistency
>> on finding what toolchain
>> is needed for what. So fixing this so that we'd only ever use
>> buildroot-provided tools instead of distro-provided, so we better
>> insulate in the future against similar things, is a non-trivial task.
> 
> It seems pretty fundamental to correctly doing cross compilation :)
> 
> Any further ideas?

Raja and Muruli - we really need the SBE's team action here.. At a minimum, we should be making it solar clear what toolchain (and utilities) should be used for each step of the compilation (from host, from target, or from ppe42 target). I don't  think using LD_LIBRARY_PATH is acceptable anywhere (i.e., Yocto outrights forbids it). Instead, we should be building the toolchains so that they can find their respective shared libraries where they need them. I believe that the Host and the Target (buildroot) toolchains are doing the right thing here, but I'm not sure about the ppe42 toolchain - then again, it's unclear on the build process which toolchain we should be using for what - specially inside those cryptic python scripts.

> It looks like the straight readelf invocation here is the issue, and that it should instead be looking at running one that's in the buildroot built toolchain.
> 
>> Re your question on objcopy, I don't see a reason why it couldn't be
>> used (even if it would still be
>> subject to the LD_LIBRARY_PATH thing), but I'll add SBE folks on cc
>> just in case.
> 
> Any further thoughts on this?

I think both readelf as well as objcopy are subject to the mess with the LD_LIBRARY_PATH here..

The right thing I guess would be to instead use ${CROSS_COMPILE}readelf and ${CROSS_COMPILE}objcopy etc.. It would be a simple change if these were being called directly from the Makefile(s), but when they are inside these python scripts, I guess we need the SBE team to figure that out.

  -Klaus

-- 
Klaus Heinrich Kiwi <klaus at linux.vnet.ibm.com>