[PATCH kernel] powerpc/llvm/lto: Allow LLVM LTO builds

Tue May 10 04:07:54 AEST 2022

Hi Alexey,

On Mon, May 09, 2022 at 05:42:59PM +1000, Alexey Kardashevskiy wrote:
> 
> 
> On 5/9/22 15:18, Alexey Kardashevskiy wrote:
> > 
> > 
> > On 5/4/22 07:21, Nick Desaulniers wrote:
> > > On Thu, Apr 28, 2022 at 11:46 PM Alexey Kardashevskiy
> > > <aik at ozlabs.ru> wrote:
> > > > 
> > > > This enables LTO_CLANG builds on POWER with the upstream version of
> > > > LLVM.
> > > > 
> > > > LTO optimizes the output vmlinux binary and this may affect the FTP
> > > > alternative section if alt branches use "bc" (Branch Conditional) which
> > > > is limited by 16 bit offsets. This shows up in errors like:
> > > > 
> > > > ld.lld: error: InputSection too large for range extension thunk
> > > > vmlinux.o:(__ftr_alt_97+0xF0)
> > > > 
> > > > This works around the issue by replacing "bc" in FTR_SECTION_ELSE with
> > > > "b" which allows 26 bit offsets.
> > > > 
> > > > This catches the problem instructions in vmlinux.o before it LTO'ed:
> > > > 
> > > > $ objdump -d -M raw -j __ftr_alt_97 vmlinux.o | egrep '\S+\s*\<bc\>'
> > > >    30:   00 00 82 40     bc      4,eq,30 <__ftr_alt_97+0x30>
> > > >    f0:   00 00 82 40     bc      4,eq,f0 <__ftr_alt_97+0xf0>
> > > > 
> > > > This allows LTO builds for ppc64le_defconfig plus LTO options.
> > > > Note that DYNAMIC_FTRACE/FUNCTION_TRACER is not supported by LTO builds
> > > > but this is not POWERPC-specific.
> > > 
> > > $ ARCH=powerpc make LLVM=1 -j72 ppc64le_defconfig
> > > $ ARCH=powerpc make LLVM=1 -j72 menuconfig
> > > <disable FTRACE, enable LTO_CLANG_THIN>
> > > $ ARCH=powerpc make LLVM=1 -j72
> > > ...
> > >    VDSO64L arch/powerpc/kernel/vdso/vdso64.so.dbg
> > > /usr/bin/powerpc64le-linux-gnu-ld:
> > > /android0/llvm-project/llvm/build/bin/../lib/LLVMgold.so: error
> > > loading plugin:
> > > /android0/llvm-project/llvm/build/bin/../lib/LLVMgold.so: cannot open
> > > shared object file: No such file or directory
> > > clang-15: error: linker command failed with exit code 1 (use -v to see
> > > invocation)
> > > make[1]: *** [arch/powerpc/kernel/vdso/Makefile:67:
> > > arch/powerpc/kernel/vdso/vdso64.so.dbg] Error 1
> > > 
> > > Looks like LLD isn't being invoked correctly to link the vdso.
> > > Probably need to revisit
> > > https://lore.kernel.org/lkml/20200901222523.1941988-1-ndesaulniers@google.com/
> > > 
> > > How were you working around this issue? Perhaps you built clang to
> > > default to LLD? (there's a cmake option for that)
> > 
> > 
> > What option is that? I only add  -DLLVM_ENABLE_LLD=ON  which (I think)

The option Nick is referring to here is CLANG_DEFAULT_LINKER, which sets
the default linker when clang is invoked as the linker driver, which the
PowerPC vDSO Makefile does. We have been trying to move all parts of the
kernel to compile with $(CC) and link with $(LD) so that the default
linker invoked by the compiler is taken out of the equation.

For what it's worth, I think that the error Nick is seeing is due to a
lack of the LLVMgold.so plugin on his system, which is built when
-DLLVM_BINUTILS_INCDIR=... is included in the list of CMake variables,
which you have. The LLVMgold.so plugin is needed when doing LTO with
GNU ld, which is the case with the vDSO currently for the reason I
mentioned above. We could work around this with Nick's proposed
'-fuse-ld=lld' patch or just disable LTO for the vDSO.

https://llvm.org/docs/GoldPlugin.html

My version of the hack would probably be:

ccflags-$(CONFIG_LD_IS_LLD) += -fuse-ld=lld
asflags-$(CONFIG_LD_IS_LLD) += -fuse-ld=lld

> > tells cmake to use lld to link the LLVM being built but does not seem to
> > tell what the built clang should do.

Right, -DLLVM_ENABLE_LLD=ON is equivalent to -DLLVM_USE_LINKER=lld,
except when doing a multi-stage build:

https://llvm.org/docs/CMake.html#llvm-related-variables

> > 
> > Without -DLLVM_ENABLE_LLD=ON, building just fails:
> > 
> > [fstn1-p1 ~/pbuild/llvm/llvm-lto-latest-cleanbuild]$ ninja -j 100
> > [619/3501] Linking CXX executable bin/not
> > FAILED: bin/not
> > : && /usr/bin/clang++ -fPIC -fvisibility-inlines-hidden
> > -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra
> > -Wno-unused-parameter -Wwrite-strings -Wcast-qual
> > -Wmissing-field-initializers -pedantic -Wno-long-long
> > -Wc++98-compat-extra-semi -Wimplicit-fallthrough
> > -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor
> > -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion
> > -Wmisleading-indentation -fdiagnostics-color -ffunction-sections
> > -fdata-sections -flto -O3 -DNDEBUG -flto
> > -Wl,-rpath-link,/home/aik/pbuild/llvm/llvm-lto-latest-cleanbuild/./lib
> > -Wl,--gc-sections utils/not/CMakeFiles/not.dir/not.cpp.o -o bin/not
> > -Wl,-rpath,"\$ORIGIN/../lib"  -lpthread  lib/libLLVMSupport.a  -lrt
> > -ldl  -lpthread  -lm  /usr/lib/powerpc64le-linux-gnu/libz.so
> > /usr/lib/powerpc64le-linux-gnu/libtinfo.so  lib/libLLVMDemangle.a && :
> > /usr/bin/ld: lib/libLLVMSupport.a: error adding symbols: archive has no
> > index; run ranlib to add one
> > clang: error: linker command failed with exit code 1 (use -v to see
> > invocation)
> > [701/3501] Building CXX object
> > utils/TableGen/CMakeFiles/llvm-tblgen.dir/GlobalISelEmitter.cpp.o
> > ninja: build stopped: subcommand failed.
> > 
> > 
> > 
> > My head hurts :(
> > The above example is running on PPC. Now I am trying x86 box:
> > 
> 
> A bit of progress.
> 
> cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang;lld" -DLLVM_TARGET_ARCH=PowerPC
> -DLLVM_TARGETS_TO_BUILD=PowerPC ~/llvm-project//llvm -DLLVM_ENABLE_LTO=ON
> -DLLVM_BINUTILS_INCDIR=/usr/lib/gcc/powerpc64le-linux-gnu/11/plugin/include/
> -DCMAKE_BUILD_TYPE=Release
> 
> produces:
> 
> -- Native target architecture is PowerPC
> 
> ....
> -- LLVM host triple: x86_64-unknown-linux-gnu
> -- LLVM default target triple: x86_64-unknown-linux-gnu
> 
> 
> and the resulting "clang" can only to "Target: x86_64-unknown-linux-gnu",
> how do you build LLVM exactly? Thanks,

I have never tried to cross compile LLVM but you can use
LLVM_DEFAULT_TARGET_TRIPLE to change the default target triple. From
what I can tell, the way you are building LLVM on your PowerPC host is
fine.

Cheers,
Nathan