[musl] Re: ppc64le and 32-bit LE userland compatibility

[Daniel Kolesa]

On Fri, Jun 5, 2020, at 19:27, Segher Boessenkool wrote:
> On Fri, Jun 05, 2020 at 04:18:18AM +0200, Daniel Kolesa wrote:
> > On Fri, Jun 5, 2020, at 01:35, Segher Boessenkool wrote:
> > > > The thing is, I've yet to see in which way the ELFv2 ABI *actually* requires VSX - I don't think compiling for 970 introduces any actual differences. There will be omissions, yes - but then the more accurate thing would be to say that a subset of ELFv2 is used, rather than it being a different ABI per se.
> > > 
> > > Two big things are that binaries that someone else made are supposed to
> > > work for you as well -- including binaries using VSX registers, or any
> > > instructions that require ISA 2.07 (or some older ISA after 970).  This
> > > includes DSOs (shared libraries).  So for a distribution this means that
> > > they will not use VSX *anywhere*, or only in very specialised things.
> > > That is a many-years setback, for people/situations where it could be
> > > used.
> > 
> > Third party precompiled stuff doesn't really need to concern us, since none really exists.
> 
> ... Yet.  And if you claim you support ELFv2, not mentioning the ways
> your implementation deviates from it, users will be unhappy.

Will they? The system explicitly mentions that the minimum target in the binary packages is 970. Users can't be expecting features that the hardware we support doesn't support :) Also, we're not the only ones that do this - there's musl of course, but there's also the BSDs. FreeBSD 13 uses ELFv2, and supports all the old hardware including processors without VMX, let alone VSX. OpenBSD is likely to take the same path. But I'm not opposed to making this explicit, if that's what it takes to make other people happy with it.

> 
> > It's also still an upgrade over ELFv1 regardless (I mean, the same things apply there).
> 
> Yeah, in mostly minor ways, but it all adds up for sure.

It's made my life simpler on numerous occasions, and allowed us to bring in software that'd otherwise take significant patching (no software is complete until it has its own assembly implementation of coroutines or something like that :P), or would be outright impossible without a lot of work (e.g. the lld linker).

> 
> > I'm also not really all that convinced that vectors make a huge difference in non-specialized code (autovectorization still has a way to go)
> 
> They do make a huge difference, depending on the application of course.

I've yet to see non-specialized things without explicit VSX impls where the difference is more than 5%, usually it's less.

> But VSX is not just vectors even: it also gives you twice as many
> floating point scalars (64 now), and in newer versions of the ISA it can
> be beneficially used for integer scalars even.
> 
> > and code written to use vector instructions should probably check auxval and take those paths at runtime.
> 
> No, that is exactly the point of requiring ISA 2.07.  Anything can use
> ISA 2.07 (incl. VSX) without checking first, and without having a
> fallback to some other implementation.  Going from ISA 2.01 to 2.07 is
> more than a decade of improvements, it is not trivial at all.

Portable code will need runtime checks anyway if they target big endian systems, especially existing ones, since modern vector instructions can be used even in legacy ABI, where you get no such guarantee. Sure, it's nice to not have to worry about it, but once everything is considered, going with a more modern ABI is still a big net gain for us, even if the guarantees are not all there :)

> 
> 
> > As for other instructions, fair enough, but from my rough testing, it doesn't make such a massive difference for average case
> 
> That depends on what you call the average case.  Code that is control
> and memory-bound will not benefit much from *anything* :-)

Average case is, quite literally, an average case - i.e. the average of all the software packages shipped in a distro :)

> 
> > (and where it does, one can always rebuild their thing with CFLAGS=-mcpu=power9)
> 
> Yeah, but it helps quite a bit if your system (shared) libraries get all
> improvements they can as well.

Well, glibc will still benefit automatically to a degree even if not built for a modern baseline, since it has runtime checks in place already; as for other things... well, at least for Void, I already mentioned before we're as much of a source distro as a binary one - people can easily rebuild things that bottleneck them, with modern CFLAGS, and still have things be interoperable.

> 
> 
> I'm not trying to dissuade you from not requiring VSX and 2.07 -- this
> sounds like your best option, given the constraints.  I'm just saying
> the cost is not trivial (even ignoring the ABI divergence).

Of course the cost is there - it's just not something I can do anything about. I generally recommend that people who can run LE should run LE. We're a bi-endian distribution, so there is a complete, fully functional, ISA-2.07-baseline ppc64le variant (in fact, it's our best-supported port, with greatest repo coverage and testing), as well as the 970-targeting ppc64 variant.

The 970-targeting ppc64 variant is still capable of running on modern hardware, as there are people who wish to run big endian even on their modern machines, and I don't want to take that choice away from them (and since kernel 4.20, you can even have one generic kernel capable of booting POWER4-POWER9, previously separate kernels were needed for <= POWER6 and >= POWER7). Those people can either deal with their computers running a bit slower, or rebuild the packages that specifically bottleneck them with modern CFLAGS.

> 
> 
> > > The target name allows to make such distinctions: this could for example
> > > be  powerpc64-*-linux-void  (maybe I put the distinction in the wrong
> > > part of the name here?  The glibc people will know better, and "void" is
> > > probably not a great name anyway).
> > 
> > Hm, I'm not a huge fan of putting ABI specifics in the triplet, it feels wrong - there is no precedent for it with POWER (ARM did it with EABI though),
> 
> Maybe look at what the various BSDs use?  We do have things like this.

Briefly, the FreeBSD 13 powerpc64 target triple had an -elfv2 suffix, this was done in clang, but I noticed it was reverted in the end and replaced with checks not based on triples. I believe there is no ABI in BSD powerpc triples right now.

> 
> > the last part should remain 'gnu' as it's still glibc; besides, gcc is compiled for exactly one target triplet, and traditionally with ppc compilers it's always been possible to target everything with just one compiler (endian, 32bit, 64bit, abi...).
> 
> This isn't completely true.
> 
> Yes, the compiler allows you to change word size, endianness, ABI, some
> more things.  That does not mean you can actually build working binaries
> for all resulting combinations.  As a trivial example, it will still
> pick up the same libraries from the same library paths usually, and
> those will spectacularly fail to work.

I know, I meant mostly from purely compiler perspective. It's good enough for bare-metal (in that way it matters for stuff like GRUB etc. where you still need to build a 32-bit big-endian ELF, etc)

> 
> We are biarch for some targets, which means that both powerpc-linux
> targets and powerpc64-linux targets can actually handle both of those,
> with just -m32 or -m64 needed to switch which configuration is used.
> But you cannot magically transparently switch to many other
> configurations: for those, you just build a separate toolchain for that
> specfic (variant) configuration, in the general case.

I know about the biarch case as well, and there is also multilib, as an even more elaborate form of that. That's not directly related to what I originally said, though

> 
> > The best way would probably be adding a new -mabi, e.g. -mabi=elfv2-novsx (just an example), which would behave exactly like -mabi=elfv2, except it'd emit some extra detection macro
> 
> Yeah, that sounds like a good idea.  Patches welcome :-)
> 
> (A separate target name is still needed, but this will make development
> simpler for sure).

I don't think a separate target is strictly necessary - it can be done, sure, but with an ABI switch it's more just informative than anything else.

> 
> 
> Segher
>

Daniel