python 2 deprecation - assistance required

Mon Mar 2 12:46:55 AEDT 2020

On Fri, Feb 28, 2020 at 11:07 PM Patrick Williams <patrick at stwcx.xyz> wrote:
>
> On Fri, Feb 28, 2020 at 11:48:53AM +0800, Lei YU wrote:
> > On Thu, Feb 27, 2020 at 6:06 AM Patrick Williams <patrick at stwcx.xyz> wrote:
> > >   Often this means something like DEPENDS += "python-native python-yaml"
>
> I want to point out that this was meant as an example, not an exact
> recipe of what needs to be fixed.  Really we need whatever the
> dependencies are for a recipe to be expressed in that recipe.
> >
> > Why is it bad for a bitbake recipe to inherits what sdbusplus
> > depends?I tried to build phosphor-power with sdbusplus + python3, here
> > is what I found:
>
> Implicit dependencies are bad because you're relying on an
> implementation detail of another recipe that could change at any time
> (like it has here as we transition from py2 to py3).  What if we decided
> to rewrite sdbus++ in Rust?  As long as the sdbus++ package contains all
> of its own dependencies, we should be able to do that without breaking a
> single other recipe, but that assumes the other recipes are correctly
> expressing their own dependencies.
>
> sdbusplus / sdbus++ packages already RDEPEND on their own runtime
> requirements so that you and *execute* the executables they contain.
> But, if *your* recipe needs one of those itself directly, that should be
> expressed in DEPEND and not on something you happen to gain because
> sdbusplus has it in RDEPEND today.
>
> One of the bad things about bitbake is that it can be hard to recognize
> these errors in recipes.  I went through just the other day and fixed a
> handful of recipes that had a DEPEND on openssl (due to code changes
> that supported cypto-signing) but hadn't ever added them to the recipe.
> This py2 to py3 actually gives us a really good opportunity to find
> existing recipe bugs and fix them.
>
> >
> > 1. Without any change, it fails with below log:
> >
> >     meson.build:36:0: ERROR: <ExternalProgram 'python' ->
> > ['/extra/leiyu/work/openbmc/build/tmp/work/witherspoon-openbmc-linux-gnueabi/phosphor-power/1.0+gitAUTOINC+7d3a9f1b54-r1/recipe-sysroot-native/usr/bin/python-native/python']>
> > is not a valid python or it is missing setuptools
>
> Correct.  That is what I also observed.  Meson is identifying that the
> native (host build system version) python2 is missing.
>
> >
> > 2. Adding ${PYTHON_PN}-native and ${PYTHON_PN}-pyyaml-native to
> > DEPENDS, it still fails with below log:
> >
> >     meson.build:36:0: ERROR: <ExternalProgram 'python' ->
> > ['/extra/leiyu/work/openbmc/build/tmp/work/witherspoon-openbmc-linux-gnueabi/phosphor-power/1.0+gitAUTOINC+7d3a9f1b54-r1/recipe-sysroot-native/usr/bin/python-native/python']>
> > is not a valid python or it is missing setuptools
>
> I don't know the details of that repository to know if it uses pyyaml at
> build time, but lets assume it does.
>
> ${PYTHON_PN}-native is what should give you either python-native or
> python3-native, depending on which 'inherit pythonnative' or 'inherit
> python3native' you do in your recipe.  ${PYTHON_PN}-native is what
> should give you a python executable / environment that can run on the
> *build* system (hence the -native portion).
> ${PYTHON_PN}-<module>-native is what supplies that python module into
> the environment.
>
> sdbus++ uses inflection, mako, and pyyaml modules so that is why you see
> them add as DEPENDS (and in RDEPENDS_sdbus++) [1].
>
> As far as why ${PYTHON_PN}-setuptools "fixes" this problem, I already
> spoke about this in my original email (highlighting section below).
>
> > 3. Adding ${PYTHON_PN}-setuptools to DEPENDS, it fails with a
> > different log, indicating it requires python mako.
> >
> >     File "../git/power-sequencer/gen-ucd90160-defs.py", line 6, in <module>
> >         from mako.template import Template
> >     ImportError: No module named mako.template
>
> Right, so I don't know the python modules used by every repository.
> This is why their DEPENDS need to be fixed to express what they use.
>
> > 4. Adding ${PYTHON_PN}-mako-native, now it builds fine.
> >
> > So now it requires a recipe using sdbuplus to have below extra DEPENDS.
> >    ${PYTHON_PN}-native
> >    ${PYTHON_PN}-pyyaml-native
> >    ${PYTHON_PN}-setuptools
> >    ${PYTHON_PN}-mako-native
>
> Great.  Commit coming?
>
> > However, it's found that if I change the `inherit pythonnative` to
> > `inherit python3native`, it builds fine.
> > So I think that is a simpler fix and makes sense: it was inheriting
> > python2 while sdbusplus is using python2; now sdbusplus is using
> > python3, we need to update the recipes using sdbusplus to use python3.
> >
> > What do you think?
>
> Sorry for the length here, but there is a lot of depth to this rabbit
> hole you've uncovered.  TL;DR is: no, this is a very bad idea. ;)
>
> I've already spoken to why we need the *real* dependencies expressed,
> but your suggestion is far far worse than it appears on the surface.
>
> (I understand how you might have arrived at that option, but there are a
> lot of bitbake details here that most people just chalk up as *magic*.)
>
> First off, sdbus++ already provides its own RDEPENDS (again see [1]).
> This means that the recipe has already expressed everything it needs to
> run (the R in RDEPENDS).  If you DEPEND on sdbus++-native, because you
> use it during the build, we already supply all the dependencies
> necessary for sdbus++ to run.  It doesn't matter if sdbus++ uses py2 or
> py3, we take care of it.  You simply DEPEND += "sdbus++-native" and
> you're done; we did entirely the right thing for you.
>
> The issue you are observing here is that this repository supplies *its
> own* python script (gen-ucd90160-defs.py) and *didn't* express the its
> own DEPENDS with all the python bits it needed.  This recipe is the one
> at fault here, not sdbusplus.  This is what I mean by "it use to get its
> dependencies as a side-effect but now needs to express them."
>
> Now, why can't we just switch 'inherit pythonnative' to 'inherit
> python3native'?  Two reasons:
>
>     1. gen-ucd90160-defs.py is currently a py2 program.  You thus really
>        don't want python3native.
>
>     2. pythonnative / python3native are doing something *very* important
>        from a Yocto perspective and you need to pick the right one.
>
> When you run 'bitbake' there are up to 3 different python environments:
>
>     a. The one included in your OS, which is where 'bitbake' itself
>        runs.
>
>     b. The one compiled by 'bitbake' as part of the ${PYTHON_PN}-native
>        recipe which can run on your build system.
>
>     c. The one compiled by 'bitbake' as part of the ${PYTHON_PN} recipe
>        which can run on your *target* system.
>
> What 'pythonnative' does is it sets up bitbake / environment variables
> for the execution of that recipe so that (b) shows up in the path before
> (a).  This is critically important!  If you don't do this, you end up in
> a odd condition where you're using your OS's python environment and not
> the one supplied by bitbake.  This can lead to a compilation succeeding
> on your machine and failing on someone else's.  It can also lead to
> scenarios where we use a python3.8 feature in a script (and
> ${PYTHON_PN}-native is 3.8) but your machine only has python3.6
> installed and you're left confused as to how any of it is suppose to
> work.
>
> When you changed 'inherit pythonnative' to 'inherit python3native' what
> you did was you made it so that the version of python supplied by
> 'python-native' wasn't visible to the build of this repository.
> Instead, it ended up falling back to *your* OS version of python2.  The
> only reason it was successful for you was because *you* happen to have
> the mako and pyyaml modules installed in your OS.  Jenkins *should* fail
> in this case (and I hope it does) because that Docker image has only the
> bare minimum python bits installed in order to run bitbake (and this is
> a good thing).
>

Thanks a lot for explaining this, now I understand what exactly the
issue is and why it needs fix.
For phosphor-power itself, ideally, we will update the script in the
repo to use pthon3, and declare necessary dependencies in the bitbake
recipe correctly.

> ======= Highlighting history about ${PYTHON_PN}-setuptools ========
> > > * I'm seeing a regular failure with python3 and build systems like
> > >   meson, which OE has also observed.  For some reason even if we DEPEND
> > >   on "python-native" or "python3-native" meson cannot find the
> > >   setuptools module correctly.  We have to also DEPEND on "python" (the
> > >   non-native variant).  Since we don't actually install python anymore,
> > >   it doesn't make sense that we have to DEPEND on it either because that
> > >   means we are building a big target package for no reason.
> > >
> > >   Maybe someone could tackle this with upstream rather than us having to
> > >   piecemeal every recipe.
> > >
> > >   https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=ac467533f0b839237c5d31ce84102d28283f99dd
> ===================================================================
>
> [1] https://gerrit.openbmc-project.xyz/c/openbmc/meta-phosphor/+/29692/9/recipes-extended/sdbusplus/sdbusplus_git.bb#18
>
> --
> Patrick Williams