python 2 deprecation - assistance required

[Lei YU]

On Thu, Feb 27, 2020 at 6:06 AM Patrick Williams <patrick at stwcx.xyz> wrote:
>
> Brad (and everyone else),
>
> Wanted to give an update on the sdbusplus conversion to python3.  There
> are lots of issues I've solved already that I want to give visibility to
> as others are doing similar conversions.  Also, there 7 recipes with bad
> dependencies that break when sdbusplus moves to python3 because they
> previous got their dependencies indirectly from sdbusplus; these need to
> be fixed before we can complete the move.
>
>
> * All of the code is done in sdbusplus and ready for merge, but I am
>   holding off on merging it until we are good to go on openbmc/openbmc.
>
>   https://gerrit.openbmc-project.xyz/c/openbmc/sdbusplus/+/29055
>   https://gerrit.openbmc-project.xyz/c/openbmc/sdbusplus/+/29545
>
>
> * The recipes for meta-phosphor are available to convert sdbusplus to
>   python3 there.
>
>   https://gerrit.openbmc-project.xyz/c/openbmc/meta-phosphor/+/29692
>
>   I will update this commit once we are ready to merge 29055 with the
>   SRCREV update instead of a patch file.
>
>
> * The recipe changes at 29692 fails to build the complete
>   openbmc/openbmc due to problems in other recipes.  Many recipes
>   assumed they were getting their "python requirements" indirectly from
>   sdbusplus (which is bad).  When I change sdbusplus to use python3, but
>   those recipes still expect python2 (and modules) they fail to build.
>
>   I ran with 'bitbake obmc-phosphor-image -k' and find the following
>   failures:
>
>     meta-phosphor/recipes-phosphor/ipmi/phosphor-ipmi-host_git.bb:do_compile
>     meta-phosphor/recipes-phosphor/power/phosphor-power_git.bb:do_compile
>     meta-phosphor/recipes-phosphor/inventory/phosphor-inventory-manager_git.bb:do_compile
>     meta-phosphor/recipes-phosphor/dump/phosphor-debug-collector_git.bb:do_compile
>     meta-phosphor/recipes-phosphor/dbus/phosphor-dbus-monitor_git.bb:do_compile
>     meta-openpower/recipes-phosphor/occ/openpower-occ-control_git.bb:do_compile
>     meta-phosphor/recipes-phosphor/leds/phosphor-led-manager_git.bb:do_compile
>
>   I'm not going to have time to fix all of these myself, so it would be
>   nice if the maintainers of each of these repositories would fix up
>   their dependencies in the Yocto recipe.
>
>   Often this means something like DEPENDS += "python-native python-yaml"
>

Why is it bad for a bitbake recipe to inherits what sdbusplus
depends?I tried to build phosphor-power with sdbusplus + python3, here
is what I found:

1. Without any change, it fails with below log:

    meson.build:36:0: ERROR: <ExternalProgram 'python' ->
['/extra/leiyu/work/openbmc/build/tmp/work/witherspoon-openbmc-linux-gnueabi/phosphor-power/1.0+gitAUTOINC+7d3a9f1b54-r1/recipe-sysroot-native/usr/bin/python-native/python']>
is not a valid python or it is missing setuptools

2. Adding ${PYTHON_PN}-native and ${PYTHON_PN}-pyyaml-native to
DEPENDS, it still fails with below log:

    meson.build:36:0: ERROR: <ExternalProgram 'python' ->
['/extra/leiyu/work/openbmc/build/tmp/work/witherspoon-openbmc-linux-gnueabi/phosphor-power/1.0+gitAUTOINC+7d3a9f1b54-r1/recipe-sysroot-native/usr/bin/python-native/python']>
is not a valid python or it is missing setuptools

3. Adding ${PYTHON_PN}-setuptools to DEPENDS, it fails with a
different log, indicating it requires python mako.

    File "../git/power-sequencer/gen-ucd90160-defs.py", line 6, in <module>
        from mako.template import Template
    ImportError: No module named mako.template

4. Adding ${PYTHON_PN}-mako-native, now it builds fine.

So now it requires a recipe using sdbuplus to have below extra DEPENDS.
   ${PYTHON_PN}-native
   ${PYTHON_PN}-pyyaml-native
   ${PYTHON_PN}-setuptools
   ${PYTHON_PN}-mako-native

However, it's found that if I change the `inherit pythonnative` to
`inherit python3native`, it builds fine.
So I think that is a simpler fix and makes sense: it was inheriting
python2 while sdbusplus is using python2; now sdbusplus is using
python3, we need to update the recipes using sdbusplus to use python3.

What do you think?

>
> * I'm seeing a regular failure with python3 and build systems like
>   meson, which OE has also observed.  For some reason even if we DEPEND
>   on "python-native" or "python3-native" meson cannot find the
>   setuptools module correctly.  We have to also DEPEND on "python" (the
>   non-native variant).  Since we don't actually install python anymore,
>   it doesn't make sense that we have to DEPEND on it either because that
>   means we are building a big target package for no reason.
>
>   Maybe someone could tackle this with upstream rather than us having to
>   piecemeal every recipe.
>
>   https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=ac467533f0b839237c5d31ce84102d28283f99dd
>
>
> * python3-setuptools does some funny mangling of our shebang's when we
>   use setuptools to install a python script.  OE's distutils3.bbclass
>   takes care of this, but often we have some hybrid autotools/setuptools
>   repository so we cannot use this bbclass.  I've added a workaround
>   here:
>
>   https://gerrit.openbmc-project.xyz/c/openbmc/meta-phosphor/+/29789/1/classes/obmc-phosphor-python3-autotools.bbclass#31
>
>   If you are using another build system (or not using this bbclass) you
>   might also run into this with other recipes.  I suspect, since we
>   aren't using python-on-target anymore, this only affects -native
>   packages, which might only ever be sdbus++?
>
>
> --
> Patrick Williams