[External] Upcoming enhancements to `run-unit-test-docker`

[Lei Yu]

Reporting a bug on building with proxy:

    base: WARNING: Retrying (Retry(total=4, connect=None, read=None,
redirect=None, status=None)) after connection broken by
'ProxyError('Cannot connect to proxy.',
NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object
at 0x7efe419e99d0>: Failed to establish a new connection: [Errno -2]
Name or service not known'))': /simple/inflection/
    RAN: /usr/bin/docker build --build-arg http_proxy=http://<myproxy>
--build-arg 'https_proxy={https_proxy}' --network=host --force-rm
--no-cache=false -t
openbmc/ubuntu-unit-test-base:2021-W07-773f263e895d8ed6 -

The https_proxy is not correctly set.
Submitted https://gerrit.openbmc-project.xyz/c/openbmc/openbmc-build-scripts/+/40699
to fix the issue.

On Fri, Feb 12, 2021 at 8:51 AM Patrick Williams <patrick at stwcx.xyz> wrote:
>
> Hello,
>
> Inside openbmc-build-scripts is a tool which is used by Jenkins to run
> repository-level CI and I think some developers also use it:
> `run-unit-test-docker`.  I've rewritten a component of this, which is
> now at `scripts/build-unit-test-docker`, in Python and done some
> enhancements to it.  The [currently] last commit in this sequence is at
> [1].
>
> Important notices:
>
>     - Location change: build-unit-test-docker.sh ->
>       scripts/build-unit-test-docker
>         + I suspect few people run this directly, but instead use
>           `run-unit-test-docker` which hides the rename.
>
>     - Python3.6 minimum.
>         + Bitbake already requires this so it really shouldn't be a
>           problem for anyone.
>
>     - Python 'sh' module needed.
>         + You can install this with the 'python3-sh' package on many
>           distros or 'pip install sh'.
>
>     - Docker stages are now tagged with different names / tag revs.
>         + I wrote a `scripts/clean-unit-test-docker` which will clean
>           out old Docker tags and can be ran periodically.
>
> Major changes:
>
>     - Rewrite in Python.
>     - Significant speed improvements:
>         - Docker stages all build in parallel to formulate the final image.
>         - All Docker stages are tagged and reused from run to run.
>         - CMake packages now use the Ninja backend.  This is the same as
>           bitbake and faster than make.
>     - Included packages are defined in a dictionary rather than as raw
>       Dockerfiles.
>
> Upcoming changes:
>
>     - Shortly, I will add support to incorporate base package changes
>       into the image itself.  We have had cases where a change to
>       a common package like `sdbusplus` broke CI because a package that
>       depended on it failed to compile with the changes.  I'll add code
>       that uses some of the GERRIT_* environment variables to determine
>       if the commit-under-test is part of one of the base packages and
>       include it in the Docker image for the test run.
>
>     - Topic-based testing.  We often run into co-dependencies between
>       phosphor-dbus-interfaces and another repository.  Within the next
>       month I'll enhance the script so that if you change PDI and your
>       repository and tag the commit with the same topic, the PDI commit
>       from that topic will be pulled into the Docker image so your
>       repository commit can pass CI.
>
>     - Some refactoring.  I did a lot of incremental changes from the
>       original shell script to the Python version I currently have and
>       didn't always rewrite in the most Pythonic way.  There will be
>       some more, mostly non-functional, changes to fix this.
>
> Feel free to comment on the review series if you have any concerns with
> these changes.
>
> 1. https://gerrit.openbmc-project.xyz/c/openbmc/openbmc-build-scripts/+/40366
>
> --
> Patrick Williams



-- 
BRs,
Lei YU