[OpenBMC]: Unit test coverage analysis on openbmc repositories

Fri Mar 6 20:27:21 AEDT 2020

It's found that the test coverage result is not correct for some repos.

E.g. for phosphor-psu-code-mgmt, there are quite a few tests for code
in `src`, but it only shows 2.5%:
https://openpower.xyz/job/openbmc-unit-test-coverage/2/artifact/openbmc-build-scripts/scripts/unit-test-meta-data/phosphor-psu-code-mgmt/build/meson-logs/coveragereport/index.html

It looks like `lcov` is not handling such case well, and there is a
similar question on StackOverflow:
https://stackoverflow.com/questions/27556872/lcov-doesnt-match-gcov-coverage-results-for-a-googletest-unit-test

I tried to remove `lcov` and use `gcovr` in local CI, and the result
looks correct.

`lcov` is required for autotools' coverage, so we can not just remove it.
If I add `gcovr` into the docker image without removing `lcov`,
unfortunately, meson still uses `lcov` for `coverage-html` when both
tools are available.

An issue is created at
https://github.com/mesonbuild/meson/issues/6747, hopefully we could
get some feedback from meson.

In OpenBMC, do you have any ideas on how to "fix" the coverage result?

On Thu, Mar 5, 2020 at 3:59 PM Lakshminarayana R Kammath
<lkammath at in.ibm.com> wrote:
>
> Hi All,
>
> please confirm on following
>
> Boost-bus - compilation fails . Is it used? Should be archived @andrewj?
>
> Google-ipmi-i2c - code compiles well, complain in formatting script @pstrinkle
> subsys_initcall(i2c_via_ipmi_init);^[[m^M
> Traceback (most recent call last):^M
>   File "/home/lkammath/final_data/unit-test.py", line 826, in <module>^M
>     check_call_cmd("./format-code.sh", CODE_SCAN_DIR)^M
>   File "/home/lkammath/final_data/unit-test.py", line 229, in check_call_cmd^M
>     check_call(cmd)^M
>   File "/usr/lib/python2.7/subprocess.py", line 190, in check_call^M
>     raise CalledProcessError(retcode, cmd)^M
> subprocess.CalledProcessError: Command '('./format-code.sh', '/home/lkammath/final_data/google-ipmi-i2c')' returned non-zero exit status 1^M
> Traceback (most recent call last):^M
>   File "/home/lkammath/final_data/dbus-unit-test.py", line 91, in <module>^M
>
> Inarp.git - Should be archived @andrewj?
> Compilation issue
> inarp.c: In function ‘arp_recv’:
> inarp.c:433:4: error: taking address of packed member of ‘struct arp_packet’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
>   433 |    &inarp_req.src_ip);
>       |    ^~~~~~~~~~~~~~~~~
> In function ‘do_ifreq’,
>     inlined from ‘get_local_ipaddr’ at inarp.c:182:7,
>     inlined from ‘arp_recv’ at inarp.c:419:7,
>     inlined from ‘main’ at inarp.c:480:4:
> inarp.c:170:2: error: ‘strncpy’ output may be truncated copying 16 bytes from a string of length 16 [-Werror=stringop-truncation]
>   170 |  strncpy(ifreq->ifr_name, ifname, sizeof(ifreq->ifr_name));
>       |  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> cc1: all warnings being treated as errors
> make[1]: *** [inarp.o] Error 1
> make[1]: Leaving directory `/esw/san5/lkammath/boost/inarp'
> make: *** [all] Error 2
> In CI environment
> ool.la -lncurses -ltinfo  -lcrypto^M
> copying selected object files to avoid basename conflicts...^M
> ar: `u' modifier ignored since `D' is the default (see `U')^M
> make[4]: Leaving directory '/home/lkammath/final_data/ipmitool/src/plugins/lan'^M
> Making all in lanplus^M
> make[4]: Entering directory '/home/lkammath/final_data/ipmitool/src/plugins/lanplus'^M
> /bin/bash ../../../libtool --silent  --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../.. -I../../../include    -g -O2 -Wall -Wextra -std=c99 -pedantic -Wformat -Wformat-nonliteral -MT lanplus_crypt_impl.lo -MD -MP -MF .deps/lanplus_crypt_impl.Tpo -c -o lanplus_crypt_impl.lo lanplus_crypt_impl.c^M
> lanplus_crypt_impl.c: In function 'lanplus_encrypt_aes_cbc_128':^M
> lanplus_crypt_impl.c:158:17: error: storage size of 'ctx' isn't known^M
>   158 |  EVP_CIPHER_CTX ctx;^M
>       |                 ^~~^M
> lanplus_crypt_impl.c:158:17: warning: unused variable 'ctx' [-Wunused-variable]^M
> lanplus_crypt_impl.c: In function 'lanplus_decrypt_aes_cbc_128':^M
> lanplus_crypt_impl.c:233:17: error: storage size of 'ctx' isn't known^M
>   233 |  EVP_CIPHER_CTX ctx;^M
>       |                 ^~~^M
>
> Smbios-mdr - @brad. Please confirm if used?
> No automake env (make)
>
> Thanks & Regards,
> Lakshminarayana Kamath
> --------------------------------------------------------------------------------------------------
>
>
>
> ----- Original message -----
> From: "Andrew Jeffery" <andrew at aj.id.au>
> To: "Lakshminarayana R Kammath" <lkammath at in.ibm.com>, openbmc at lists.ozlabs.org
> Cc:
> Subject: [EXTERNAL] Re: [OpenBMC]: Unit test coverage analysis on openbmc repositories
> Date: Tue, Mar 3, 2020 10:18 AM
>
>
> On Sat, 29 Feb 2020, at 02:05, Lakshminarayana R Kammath wrote:
>
> > https://github.com/openbmc/obmc-console.git  YES, UNIT TEST
>
> I need to add coverage support to obmc-console.
>
> > https://github.com/openbmc/openbmc-tools.git  NO
>
> Not sure we can expect anything for openbmc-tools - it's a collection
> of random scripts people have written to make their life developing
> openbmc easier. It's not something that should have strict QA
> processes.
>
> > https://github.com/openbmc/openpower-host-ipmi-flash.git  NO
>
> I should add some tests here where I can.
>
> > https://github.com/openbmc/hiomapd.git  YES, UNIT TEST
> > https://github.com/openbmc/libmctp.git  YES, UNIT TEST
> > https://github.com/openbmc/phosphor-led-sysfs.git  YES, UNIT TEST
>
> These three do in-fact have GCOV/LCOV support, you need to do the usual autotools invocations to get it:
>
> ```
> ./configure --enable-code-coverage && make check-code-coverage
> ```
>
> > https://github.com/openbmc/phosphor-mboxd.git  ERROR
>
> This is archived.
>
> ...
>
> > https://github.com/openbmc/u-boot  NO
>
> This uses Kbuild as a build system. I notice openbmc/linux and openbmc/qemu aren't included in the list.
>
> > https://github.com/openbmc/webui-vue.git  NO
> > https://github.com/openbmc/witherspoon-pfault-analysis.git  YES, COVERAGE
> > https://github.com/openbmc/x86-power-control.git  NO
> > *************************UNIT TEST COVERAGE
> > REPORT*************************
> > SUMMARY:
> > TOTAL REPOSITORIES : 110
> > TESTED REPOSITORIES : 43
> > ERROR : 11
> > COVERAGE REPORT : 12
> > UNIT TEST REPORT : 31
> > NO REPORT : 56
> > ***************************************************************************
> > Following actions, we can draw from above
> >
> >  1. *There are 12 repo's that have GCOV/LCOV plugged in* - We need to
> > maintain which has good coverage and try to improve those which has
> > less coverage
> >  2. *There are 31 repo's that have unit test cases added but no
> > coverage analysis because they aren't plugged in yet to GCOV/LCOV* - Is
> > there a plan for plugging in GCOV
>
> As above at least three of these are false-negatives for test coverage.
>
> Andrew
>
>
>
>