Expanding CI to live testing

[Andrew Geissler]

> On Jun 24, 2020, at 3:16 PM, Verdun, Jean-Marie <jean-marie.verdun at hpe.com> wrote:
> 
> Hi,
>  
> As some of you are aware I am working on a Continuous Integration system which allow developer to test their build on a real hardware. I built a proof of concept before we had to lockdown our Houston Campus. The good news is that it starts to work, and I am using it extensively to work on linuxboot (it is available here: https://osfci.tech). So what can I do ?

Hi Jean-Marie, welcome to OpenBMC. My name is Andrew and I’m involved with a lot of our OpenBMC CI efforts.

>  
> My secondary goal is to automatize live testing on real hardware and probably interface the CTRL pool to a Robot server ( https://robotframework.org/). This part still need to be developed, and the current API has the basic coding to support it, but seriously need renaming, and convention build up.

There are two types of CI in OpenBMC, repository CI, where we build and compile an individual software based repository and run it’s unit tests. This all happens within a docker container and does a variety of other tests like code formatting and valgrind type checks.

The second type of CI is where we do the full bitbake and build a real image that can be verified within QEMU and on real hardware. This CI happens once a change has been merged into a software repository.  This CI is also all driven from within docker containers. Our public OpenBMC jenkins builds a variety of system configurations. The systems built in CI are chosen based on getting the most coverage of openbmc code. Once HPE has a system in upstream, we could discuss adding it into our public CI.

https://github.com/openbmc/openbmc/wiki/OpenBMC-Infrastructure-Workgroup#infrastructure-scripts has a somewhat dated but still relevant overview of the scripts we use for CI within openbmc.

>  
> My current challenge with OpenBMC is related to build time and not compete with the existing infrastucture but more being integrated to it. I tried to understand how we test new pull request and it looks like that we are using Jenkins. I have no experience with it, but that is fine (I used travisci and Appveyorci).

Yes, it’s better to just get the system you need added to the openbmc upstream CI.  The way we do hardware CI within IBM is the following:

- We have our own Jenkins running within our lab.
- This jenkins monitors for the upstream jenkins to mark a gerrit commit as Verified (i.e it’s passed all upstream CI)
- Once this occurs, the downstream jenkins runs some logic to find the flash image it needs from the upstream jenkins
- It then uses the openbmc robot test framework suite (https://github.com/openbmc/openbmc-test-automation) to flash the image and run a set of test cases on one of our servers
- Upon completion of the downstream hardware CI, the downstream jenkins will add a comment to the gerrit review on whether it passed or failed 
 
> So I do have a couple of questions:
>  
> 	• Does the Jenkins build could be made into a Docker image knowing that my compile node runs under Ubuntu (I believe 18.04)
> 	• Could we find a way when our Jenkins cluster build is done to extract the build result, automatize the transfer of it to perhaps an object storage pool with a unique UUID identifier. The challenge will be to retrieve that UUID into the gerrit page, or the Jenkins log.
> 	• If the build is successful, the end user could use that unique UUID to test on a live system. The osfci system will then extract from the object storage backend the build results and bootstrap it on the first CTRL node available.
> 	• Then an interactive session could start, or the robot framework system could have a look to the results and feedback jenkins, or the end user 

I think it would be great if we could have your infrastructure follow a similar design as laid out above. Have it monitor gerrit for the Verify tags and then kick off validating the image(s) on your collection of hardware and report status back via a comment to the gerrit review.

> vejmarie