Hardware Continuous Integration

[Andrew Geissler]

Team,

We've had a fairly regular cadence of commit's getting merged into
master that then break our integration testing on our different
openbmc platforms.  Sometimes the system will no longer boot,
sometimes we can't get to BMC_READY, and other times we completely
corrupt things so bad we have to pull the flash chip and reprogram it.

The solution to preventing this is to ensure these commits don't get
merged (i.e. finding the issue beforehand).  Yes we'll still find the
issue but at least it's only on one system and the change does not
make it out to the broader audience.

We've tried for months within IBM to get some systems in a lab network
configuration that would allow us to expose them to the outside world
(openpower.xyz, docker.com, ...) but we're really not setup to handle
this currently in our lab.  We're going to continue working it but
we'd like to propose an interim solution.

1) We'll have a jenkins master running internally to IBM start
triggering off of gerrit commit's that have passed the initial build
and QEMU run (https://openpower.xyz/job/openbmc-build-gerrit-trigger-multi/)

2) This job will extract the image from openpower.xyz and run it
through a basic hardware CI (code update, verify it boots to
petitboot, verify the basic REST API's) - Note we can add on as needed
later to this test bucket

3) If the job succeeds, we give an additional HW CI +1 Verified to the
gerrit record

4) If it fails, we push the jenkins artifacts (console log, robot test
output) out to a new github repo and put a link to that data in the
gerrit commit.  We do this because the jenkins job will not be
accessible to people outside of IBM.

For the artifact storage, we considered using a dropbox or something
but using github to store the artifacts seems the easiest since it
will just be text files and such.  We can make a cron that
periodically cleans up the older data.

Initially, a failed HW CI would not prevent a merge, it would just be
a hint for people to look at.  We do this because of the current state
of master.  Once we get things more stable, we can discuss making it
mandatory.

Thoughts/Comments?
Andrew