Continuous Integration Build Failures

[Andrew Geissler]

> On Oct 24, 2019, at 10:40 AM, Brad Bishop <bradleyb at fuzziesquirrel.com> wrote:
> 
> 
> 
>> On Oct 24, 2019, at 11:16 AM, Johnathan Mantey <johnathanx.mantey at intel.com> wrote:
>> 
>> I would like to propose a change to how the continuous integration system works.
>> 
>> I understand there are many builds, and there is a lot of data associated with the builds. Thus the current desire to remove the log file data in a short amount of time is a requirement. This works alright when the build succeeds. It's unhelpful when the build fails. Identifying where the build fails is impossible after approximately an hour. As an ordinary contributor I don't know how to make the CI system rebuild the source code so that the log files are available again without pushing some new change that consists of a useless piece of whitespace (or some other pointless change).  It shouldn't be necessary for the contributor to make requests for a build restart to the CI maintainers, they have their own agenda. 
> 
> Are you referring to the bitbake CI jobs or the repository CI jobs?
> 
>> 
>> Are the maintainers of the CI system willing to make a change that aids in debug?
>> 
>> Suggestions:
>> 	• Don't delete the log on build fails.

Other then some old unsupported jenkins plugins, I’m not seeing any obvious way
to this one.

>> 	• Delete everything but the log ascii output on build fails.

The CI jobs really don’t have much, there are no artifacts associated with them.
Lets try delaying the build deletes for 5 days and see if anything horrible
happens. I just changed this in the job.

>> 	• Email the ascii logfile for build fails

I don’t believe we have email support on our Jenkins instance currently but if we did,
this could be possible. The email address is known via the gerrit plugin env
variables. There seem to be some jenkins plugins for sending emails with
data on failures.

>> 	• Email a compressed debug bundle to the submitter?

See above

>> 	• Allow build fails to be restarted by the submitter so the logs can be regenerated, inspected, and captured.

Sounds like if you are logged into jenkins you can resubmit the job, but that’s
probably not all that useful because you could just look at the fail. What a
user really wants is to be able to retrigger the CI build from within gerrit.
Maintainers can just add a +1 to Ok-To-Test to cause this but
non-maintainers can not. The current “Ok-To-Test” design is in place in
gerrit to ensure only trusted people are running code on our CI
infrastructure. We can’t just let anyone with a github id approve the
CI runs. If there was a way to tie that ability to the user being in one
of the gerrit CI groups then that would be great. I don’t have much
experience in that area though.

>> 	• other…?
> 
> At first glance these are all good ideas.  Andrew how many of these can Jonathan implement himself and how many of them require access to the Jenkins instance?

Unfortunately I think all require access to the jenkins instance and for some
access to the actual VM (email support).