[kvm-unit-tests PATCH v9 02/31] report: Add known failure reporting option

[Thomas Huth]

On 06/05/2024 10.01, Andrew Jones wrote:
> On Mon, May 06, 2024 at 09:25:37AM GMT, Thomas Huth wrote:
>> On 04/05/2024 14.28, Nicholas Piggin wrote:
>>> There are times we would like to test a function that is known to fail
>>> in some conditions due to a bug in implementation (QEMU, KVM, or even
>>> hardware). It would be nice to count these as known failures and not
>>> report a summary failure.
>>>
>>> xfail is not the same thing, xfail means failure is required and a pass
>>> causes the test to fail. So add kfail for known failures.
>>
>> Actually, I wonder whether that's not rather a bug in report_xfail()
>> instead. Currently, when you call report_xfail(true, ...), the result is
>> *always* counted as a failure, either as an expected failure (if the test
>> really failed), or as a normal failure (if the test succeeded). What's the
>> point of counting a successful test as a failure??
>>
>> Andrew, you've originally introduced report_xfail in commit a5af7b8a67e,
>> could you please comment on this?
>>
> 
> An expected failure passes when the test fails and fails when the test
> passes, i.e.
> 
>    XFAIL == PASS (but separately accounted with 'xfailures')
>    XPASS == FAIL
> 
> If we expect something to fail and it passes then this may be due to the
> thing being fixed, so we should change the test to expect success, or
> due to the test being written incorrectly for our expectations. Either
> way, when an expected failure doesn't fail, it means our expectations are
> wrong and we need to be alerted to that, hence a FAIL is reported.

Ok, so this was on purpose, indeed. Maybe we should add this information in 
a comment right in front of the function, so that others don't scratch their 
head, too?

Anyway, this patch here is fine then:
Reviewed-by: Thomas Huth <thuth at redhat.com>