[PATCH] tools/perf/tests/shell/base_probe: Enhance print_overall_results to print summary information

[Athira Rajeev]

> On 17 Oct 2024, at 3:52 PM, Michael Petlan <mpetlan at redhat.com> wrote:
> 
> On Mon, 14 Oct 2024, Athira Rajeev wrote:
>> Currently print_overall_results prints the number of
>> fails in the summary, example from base_probe tests in
>> testsuite_probe:
>> 
>> ## [ FAIL ] ## perf_probe :: test_invalid_options SUMMARY ::
>> 11 failures found
>> 
>> test_invalid_options contains multiple tests and out
>> of that 11 failed. Sometimes it could happen that it
>> is due to missing dependency in the build or environment
>> dependency.
>> 
>> Example, perf probe -L requires DWARF enabled. otherwise
>> it fails as below:
>> ./perf probe -L
>>  Error: switch `L' is not available because NO_DWARF=1
>> 
>> "-L" is tested as one of the option in :
>>   for opt in '-a' '-d' '-L' '-V'; do
>>   <<perf probe test>>
>>   print_results $PERF_EXIT_CODE $CHECK_EXIT_CODE "missing argument
>> for $opt"
>> 
>> Here -a and -d doesn't require DWARF. Similarly there
>> are few other tests requiring DWARF. To hint the user that
>> missing dwarf could be one issue, update print_overall_results
>> to print a comment string along with summary hinting the possible
>> cause. Update test_invalid_options.sh and test_line_semantics.sh
>> to pass the info about dwarf requirement since these tests
>> failed when perf is built without DWARF.
>> 
>> With the change:
>> ## [ FAIL ] ## perf_probe :: test_invalid_options SUMMARY ::
>> 11 failures found :: Some of the tests need DWARF to run
> 
> Hello Athira,
> 
> I admit that the idea of some hint why the test failed
> might be useful, however a static hint that might or
> might not be related to the particular failure seems to
> be rather misleading to me. If the test fails for any
> other reason, the user is still told about DWARF which
> might not be true.
> 
> I think that such hints would need some test result post-
> processing to determine the actual reason.

Sure, agree to your point. I will add post-processing check for dwarf and accordingly add the summary information
Will post a V2 

Thanks
Athira
> 
> Michael
> 
>> 
>> Signed-off-by: Athira Rajeev <atrajeev at linux.vnet.ibm.com>
>> ---
>> tools/perf/tests/shell/base_probe/test_invalid_options.sh | 2 +-
>> tools/perf/tests/shell/base_probe/test_line_semantics.sh  | 2 +-
>> tools/perf/tests/shell/common/init.sh                     | 3 ++-
>> 3 files changed, 4 insertions(+), 3 deletions(-)
>> 
>> diff --git a/tools/perf/tests/shell/base_probe/test_invalid_options.sh b/tools/perf/tests/shell/base_probe/test_invalid_options.sh
>> index 1fedfd8b0d0d..e133bb0aa949 100755
>> --- a/tools/perf/tests/shell/base_probe/test_invalid_options.sh
>> +++ b/tools/perf/tests/shell/base_probe/test_invalid_options.sh
>> @@ -75,5 +75,5 @@ done
>> 
>> 
>> # print overall results
>> -print_overall_results "$TEST_RESULT"
>> +print_overall_results "$TEST_RESULT" "Some of the tests need DWARF to run"
>> exit $?
>> diff --git a/tools/perf/tests/shell/base_probe/test_line_semantics.sh b/tools/perf/tests/shell/base_probe/test_line_semantics.sh
>> index d8f4bde0f585..99f4f56a3292 100755
>> --- a/tools/perf/tests/shell/base_probe/test_line_semantics.sh
>> +++ b/tools/perf/tests/shell/base_probe/test_line_semantics.sh
>> @@ -51,5 +51,5 @@ done
>> 
>> 
>> # print overall results
>> -print_overall_results "$TEST_RESULT"
>> +print_overall_results "$TEST_RESULT" "Some of the tests need DWARF to run"
>> exit $?
>> diff --git a/tools/perf/tests/shell/common/init.sh b/tools/perf/tests/shell/common/init.sh
>> index 075f17623c8e..0862cbc1c6f7 100644
>> --- a/tools/perf/tests/shell/common/init.sh
>> +++ b/tools/perf/tests/shell/common/init.sh
>> @@ -46,10 +46,11 @@ print_results()
>> print_overall_results()
>> {
>> RETVAL="$1"; shift
>> + TASK_COMMENT="$*"
>> if [ $RETVAL -eq 0 ]; then
>> _echo "$MALLPASS## [ PASS ] ##$MEND $TEST_NAME :: $THIS_TEST_NAME SUMMARY"
>> else
>> - _echo "$MALLFAIL## [ FAIL ] ##$MEND $TEST_NAME :: $THIS_TEST_NAME SUMMARY :: $RETVAL failures found"
>> + _echo "$MALLFAIL## [ FAIL ] ##$MEND $TEST_NAME :: $THIS_TEST_NAME SUMMARY :: $RETVAL failures found :: $TASK_COMMENT"
>> fi
>> return $RETVAL
>> }
>> -- 
>> 2.43.5
>> 
>> 
>> 
>