Open BMC FFDC ( First Failure Data Capture) on automated Test failure
George Keishing
gkeishin at in.ibm.com
Thu Jul 21 18:22:42 AEST 2016
Proposal : Method to trigger data collection of state logs from
applications, BMC and other information on failure.
Background : Open BMC test is written in ROBOT framework which generates
XML/HTML logs but not sufficient for debugging failures in general.
- Robot Test framework HTML/XML logs gives good info but
generic
- FFDC supplements with more Open BMC state logs for analysis
- Asserts Test failure from Dev and Test perspective by
-- capturing generic logs
-- providing custom user define/preference data
-- point of failure data
Layout : Capture data with a default name openbmc_ffdc_report.txt
under current execution directory as logs/testSuite/testcaseName/ on
failure.
$ tree logs
logs
??? TestSample
??? VerifyConnection
??? openbmc_ffdc_report.txt
2 directories, 1 file
Method : Robot framework provides granular Test cases Setup and
Teardown capability
Framework methods :
Calls the Test case failure FFDC collection
keyword as part of test case exit
[Teardown] Log FFDC If Test Case Failed
Ability to use Resource/Library modules to support
functionality
Provides in-built Automatic variables updated by framework
for checking if test case PASSED or FAILED and getting test suite and case
name and paths
Auto Variable : ${TEST_STATUS} ${TEST_NAME} ${SUITE_NAME}
${EXECDIR}
# Return if the test case is a PASS
Return From Keyword If '${TEST_STATUS}' != 'FAIL'
... ELSE do the log collection
Connection methods:
Current base support for SSH and SCP.
Example :
*** Settings ***
Resource openbmc_ffdc.robot
*** Test Cases ***
Verify Connection
[Teardown] Log FFDC If Test Case Failed
Should Be Equal testing OOPS
Current code on Gerrit for
review : https://gerrit.openbmc-project.xyz/#/c/49/
output :
==============================================================================
Test Sample
==============================================================================
Verify Connection F
FFDC Collection Started : 2016-07-21
01:41:14,252163
FFDC Collection Completed : 2016-07-21
01:41:24,062357
Verify Connection | FAIL |
testing != OOPS
------------------------------------------------------------------------------
Test Sample | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
default ffdc list construct :
#-----------------------------------------------------------------
#Dict Name { Index string : { Key String : Command string} }
#-----------------------------------------------------------------
FFDC_CMD = {
'DRIVER INFO' :
{
'FW Level' : 'cat /etc/os-release',
'OS Details' : 'uname -a',
'Build Info' : 'cat /etc/version',
},
'BMC DATA' :
{
'System journal log' : 'journalctl
--no-pager',
'Displays processor activity' :
'top -n 1 -b',
},
'APPLICATION DATA' :
{
'BMC state' : '/usr/sbin/obmcutil state',
},
}
FFDC o/p :
$ cat logs/TestSample/VerifyConnection/openbmc_ffdc_report.txt
----------------------------------------------------------------------------------------------
OPEN BMC TEST FAILURE DATA CAPTURE
----------------------------------------------------------------------
TEST SUITE FILE : Test Sample
TEST CASE NAME : Verify Connection
FAILURE TIME STAMP : 2016-07-21 01:41:14,275235
This document contains the following information:
* APPLICATION DATA
* DRIVER INFO
* BMC DATA
-----------------------------------------------------------------------------------------------------
[ Detailed Logs Captured Section ]
--------------------------------------------------------------------------------------------------------
APPLICATION DATA : BMC state Executed : /usr/sbin/obmcutil state
--------------------------------------------------------------------------------------------------------
= BMC_STARTING
--------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------
DRIVER INFO : Build Info Executed : cat /etc/version
--------------------------------------------------------------------------------------------------------
20160716161753
--------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------
DRIVER INFO : OS Details Executed : uname -a
--------------------------------------------------------------------------------------------------------
Linux barreleye 4.6.4-openbmc-20160713-1 #1 Wed Jul 13 16:38:10 UTC 2016
armv5tejl GNU/Linux
--------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------
DRIVER INFO : FW Level Executed : cat /etc/os-release
--------------------------------------------------------------------------------------------------------
ID="openbmc-phosphor"
NAME="Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro)"
VERSION="0.1.0 (krogoth)"
VERSION_ID="v1.0-89-g5bc571e-dirty"
PRETTY_NAME="Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro)
0.1.0 (krogoth)"
BUILD_ID="v1.0"
--------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------
BMC DATA : System journal log Executed : journalctl --no-pager
--------------------------------------------------------------------------------------------------------
-- Logs begin at Mon 2016-07-11 22:01:17 UTC, end at Tue 2016-07-12
10:45:00 UTC. --
Jul 11 22:01:17 barreleye systemd-journald[395]: Runtime journal
(/run/log/journal/) is 3.1M, max 25.1M, 22.0M free.
Jul 11 22:01:17 barreleye kernel: Booting Linux on physical CPU 0x0
Jul 11 22:01:17 barreleye kernel: Linux version 4.6.4-openbmc-20160713-1
(jenkins at hudson) (gcc version 4.9.3 (GCC) ) #1 Wed Jul 13 16:38:10 UTC
2016
Jul 11 22:01:17 barreleye kernel: CPU: ARM926EJ-S [41069265] revision 5
(ARMv5TEJ), cr=0005317f
...
....
FFDC Wish List :
For now we are delivering only for open BMC. The base infrastructure is
kept generic and simple to extend for Open Power test framework once the
porting is complete to ROBOT framework.
* Wish list from Open BMC
- journalctl --no-pager
- bmc fw level (version)
* Wish list from Open power
- sel clear/list
- sensor list
- Inventory
- SOL output
- Driver Name
- System ID (MTMS)
- System Type
- esel support (parsing)
Thanks and Regards,
George Keishing
IBM Systems &Technology Lab, Firmware Development,
“There isn't enough time in a day to be lazy!!! .”
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/openbmc/attachments/20160721/a950ddf7/attachment-0001.html>
More information about the openbmc
mailing list