[Skiboot] op-test-framework v0.4 released!

[Stewart Smith]

Hi all,

I just tagged op-test-framework v0.4.

This is our test suite for OpenPOWER Firmware.
You can grab it from github:
https://github.com/open-power/op-test-framework

and read pretty release notes here:
https://github.com/open-power/op-test-framework/releases/tag/v0.4

Below is the lovely plain text release notes:

OpenPower Test Framework
========================

This is the first tagged release of op-test-framework. We are in a march
towards releasing v1.0 near the end of 2017, and will be making
regularly tagged releases based on our progress.

As this is the first release, we're starting with a pretty clean slate
(although a lot of changes has occurred during development to get to
this point).


Current status / Known Issues
=============================
As of today, the default test suite should mostly pass on current POWER8
OpenPOWER systems running current op-build. Expect a few failures from
current FWTS (which should be addressed in upcoming FWTS releases). We
also have a funny thing going on around CAPI cards, where the PCI device
presence test may fail due to the list of PCI devices in petitboot and a
booted OS with CAPI support being different.


There is preliminary support for OpenBMC based systems (just access to
the console, no support for boot/reboot).

There is quite decent support for IBM FSP based systems.

There is also basic support for POWER9 systems, and this
op-test-framework release has been run on them. There's a few rough
patches still though, and we're not complete enough there yet.

By default, we run the "basic" suite of tests. You can also run the
"full" suite, which will take longer and be a bit more
complete. Currently, the 'basic' suite is being used to gate any changes
to the skiboot component of firmware.

Feedback is very much appreciated, either on the
openpower-firmware at lists.ozlabs.org mailing list, or via github issues:
https://github.com/open-power/op-test-framework/issues

v0.4 README
===========

Below, is what is currently in README.md, and is a good place to get started.

This repository provides a collection of tools that enable automated testing of
OpenPower systems. The op-test-framework suite is designed to test a machine
largely out of band - that is, it is designed for tests that do things like
power cycle the machine, test booting different configurations. As part of
the op-test-framework, we may run tests on the host itself (such as fwts
and HTX)

The end goal is to have a collection of tests that can be run against any
OpenPower system to validate it's function. The tests are automation/jenkins
ready.

Requirements
------------

This framework runs on most Linux based systems.

You need python 2.7 or greater.

You will also need (recent) ipmiutil - 1.8.15 or above should be adequate.

You will need to run the test suite on a machine that has access to both
the BMC and the host of the machine(s) you're testing.
Preparation

The target system will need to have an OS that can boot. That OS will
need to have several things installed on it.
Target System Requirements

A basic Linux install is assumed.

You MUST have fwts installed. To do this:

sudo apt-get install software-properties-common
sudo add-apt-repository ppa:firmware-testing-team/ppa-fwts-stable
sudo apt-get update
sudo apt-get install fwts

It must also have (package names for Debian/Ubuntu systems):

linux-tools-common linux-tools-generic lm-sensors ipmitool i2c-tools pciutils opal-prd

On RHEL-like systems, package names are:

lm_sensors ipmitool i2c-tools pciutils kernel-tools

>From skiboot, you will need the xscom-utils and gard installed:

git clone https://github.com/open-power/skiboot
cd external/xscom-utils
make
sudo make install
cd ../gard
make
sudo make install

Running the tests
-----------------

./op-test -h

Gets you help on what you can run. You will need to (at a minimum) provide
BMC and host login information. For example, to run the default test suite:

./op-test --bmc-ip bmc.example.com   \
	      --bmc-username sysadmin    \
      --bmc-password superuser   \
      --bmc-usernameipmi ADMIN   \
      --bmc-passwordipmi admin   \
      --host-ip host.example.com \
      --host-user root		 \
      --host-password 1234	 \
      --host-lspci host.example.com-lspci.txt

The default test suite will then run.

To get a list of test suites:

./op-test --list-suites

You cun run one or more suites by using the --run-suite command line option.
For example, you can choose to run tests that are only at the petitboot
command line. By default, the test runner doesn't know what state the machine
is in, so will attempt to turn everything off to get it into a known state.
You can override this initial state with the --machine-state parameter.
You can also run individual tests by using the --run option.

For example:

  ./op-test --bmc-ip bmc.example.com \
  		--bmc-username sysadmin  \
	--bmc-password superuser \
	--bmc-usernameipmi ADMIN \
	--bmc-passwordipmi admin \
	--host-ip host.example.com \
	--host-user root 	   \
	--host-password 1234	   \
	--host-lspci host.example.com-lspci.txt \
	--machine-state PETITBOOT_SHELL \
	--run testcases.OpTestPCI.OpTestPCISkiroot

The above will assume the machine is sitting at the petitboot prompt
and will run the OpTestPCISkiroot test.

-- 
Stewart Smith
OPAL Architect, IBM.