From 3018ecc2350ce1b575195b9e5eca4f9e581ef57e Mon Sep 17 00:00:00 2001 From: Petr Šplíchal Date: May 15 2019 15:05:36 +0000 Subject: Improve test results reporting in STI [fix #3] Introduces a new file `results.yml` which allows to clearly distinguish test failures from infrastructure errors. See the following ticket for the whole context and motivation: https://pagure.io/fedora-ci/general/issue/18 --- diff --git a/modules/ROOT/pages/standard-test-interface.adoc b/modules/ROOT/pages/standard-test-interface.adoc index 7ca6c66..59f6837 100644 --- a/modules/ROOT/pages/standard-test-interface.adoc +++ b/modules/ROOT/pages/standard-test-interface.adoc @@ -4,7 +4,7 @@ Standard Discovery, Staging and Invocation of Integration Tests. -Version: `1.1.0` +Version: `2.0.0` == Summary == @@ -28,7 +28,9 @@ By staging and invoking tests consistently in Fedora we create an eco-system for that allows varied test frameworks as well as CI system infrastructure to interoperate. The integration tests outlast the implementation details of either the frameworks they're written in or the CI systems running them. -== Terminology == +== Definitions == + +=== Terminology === Test Subject:: The items that are to be tested. @@ -76,6 +78,30 @@ _Testing system_ delegates running of the test to _test runner_ which can be dif For example ansible is run on the _test runner_ and tests are executed on the managed host. Usually a stable version of OS is used for _test runner_. +=== Results Format === + +The following format should be used to report results of individual tests in the `results.yml` file: + + results: + - {result: pass, test: test1, logs: [test1.log]} + - {result: fail, test: test2, logs: [test2.log, debug.log]} + - {result: error, test: test3, logs: [test3.log, debug.log, error.log]} + +result:: Test result. One of `pass`, `fail` or `error`. Mandatory. +test:: Test name. A unique identifier. Mandatory. +logs:: +One or more logs with detailed test output. Optional. +Path should be relative to the artifacts directory. +Some user interfaces might show only a single log by default. +In that case first log from the list should be presented to the user. + +The `result` field can contain following values: + +pass:: Test has successfully finished and passed. +fail:: Test has successfully finished and failed. +error:: There has been a problem with test execution. + + == Responsibilities == The *Testing System* is responsible to: @@ -186,7 +212,8 @@ tasks for each _test suite_ playbook: . MUST execute Ansible with inventory set to the full path of the file or directory `tests/inventory` if it exists. .. If the `tests/inventory` file doesn't exist, then the following inventory SHOULD be used as a default:
`/usr/share/ansible/inventory` . MUST execute the playbook as root. -. MUST examine the exit code of the playbook. A zero exit code is successful _test result_, non-zero is failure. +. MUST examine the exit code of the playbook. A zero exit code means tests completed successfully, non-zero means a problem with running tests. +. MUST examine the file `results.yml` in the `artifacts` folder to detect whether tests passed of failed. . MUST treat the file `test.log` in the `artifacts` folder as the main readable output of the test. . SHOULD place the textual stdout/stderr of the `ansible-playbook` command in the `ansible.log` file in the `artifacts` folder. . SHOULD treat the contents of the `artifacts` folder as the _test artifacts_. @@ -198,7 +225,8 @@ Each _test suite_ playbook or _test framework_ contained therein: . MUST provision the _test subject_ listed in the `subjects` variable appropriately for its playbook name (described above) and MUST fail if this is not possible. . MUST place the main readable output of the _test suite_ into a `test.log` file in the `artifacts` variable folder. This MUST happen even if some of the test suites fail. . SHOULD place additional _test artifacts_ in the folder defined in the `artifacts` variable. -. MUST return a zero exit code of the playbook if the _test result_ is a pass, or a non-zero exit code if the _test result_ is a fail. +. MUST return a zero exit code of the playbook if tests have been executed successfully, or a non-zero exit code if failed to run any test (e.g. because of an infrastructure error). +. MUST create a `results.yml` file in the `artifacts` directory with test results in the _results format_ defined above. If an inventory file or script exists, it: