From 069a46a0e1a5e2baf8d06912061fdbb7d2904fff Mon Sep 17 00:00:00 2001 From: Petr Šplíchal Date: Aug 07 2020 11:35:52 +0000 Subject: Add detailed tmt Quick Start Guide instructions --- diff --git a/modules/ROOT/pages/tmt.adoc b/modules/ROOT/pages/tmt.adoc index 4a0666f..315b400 100644 --- a/modules/ROOT/pages/tmt.adoc +++ b/modules/ROOT/pages/tmt.adoc @@ -10,38 +10,331 @@ It implements the https://tmt.readthedocs.io/en/latest/spec.html[Test Metadata S The same configuration can be used for enabling tests in the Fedora CI (near future), RHEL CI and https://packit.dev/[Packit]. Tests can be easily executed in your preferred environment, e.g. in virtual machine, container or directly on the localhost. -== Examples == +== First Steps == -Enable basic smoke test for my component: +=== Install === - dnf install -y tmt - tmt init --template mini - vim plans/example.fmf - git add . && git commit -m tests && git push +Install tmt on your laptop: + + sudo dnf install -y tmt # basic features, executing tests on localhost + sudo dnf install -y tmt-all # install all available tmt subpackages including all dependencies + +You can also install selected provision plugins only: + + sudo dnf install -y tmt-provision-container # additional dependencies for executing tests in containers + sudo dnf install -y tmt-provision-virtual # support for running tests in a virtual machine using testcloud + +See the tmt https://tmt.readthedocs.io/en/latest/overview.html#install[install] section for more installation options. + +=== Git Repo === + +Check out the desired dist git branch using fedpkg: + + fedpkg clone -a bash + cd bash + git checkout f32 + +Or clone your GitHub project repository: + + git clone https://github.com/psss/tmt/ + cd tmt + git checkout -b enable-tests + +=== Smoke Test === + +Let's enable a simple smoke test using the minimal plan template: + + $ tmt init --template mini + Tree '/tmp/bash' initialized. + Applying template 'mini'. + Directory '/tmp/bash/plans' created. + Plan '/tmp/bash/plans/example.fmf' created. + +Edit the newly created plan as needed, for example like this: -Create a new test based on a shell/beakerlib template: + summary: + Basic smoke test for bash + execute: + script: bash --version - tmt test create --template shell tests/basic - tmt test create --template beakerlib tests/advanced +== Execute Tests == -Run all/selected tests safely in a virtual machine: +=== Run Tests === + +Execute all available tests safely in a virtual machine: tmt run + +Run only tests matching given name or located under the current directory: + tmt run test --name smoke + tmt run test --name . + +Show detailed test results from the latest tmt run executed by current user: + + tmt run --last report -fvvv + +[NOTE] +==== +Executing tests enabled using the Standard Test Interface in tests/tests.yml is not supported yet but we are working on it. +==== + + +=== Select Steps === + +Explicitly choose which steps should be run: + + tmt run discover + +This will provide an overview of tests which would be run. +To list individual tests enable the verbose mode: + + tmt run discover --verbose + tmt run discover -v + + +=== Provision Options === -Execute tests in my preferred environment: +Choose `local` as the provision method but run `--all` steps: - tmt run --all provision --how container - tmt run --all provision --how virtual tmt run --all provision --how local - tmt run --all provision --how ... + +Execute inside a container or virtual machine: + + tmt run --all provision --how container --image fedora + tmt run --all provision --how virtual --image fedora-32 + +Check all available provision plugins: + + tmt run provision --help + +=== Prepare Options === + +Install additional packages on the guest: + + tmt run --all prepare --how install --package httpd + +Get the latest package from provided copr repository: + + tmt run --all prepare --how install --copr psss/tmt --package tmt + +Use the freshly build local rpm or all rpms from provided local directory: + + tmt run --all prepare --how install --package tmp/RPMS/noarch/tmt-0.20-1.fc32.noarch.rpm + tmt run --all prepare --how install --directory tmp/RPMS/noarch + +Check all available prepare options: + + tmt run prepare --help + +=== Pull Requests === + +In order to test a pull request on GitHub enable the https://github.com/marketplace/packit-as-a-service[Packit-as-a-Service] integration and add a `.packit.yaml` configuration file: + + jobs: + - job: tests + trigger: pull_request + metadata: + targets: + - fedora-all + +For more details see the https://packit.dev/testing-farm/[Testing Farm] documentation. +Once the integration is enabled push the branch, create a new pull request as ususal and wait for results: + + git push origin -u enable-tests + +[NOTE] +==== +Fedora CI support for tmt tests is coming in August/September 2020. +==== + + +== Create Test == + +In order to create more complex tests let's use the base plan template: + + tmt plan create /plans/basic --template base + tmt plan create /plans/basic -t base + +Update summary as needed, keep discover method to `fmf` and choose whether tests should be executed as `shell` scripts (just check the exit code) or `beakerlib` tests (investigate journal for test results): + + summary: + Check basic bash features + discover: + how: fmf + execute: + how: beakerlib + +=== Shell Test === + +In order to create a simple shell test skeleton use the shell template: + + $ tmt test create /tests/smoke + Template (shell or beakerlib): shell + Directory '/tmp/bash/tests/smoke' created. + Test metadata '/tmp/bash/tests/smoke/main.fmf' created. + Test script '/tmp/bash/tests/smoke/test.sh' created. + +Update metadata file: + + summary: Check bash version + contact: Petr Šplíchal + test: ./test.sh + +Adjust the test script as desired: + + #!/bin/sh -eux + tmp=$(mktemp) + bash --version > $tmp + grep 'GNU bash' $tmp + grep 'Free Software Foundation' $tmp + rm $tmp + +Use `tmt run` to verify the test is working as expected. + +=== BeakerLib Test === + +Use beakerlib template to create a new beakerlib test: + + $ tmt test create /tests/smoke -t beakerlib + Directory '/tmp/bash/tests/smoke' created. + Test metadata '/tmp/bash/tests/smoke/main.fmf' created. + Test script '/tmp/bash/tests/smoke/test.sh' created. + +Update test metadata and code as needed, use `tmt run` to verify everything is working fine. + +== Manage Tests == + +Explore available tests, convert old metadata, share test code. + +=== Explore Tests === + +In order to see which tests are available: + + tmt test ls + +To show more details about individual tests: + + tmt test show + +To see an overview of all metadata: + + tmt + +Explore all available options and commands using `--help`. + +=== Share Tests === + +Test code does not have to reside in the same git repository (e.g. dist git rpms namespace). +It is possible to store tests in a dedicated repository and share them across components or product versions. +You only need to reference the repository in the discover step. Use the full plan template to get quickly started: + + tmt plan create /plans/upstream -t full + +Update the repository url to point to the right place: + + summary: + Essential command line features + discover: + how: fmf + url: https://github.com/psss/tmt + execute: + how: beakerlib + +Now you will be able to run tests from the remote repository. +See the https://tmt.readthedocs.io/en/latest/spec/steps.html#discover[discover] step documentation for details. + + +== Various Hints == + +=== Multiple Commands === + +Multiple shell commands can be provided under the `script` attribute as well: + + summary: + Basic smoke test for bash + execute: + script: + - bash --version + - bash -c 'echo $((1+1+1))' | grep 3 + +See the https://tmt.readthedocs.io/en/latest/spec/steps.html#spec-steps-execute-shell[shell] method documentation for details. + +=== Multiple Repositories === + +In the discover step it is possible to reference multiple repositories as well. +In this way you can for example easily execute both upstream and fedora tests as part of a single plan: + + discover: + - name: fedora + how: fmf + repository: https://src.fedoraproject.org/tests/selinux.git + - name: upstream + how: fmf + repository: https://github.com/SELinuxProject/selinux-testsuite + +See also https://github.com/psss/tmt/blob/master/examples/multiple/basic.fmf[multiple config] example in tmt repo to get a better idea. + +=== Minimal Path === + +Here is an example of a minimal test creation path: + + dnf install -y tmt-all + git clone https://src.fedoraproject.org/rpms/bash + cd bash + tmt init -t mini + vim plans/example.fmf + tmt run + +A slightly extended example with custom test and plan template and executing test directly on the local host: + + dnf install -y tmt-all + git clone https://src.fedoraproject.org/rpms/bash + cd bash + tmt init + tmt plan create --template base plans/smoke + tmt test create --template beakerlib tests/smoke + vim plans/smoke.fmf tests/smoke/* + tmt run --all provision -h local + git add . + git commit -m "Enable basic tests" + git push + +=== Virtualization Tips === + +Make sure libvirtd is running: + + sudo systemctl start libvirtd + +Add your user account to the libvirt group. +You might need to restart your desktop session to get it fully working. + + sudo usermod -a -G libvirt $(whoami) + newgrp libvirt + +Here you can find vm https://kojipkgs.fedoraproject.org/compose/[images for download]. + + +== More Info == + +=== Test Examples === + +Example projects with tmt tests: + + * https://github.com/InfrastructureServices/bind-tests + * https://github.com/psss/tmt + * https://github.com/psss/fmf + * https://github.com/psss/did See the tmt https://tmt.readthedocs.io/en/latest/examples.html[examples] page for more inspiration. -== Links == - * https://tmt.readthedocs.io/ - * https://packit.dev/testing-farm/ +=== Links === + + * https://tmt.readthedocs.io/[Test Management Tool] + * https://fmf.readthedocs.io/[Flexible Metadata Format] + * https://packit.dev/testing-farm/[Packit Testing Farm] + == Questions == @@ -50,7 +343,7 @@ Does the tool replace/deprecate STI?:: Both `tmt` and `sti` approach to CI configuration can be used in parallel. Are these tests supported in Fedora CI?:: - Fedora CI support is coming in July/August 2020. + Fedora CI support is coming in August/September 2020. Which Linux distributions does the tool support?:: As a system under test (on which the tests are executed) all supported Fedora versions, Centos 6+ and Red Hat Enterprise Linux 6+ can be used.