API Docs for libtaskotron¶
Libtaskotron python library is mainly used for Taskotron runner, but it also
contains some useful helper methods which can be imported and used from tasks.
One of the most commonly used helper is the check
module for easily
managing and exporting results in the Task Result Format. But there are also
helpers e.g. simplifying interactions with Koji or Bodhi services, or
retrieving current information about Fedora’s DNF repositories layout.
There’s also a second set of helpers you can use in your tasks. They are called directives and you can read about them on the Directives Modules page.
Core modules¶
arch_utils¶
-
class
libtaskotron.arch_utils.
Arches
¶ Helper class for working with supported architectures inside taskotron
-
base
= ['x86_64', 'armhfp', 'ppc64le', 'aarch64', 'ppc64', 's390x', 'i386']¶ base architectures
-
binary
= {'aarch64': ['aarch64'], 'armhfp': ['armhfp', 'armv7hl'], 'i386': ['i386', 'i486', 'i586', 'i686'], 'ppc64': ['ppc64'], 'ppc64le': ['ppc64le'], 's390x': ['s390x'], 'x86_64': ['x86_64']}¶ a mapping from base arches to concrete binary arches
-
known
= ['aarch64', 'armhfp', 'armv7hl', 'i386', 'i486', 'i586', 'i686', 'noarch', 'ppc64', 'ppc64le', 's390x', 'src', 'x86_64']¶ all known architectures
-
meta
= ['noarch', 'src']¶ meta arches
-
-
libtaskotron.arch_utils.
basearch
(arch=None)¶ This returns the ‘base’ architecture identifier for a specified architecture (e.g.
i386
fori[3-6]86
), to be used by YUM etc.Parameters: arch (str) – an architecture to be converted to a basearch. If None
, then the arch of the current machine is used.Returns: basearch, or arch
if no basearch was found for itReturn type: str
buildbot_utils¶
Utility functions for dealing with Buildbot
-
libtaskotron.buildbot_utils.
get_urls
(jobid, masterurl, task_stepname)¶ Form url to the job and its logs based on given jobid.
Parameters: - jobid (str) – jobid from runner
- masterurl (str) – taskotron master url
- task_stepname (str) – taskotron master step name
Returns: (job_url, log_url)
-
libtaskotron.buildbot_utils.
parse_jobid
(jobid)¶ Parse the incoming jobid which should either be ‘-1’ to indicate that dummy values should be used or in the form of <builder>/<buildid>.
If the passed in format is invalid, no exceptions will be raised but a warning message will be emitted to logs
Parameters: jobid (str) – jobid from runner Returns: (builder, buildid)
check¶
Helper tools for managing check status, outcome, and output.
-
class
libtaskotron.check.
CheckDetail
(item, report_type=None, outcome=None, note='', output=None, keyvals=None, checkname=None, artifact=None)¶ Class encompassing everything related to the outcome of a check run. It contains convenience methods for tracking, evaluating and reporting check results.
For some checks, it’s trivial to parse its output at the end of its execution and evaluate the results. For some, however, it’s much easier to do this continuously during check execution. This is especially true for multi-item checks (a single check providing results for many items - builds, updates, etc). That’s when this class comes very handy (it can be used even for the simple use cases, of course).
Variables: - outcome_priority (tuple) – a tuple of
outcome
keywords sorted by priority from the least important to the most important - item (str) – a description the item being tested; for example a build NVR
(
vpnc-0.5-1.fc20
), update ID (FEDORA-2014-3309
) or a repository name (f20-updates
) - report_type (str) – a definition of the type of the object being checked;
for example a Koji build or a Bodhi update, The
allowed values are attributes in
ReportType
. You don’t have to fill this in (or you can provide a custom string value), but the reporting directives react only to the known types (you can always find it in ResultsDB, though). - outcome (str) –
a keyword specifying the final outcome of the check. Available outcome keywords:
- PASSED - everything went well
- INFO - everything went well, but there is some important information that needs to be pointed out
- FAILED - the item in question fails the check
- NEEDS_INSPECTION - the outcome can’t be determined and a manual inspection is needed
- ABORTED - the check aborted itself because of some unexpected problems, i.e. a necessary network server is not reachable. Running this check with the same arguments later can help to mitigate this problem.
- CRASHED - the check crashed and did not provide usable results
If no outcome has been set, this attribute returns NEEDS_INSPECTION.
Raises
TaskotronValueError
if you try to assign an unknown keyword. - note (str) – a few words or one-sentence note about the result of
the check run (e.g.
5 WARNINGS, 1 ERROR
for an rpmlint result). Should not unnecessarily duplicateitem
oroutcome
, if possible. - output (list) – output from the check run (a list of strings). You can easily
populate this by using
store()
, or you can modify it directly (for example filter out some messages you don’t want to see in the check output). - keyvals (dict) – all key-value pairs in this dictionary are stored in ResultsDB as ‘extra data’. This can be used to store e.g. a compose id or a kernel version, if that information is important for querying the results. Keep it as short as possible.
- checkname (str) – name of the check, which the CheckDetail belongs to. This is usually not needed, as the check name is devised from the task metadata, but if a single task produces results for multiple checks, this is the way to override the default behavior.
- artifact (str) – path to a file or directory placed in the artifacts directory,
either absolute
$artifactsdir/report.html
or relativereport.html
. It will represent task output specific for this particularitem
.
-
broken
()¶ Tells whether the check
outcome
is set to one of the broken states (i.e. ABORTED or CRASHED).
-
classmethod
cmp_outcome
(outcome1, outcome2)¶ Compare two outcomes according to
outcome_priority
and return -1/0/1 ifoutcome1
has lower/equal/higher priority thanoutcome2
.Parameters: - outcome1 (str) – an outcome keyword to compare
- outcome2 (str) – an outcome keyword to compare
Raises: TaskotronValueError – if any of the outcome keywords are not specified in
outcome_priority
-
classmethod
create_multi_item_summary
(outcomes)¶ Create a string containing a sum of all outcomes, like this:
3 PASSED, 1 INFO, 2 FAILED
Parameters: outcomes – either one CheckDetail
instance or an iterable ofCheckDetails
or an iterable ofoutcome
strings
-
store
(message, printout=True)¶ Add a string to the
output
, and print it optionally as well.This is just a convenience method for most common use case, you can of course always access and modify
output
directly, and print or use logging facilities directly as well. This combines both into a single call.Parameters: - message (str) – a string to store in
output
- printout (bool) – whether to print to standard output or not
- message (str) – a string to store in
-
update_outcome
(outcome)¶ Update
outcome
with the provided value only and only if it has a higher priority than its current value (according tooutcome_priority
). Otherwise this call does nothing.This is useful if your check performs a number of ‘sub-tasks’, each passing or failing, and you want to final outcome to be the worst/most serious one of those. You can just keep calling
update_outcome()
and the highest priority outcome type will be stored atoutcome
at the end.Parameters: outcome (str) – the outcome value to assign to outcome
if it has a higher priority than its current value. HandlesNone
values gracefully (no action).Raises: TaskotronValueError – if any of the outcome keywords are not specified in outcome_priority
- outcome_priority (tuple) – a tuple of
-
libtaskotron.check.
RESERVED_KEYS
= ('item', 'type', 'outcome', 'note', 'results', 'checkname', 'artifact', '_internal')¶ a list of reserved keywords for ResultYAML output, which can’t be overridden in keyvals
-
class
libtaskotron.check.
ReportType
¶ Enum for different types of
CheckDetail.report_type
-
BODHI_UPDATE
= 'bodhi_update'¶
-
COMPOSE
= 'compose'¶
-
DIST_GIT_COMMIT
= 'dist_git_commit'¶
-
DOCKER_IMAGE
= 'docker_image'¶
-
GIT_COMMIT
= 'git_commit'¶
-
KOJI_BUILD
= 'koji_build'¶
-
KOJI_TAG
= 'koji_tag'¶
-
MODULE_BUILD
= 'module_build'¶
-
PULL_REQUEST
= 'pull_request'¶
-
-
libtaskotron.check.
export_YAML
(check_details)¶ Generate YAML output used for reporting to ResultsDB.
Note: You need to provide all your
CheckDetail
s in a single pass in order to generate a valid YAML output. You can’t call this method several times and then simply join the outputs simply as strings.Parameters: check_details – iterable of CheckDetail
instances or single instance ofCheckDetail
Returns: YAML output with results for every CheckDetail
instance providedReturn type: str Raises: TaskotronValueError – if CheckDetail.item
orCheckDetail.outcome
is empty for any parameter providedExample output:
results: - item: xchat-0.5-1.fc20 type: koji_build outcome: PASSED note: 5 ERRORS, 10 WARNINGS artifact: xchat-0.5-1.fc20.x86_64.log
-
libtaskotron.check.
import_YAML
(source)¶ Parses YAML and returns a list of
CheckDetails
.Parameters: source (str) – YAML-formatted text Raises: TaskotronValueError – if YAML syntax is incorrect
config¶
Global configuration for Taskotron and relevant helper methods.
-
libtaskotron.config.
CONF_DIRS
= ['/home/fanys/libtaskotron/conf', '/etc/taskotron']¶ A list of directories where config files are stored. The config files are loaded from these locations in the specified order, and only the first config file found is used (further locations are ignored). The first location is dynamic, relative to the package location and it reflects the usual config location in a git checkout.
-
libtaskotron.config.
CONF_FILE
= 'taskotron.yaml'¶ the name of our configuration file
-
libtaskotron.config.
NS_CONF_FILE
= 'namespaces.yaml'¶ the name of result namespaces configuration file
-
libtaskotron.config.
PROFILE_VAR
= 'TASKOTRON_PROFILE'¶ environment variable name for setting config profile
-
libtaskotron.config.
get_config
()¶ Get the Config instance. This method is implemented using the singleton pattern - you will always receive the same instance, which will get auto-initialized on the first method call.
Returns: either Config
or its subclass, depending on taskotron profile used.Raises: TaskotronConfigError – if config file parsing and handling failed
-
libtaskotron.config.
load_namespaces_config
()¶ Load and parse namespaces config file into a dictionary.
Returns: dictionary constructed from the yaml document
-
libtaskotron.config.
parse_yaml_from_file
(filename)¶ Parse given file in YAML format into a dictionary.
Parameters: filename (str) – a filename that yaml data are loaded from Returns: dictionary constructed from the yaml document Raises: TaskotronConfigError – when YAML parsing fails
config_defaults¶
This includes the default values for Taskotron configuration. This is automatically loaded by config.py and then overridden by values from config files available in system-wide location.
-
class
libtaskotron.config_defaults.
Config
¶ Global configuration for Taskotron (development profile).
The documentation for individual options is available in the config files (unless they’re not present in the config files, then they’re documented here).
Implementation notes:
- If you want to add a new option, put it here and optionally into the config file as well.
- If you modify a default value for some option, don’t forget to modify it in both places - here and in the config file (if present).
- Don’t assign
None
as a default value. We need to know a value type in order to check for correct type of user-provided values.
-
artifacts_baseurl
= 'http://localhost/artifacts'¶
-
artifactsdir
= '/var/lib/taskotron/artifacts'¶
-
bodhi_staging
= False¶
-
buildbot_task_step
= 'runtask'¶
-
cachedir
= '/var/cache/taskotron'¶
-
client_taskdir
= '/var/tmp/taskotron/taskdir'¶
-
default_disposable_arch
= 'x86_64'¶
-
default_disposable_distro
= 'fedora'¶
-
default_disposable_flavor
= 'taskotron_cloud'¶
-
default_disposable_release
= '27'¶
-
download_cache_enabled
= True¶
-
execdb_server
= 'http://localhost:5003'¶
-
force_imageurl
= True¶
-
imagesdir
= '/var/lib/taskotron/images'¶
-
imageurl
= 'http://download.fedoraproject.org/pub/fedora/linux/releases/27/CloudImages/x86_64/images/Fedora-Cloud-Base-27-1.6.x86_64.qcow2'¶
-
koji_url
= 'https://koji.fedoraproject.org/kojihub'¶
-
log_file_enabled
= False¶
-
log_level_file
= 'DEBUG'¶
-
log_level_stream
= 'INFO'¶
-
logdir
= '/var/log/taskotron'¶
-
minion_repos
= []¶
-
pkg_url
= 'https://kojipkgs.fedoraproject.org/packages'¶
-
profile
= 'development'¶
-
report_to_resultsdb
= False¶
-
resultsdb_frontend
= 'http://localhost:5002'¶
-
resultsdb_server
= 'http://localhost:5001/api/v2.0'¶
-
runtask_mode
= 'local'¶
-
spawn_vm_retries
= 3¶
-
ssh_privkey
= ''¶
-
supported_arches
= ['x86_64', 'armhfp']¶
-
taskotron_master
= 'http://localhost/taskmaster'¶
-
tmpdir
= '/var/tmp/taskotron'¶
-
class
libtaskotron.config_defaults.
ProductionConfig
¶ Configuration for production profile. Inherits values from
Config
and overrides some. Read Config documentation.-
download_cache_enabled
= False¶
-
log_file_enabled
= True¶
-
log_level_file
= 'DEBUG'¶
-
log_level_stream
= 'INFO'¶
-
profile
= 'production'¶
-
report_to_resultsdb
= True¶
-
runtask_mode
= 'libvirt'¶
-
-
class
libtaskotron.config_defaults.
ProfileName
¶ Enum of available profile names. These can be specified in the config file or as the environment variable.
-
DEVELOPMENT
= 'development'¶
-
PRODUCTION
= 'production'¶
-
TESTING
= 'testing'¶
-
-
class
libtaskotron.config_defaults.
RuntaskModeName
¶ Enum of available runtask mode names. These can be specified in the config file.
-
LIBVIRT
= 'libvirt'¶
-
LOCAL
= 'local'¶
-
-
class
libtaskotron.config_defaults.
TestingConfig
¶ Configuration for testing suite profile. Inherits values from
Config
and overrides some. Read Config documentation.-
artifactsdir
= '/var/tmp/taskotron-test/artifacts'¶
-
cachedir
= '/var/tmp/taskotron-test/cache'¶
-
log_level_file
= 'DEBUG'¶
-
log_level_stream
= 'DEBUG'¶
-
logdir
= '/var/tmp/taskotron-test/log'¶
-
profile
= 'testing'¶
-
tmpdir
= '/var/tmp/taskotron-test/tmp'¶
-
exceptions¶
This module contains custom Taskotron exceptions
-
exception
libtaskotron.exceptions.
TaskotronConfigError
¶ All errors related to Taskotron config files
-
exception
libtaskotron.exceptions.
TaskotronDirectiveError
¶ All errors related to Taskotron directives
-
exception
libtaskotron.exceptions.
TaskotronError
¶ Common ancestor for Taskotron related exceptions
-
exception
libtaskotron.exceptions.
TaskotronImageError
¶ All generic image related errors
-
exception
libtaskotron.exceptions.
TaskotronImageNotFoundError
¶ Requested image not found error
-
exception
libtaskotron.exceptions.
TaskotronImportError
(e)¶ All issues with Extensions
-
exception
libtaskotron.exceptions.
TaskotronInterruptError
(signum, signame)¶ The execution was interrupted. This might occur in certain parts of code when e.g. SIGINT or SIGTERM signals are received.
Variables: - signum (int) – number of raised signal
- signame (str) – name of raised signal. Can be also
None
orUNKNOWN
.
-
exception
libtaskotron.exceptions.
TaskotronMinionError
¶ All errors related to persistent or disposable minion handling
-
exception
libtaskotron.exceptions.
TaskotronNotImplementedError
¶ NotImplementedError for Taskotron classes, methods and functions
-
exception
libtaskotron.exceptions.
TaskotronPermissionError
¶ Insufficient permissions or privileges
-
exception
libtaskotron.exceptions.
TaskotronPlaybookError
¶ Error in Ansible playbook file of the executed check
-
exception
libtaskotron.exceptions.
TaskotronRemoteError
(e=None, errno=None)¶ All network and remote-server related errors
-
exception
libtaskotron.exceptions.
TaskotronRemoteProcessError
(e=None, errno=None)¶ A process/command executed remotely returned a non-zero exit code, crashed or failed in a different way
-
exception
libtaskotron.exceptions.
TaskotronRemoteTimeoutError
(e=None, errno=None)¶ A remote-server did not send any output in a given timeout
-
exception
libtaskotron.exceptions.
TaskotronValueError
¶ Taskotron-specific
ValueError
executor¶
-
class
libtaskotron.executor.
Executor
(arg_data)¶ Executor executes given task in the format of Ansible playbook. Before actual execution, executor decides where the playbook will be executed according to the config and command line options: locally, on existing machine or in a disposable client.
Variables: - arg_data (dict) – processed cli arguments with some extra runtime variables
- task_vm – disposable client, instance of
TestCloudMachine
- run_remotely (bool) – whether the task is run on a remote machine
-
ACCEPTED_VARS
= ['taskotron_generic_task', 'taskotron_keepalive_minutes', 'taskotron_match_host_arch', 'taskotron_match_host_distro', 'taskotron_match_host_release']¶ variables retrieved from tests.yml
-
FORWARDED_VARS
= ['artifacts', 'taskotron_arch', 'taskotron_item_type', 'taskotron_item', 'taskotron_supported_arches', 'taskotron_supported_binary_arches']¶ variables to be exposed in the task playbook
-
execute
()¶ Execute all the tasks in the taskdir
Returns: True
if execution finished successfully for all playbooks present.False
if some of them crashed, haven’t produced any results or the execution was interrupted (e.g. a system signal).Return type: bool
file_utils¶
-
class
libtaskotron.file_utils.
Tee
(*files)¶ Helper class for writing data to different streams.
-
libtaskotron.file_utils.
download
(url, dirname, filename=None, cachedir=None)¶ Download a file.
Parameters: - url (str) – file URL to download
- dirname (str) – directory path; if the directory does not exist, it gets created (and all its parent directories).
- filename (str) – name of downloaded file; if not provided, the basename is extracted from URL
- cachedir (str) – If set, the file will be downloaded to a cache
directory specified by this parameter. If the file is
already present and of the same length, download is skipped.
The requested destination file (
dirname/filename
) will be a symlink to the cached file. This directory is automatically created if not present.
Returns: the path to the downloaded file
Return type: str
Raises: TaskotronRemoteError – if download fails
-
libtaskotron.file_utils.
makedirs
(fullpath)¶ This is the same as
os.makedirs()
, but does not raise an exception when the destination directory already exists.Raises: OSError – if directory doesn’t exist and can’t be created
image_utils¶
-
libtaskotron.image_utils.
devise_environment
(arg_data, playbook_vars)¶ Takes an input item and type, and returns a required run-environment, or a default one, if the task doesn’t require anything specific.
Parameters: - arg_data (dict) – parsed command-line arguments (item, type and arch are used in this method)
- playbook_vars (dict) – vars dict created by
executor._create_playbook_vars()
Returns: dict containing
distro
,release
,flavor
andarch
. Each either set, orNone
.
logger¶
Configure logging in libtaskotron.
There are two modes how to operate - as an external library, or as the main script runner:
- In the external library mode, we try not to change any global defaults, not touch the root logger, and not attach any handlers. The main script author should be in control of all these things.
- In the main script runner mode, we control everything - we configure the root logger, attach handlers to it, and set verbosity of this and any other library as we see fit.
-
libtaskotron.logger.
add_filehandler
(level_file=None, filelog_path=None)¶ Add a file handler.
Parameters: - level_file (int) – level of file logging as defined in
logging
. IfNone
, a default level from config file is used. - filelog_path (str) – path to the log file. If
None
, the value is loaded from config file, and it is only used when file logging is enabled in the config file (otherwise nothing happens).
Returns: created file handler or
None
if logging was not enabled in the config fileRaises: IOError – if log file can’t be opened for writing
- level_file (int) – level of file logging as defined in
-
libtaskotron.logger.
init
(level_stream=None, stream=True, syslog=False)¶ Initialize Taskotron logging.
Note: Since this touches the root logger, it should be called only when Taskotron is run as the main program (through its runner), not when it is used as a library.
Parameters: - level_stream (int) – level of stream logging as defined in
logging
. IfNone
, a default level from config file is used. - stream (bool) – enable logging to process stream (stderr)
- syslog (bool) – enable logging to syslog
- level_stream (int) – level of stream logging as defined in
-
libtaskotron.logger.
init_prior_config
(level_stream=None)¶ Initialize Taskotron logging with default values which do not rely on a config file. Only stream logging is enabled here. This is used before the config file is loaded. After that a proper initialization should take place through the
init()
method.Note: Since this touches the root logger, it should be called only when Taskotron is run as the main program (through its runner), not when it is used as a library.
Parameters: level_stream (int) – message level of the stream logger. The level definitions are in logging
. IfNone
, the default level is used (i.e.logging.NOTSET
).
-
libtaskotron.logger.
log
= <logging.Logger object>¶ the main logger for libtaskotron library, easily accessible from all our modules
-
libtaskotron.logger.
mem_handler
= None¶ our current memory handler sending logged messages to memory log prior to creating file log (after its creation, content of the memory log is flushed into the file log)
-
libtaskotron.logger.
remove_mem_handler
()¶ After adding last file handler, the memory handler is no longer needed and you should call this to have it removed and avoid filling the memory buffer with all log messages
-
libtaskotron.logger.
stream_handler
= None¶ our current stream handler sending logged messages to stderr
-
libtaskotron.logger.
syslog_handler
= None¶ our current syslog handler sending logged messages to syslog
main¶
Main class used by the runtask runner
Below is a list of examples of item
input values for different item types:
- bodhi_update: Bodhi update ID,
- e.g.
FEDORA-2017-04459ef8cf
- compose: Full URL to the compose or image,
- e.g.
http://server/Fedora-Atomic-25-20170512.0.x86_64.qcow2
- dist_git_commit: Triplet of
namespace/repo#commit
from distgit, - e.g.
rpms/gcc#fe7fce2ad1cbf374be1a1b99af27eff733f639a0
ormodules/dhcp#5c6331592b6c92bf7c52a2ac735f92cb8fc11304
- dist_git_commit: Triplet of
- docker_image: Docker registry URL,
- e.g.
candidate-registry.fedoraproject.org/f26/mongodb:0-1.f26docker
- git_commit: Quadruplet of
server/repo#branch#commit
, - e.g.
https://pagure.io/rpmdeplint#refs/heads/master#080d9599f01978ad554658c6a5da1642a383b969
- git_commit: Quadruplet of
- koji_build: Build NVR from Koji (without epoch, even if it exists),
- e.g.
cups-2.2.0-9.fc25
- koji_tag: Koji tag name,
- e.g.
f25-updates-testing-pending
- module_build: Triplet of
modulename-stream-version
, - e.g.
nodejs-f26-20170511113257
- module_build: Triplet of
- pull_request: Pull request URL,
- e.g.
https://github.com/container-images/memcached/pull/12
orhttps://pagure.io/taskotron/taskotron-trigger/pull-request/42
-
libtaskotron.main.
check_args
(parser, args)¶ Check if passed args doesn’t have conflicts and have proper format. In case of error, this function prints error message and exits the program.
Parameters: - parser (argparse.ArgumentParser) – parser object used to show error message and exit the program
- args (dict) – arguments previously returned by argument parser converted to dict
-
libtaskotron.main.
get_argparser
()¶ Get the cmdline parser for the main runner.
Return type: argparse.ArgumentParser
-
libtaskotron.main.
main
()¶ Main entry point executed by runtask script
-
libtaskotron.main.
process_args
(raw_args)¶ Processes raw input args and converts them into specific data types that can be used by tasks. This includes e.g. creating new args based on (item, item_type) pairs, or adjusting selected architecture.
Parameters: raw_args (dict) – dictionary of raw arguments. Will not be modified. Returns: dict of args with appended/modified data
os_utils¶
Utility methods related to an operating system
-
libtaskotron.os_utils.
has_sudo
()¶ Determine whether we currently have a password-less access to sudo.
Note: It’s not possible to say whether the access will stay password-less in the future (the credentials might be set to expire in time), just for this exact moment.
Return type: bool
-
libtaskotron.os_utils.
is_root
()¶ Determine whether we’re currently running under the root account.
Return type: bool
-
libtaskotron.os_utils.
popen_rt
(cmd, stderr=-2, bufsize=1, **kwargs)¶ This is similar to
subprocess.check_output()
, but with real-time printing to console as well. It is useful for longer-running tasks for which you’d like to both capture the output and also see it printed in terminal continuously.Please note that by default both
stdout
andstderr
are merged together. You can usestderr=subprocess.PIPE
, and it will be returned to you separated, but it won’t be printed out to console (onlystdout
will).The parameters are the same as for
subprocess.Popen
. You can’t usestdout
parameter, that one is hardcoded tosubprocess.PIPE
.Returns: tuple of
(stdoutdata, stderrdata)
.stderrdata
will beNone
by default (becausestderr
is redirected tostdout
).Raises: - TaskotronValueError – if you provide
stdout
parameter - subprocess.CalledProcessError – if the command exits with non-zero exit code (helpful attributes are provided, study its documentation)
- TaskotronValueError – if you provide
python_utils¶
A collection of convenience methods related to Python base libraries.
-
libtaskotron.python_utils.
cmp
(x, y)¶ Replacement for built-in function cmp that was removed in Python 3
Compare the two objects x and y and return an integer according to the outcome. The return value is negative if x < y, zero if x == y and strictly positive if x > y.
https://portingguide.readthedocs.io/en/latest/comparisons.html#the-cmp-function
-
libtaskotron.python_utils.
iterable
(obj, item_type=None)¶ Decide whether
obj
is anIterable
(you can traverse through its elements - e.g.list
,tuple
,set
, evendict
), but notbasestring
(which satisfies most collections’ requirements, but we don’t often want to consider as a collection). You can also verify the types of items in the collection.Parameters: - obj (any) – any object you want to check
- item_type (type) – all items in
obj
must be instances of this provided type. IfNone
, no check is performed.
Returns: whether
obj
is iterable but not a string, and whetherobj
contains onlyitem_type
itemsReturn type: bool
Raises: TaskotronValueError – if you provide invalid parameter value
-
libtaskotron.python_utils.
reverse_argparse
(args, ignore=())¶ Take cmdline arguments parsed by
argparse
and revert it back to a command line.Example input:{item: 'foo', debug: True, arch: ['x86_64', 'i386'], job_id: 1234, t: 'koji'}
Example output:['--item', 'foo', '--debug', '--arch', 'x86_64', '--arch', 'i386', '--job-id', '1234', '-t', 'koji']
This is a very naive implementation and has several limitations:
- You should put positional arg names into the ignore list. Handling those is not implemented because we don’t need them.
- All options need to use
action='store'
,action='store_true'
oraction='append'
. Nothing else is supported. If you use some other actions, you need to put those options into the ignore list. - All multi-word options names will be assumed to be using dashes as a separator. Be sure you do not use underscores, or convert it accordingly yourself.
Parameters: - args (dict) – arguments parsed from
argparse
converted to adict
. This is what you get by runningvars(parser.parse_args())
. - ignore (list of str) – list of option names which should be ignored. If you use any positional arguments, you need to include their variables names here as well. The ignore list is checked before the underscore-to-dash conversion takes place.
Returns: reversed command line as a list of strings. Please note the returned strings are not shell-escaped. Be sure to do that if you run it in a shell mode.
Return type: list of str
-
libtaskotron.python_utils.
sequence
(obj, item_type=None, mutable=False)¶ This has the same functionality and basic arguments as
iterable()
(read its documentation), but decides whetherobj
is aSequence
(ordered and indexable collection - e.g.list
ortuple
, but notset
ordict
).Parameters: mutable (bool) – if True
, theobj
must be a mutable sequence (e.g.list
, but nottuple
)
Disposable minion modules¶
vm¶
Interface to locally spawned virtual machines that are used as disposable clients for executing Taskotron tasks.
-
class
libtaskotron.ext.disposable.vm.
ImageFinder
¶ Retrieve images from either local or remote sources
-
classmethod
get_all_filenames
(imagesdir=None)¶ Get list of images present on the system.
Parameters: imagesdir – absolute path to directory containing the images, path from config is used if None
-
classmethod
get_latest
(distro, release, flavor, arch='x86_64', imagesdir=None)¶ Search for the most recent image available on the system.
Parameters: - distro – distro of the image (e.g.
fedora
) - release – release of the image (e.g.
23
) - flavor – flavor of the image (e.g.
minimal
) - imagesdir – absolute path to directory containing the images, path from config is used if None
- arch – architecture of the image
Returns: file:// url of the latest image available
Raises: TaskotronImageError – if no such image for given release and flavor was found
- distro – distro of the image (e.g.
-
classmethod
get_latest_metadata
(distro, release, flavor, arch='x86_64', imagesdir=None)¶ Search for the most recent image available on the system.
Parameters: - distro – distro of the image (e.g.
fedora
) - release – release of the image (e.g.
23
) - flavor – flavor of the image (e.g.
minimal
) - imagesdir – absolute path to directory containing the images, path from config is used if None
- arch – arch of the image (e.g. ‘x86_64’)
Returns: metadata of the most recent image
Return type: dict {‘date’: str, ‘version’: str, ‘release’: str, ‘arch’: str, ‘filename’: str}
- distro – distro of the image (e.g.
-
classmethod
-
class
libtaskotron.ext.disposable.vm.
TestCloudMachine
(uuid)¶ Launch virtual machines with TestCloud and prepare them for executing tasks (install packages etc.)
Parameters: uuid – unicode string uuid for the task being executed -
hostname
= None¶ hostname to use for spawned instance - based on username of current user
-
instancename
= None¶ name of the testcloud instance spawned for this task
-
ipaddr
= None¶ ip address of the remote machine (only set after prepared and spawned successfully)
-
password
= None¶ password for user on the remote machine
-
prepare
(distro, release, flavor, arch)¶ Prepare a virtual machine for running tasks. :param str distro: Distro to use in image discovery :param str release: Distro’s release to use in image discovery :param str flavor: base-image flavor to use in image discovery :param str arch: arch to use in image discovery
Raises: TaskotronRemoteError – if there are any errors while preparing the virtual machine.
-
teardown
()¶ Tear down the virtual machine by stopping it and removing it from the host machine.
Raises: TaskotronRemoteError – if there is a failure while stopping or removing the virtual machine instance
-
username
= None¶ username to use when connecting to the virtual machine
-
uuid
= None¶ uuid of the task which spawned this instance
-
wait_for_port
(port, timeout=60)¶ Wait until port is open. Repeatedly tries to socket.connect on given port.
Parameters: - port – port to check
- timeout – timeout in seconds
Raises: TaskotronInstanceError – when timeouted
-
Fedora modules¶
bodhi_utils¶
Utility functions for dealing with Bodhi
-
class
libtaskotron.ext.fedora.bodhi_utils.
BodhiUtils
(client=None)¶ Helper Bodhi methods.
Variables: client (fedora.client.Bodhi2Client) – Bodhi2 client instance Create a new BodhiUtils instance.
Parameters: client – custom Bodhi2Client
instance. IfNone
, a default Bodhi2Client instance is used.-
build2update
(builds, strict=False)¶ Find matching Bodhi updates for provided builds.
Parameters: - builds (iterable of str) – builds to search for in N(E)VR format (
foo-1.2-3.fc20
orfoo-4:1.2-3.fc20
) - strict (bool) – if
False
, incomplete Bodhi updates are allowed. IfTrue
, every Bodhi update will be compared with the set of provided builds. If there is an Bodhi update which contains builds not provided inbuilds
, that update is marked as incomplete and removed from the result - i.e. all builds frombuilds
that were part of this incomplete update are placed in the second dictionary of the result tuple.
Returns: a tuple of two dictionaries:
The first dict provides mapping between
builds
and their updates where no error occured.{build (string): Bodhi update (Munch)}
The second dict provides mapping between
builds
and their updates where some error occured. The value isNone
if the matching Bodhi update could not be found (the only possible cause of failure ifstrict=False
). Or the value is a Bodhi update that was incomplete (happens only ifstrict=True
).{build (string): Bodhi update (Munch) or None}
The set of keys in both these dictionaries correspond exactly to
builds
. For every build provided inbuilds
you’ll get an answer in either the first or the second dictionary, and there will be no extra builds that you haven’t specified.
Raises: TaskotronValueError – if
builds
type is incorrect- builds (iterable of str) – builds to search for in N(E)VR format (
-
get_update
(updateid)¶ Get the last Bodhi update for the specified update ID.
Parameters: updateid (str) – update ID, e.g. ‘FEDORA-2015-13787’ Returns: Bodhi update object with that ID, or None
when no such update is foundReturn type: munch.Munch
-
koji_utils¶
Utility methods related to Koji
-
class
libtaskotron.ext.fedora.koji_utils.
KojiClient
(koji_session=None)¶ Helper Koji methods.
Variables: session (koji.ClientSession) – Koji client session Create a new KojiClient
Parameters: koji_session ( koji.ClientSession
) – an existing Koji session instance orNone
if you want a new default session to be created-
get_build_log
(nvr, dest, arches=['all'], arch_exclude=[])¶ Download a build.log file for NVR for each specified architecture and save it in
dest
asbuild.log.<arch>
.Note: Koji cleans up old log files regularly, i.e. they might not be available for a certain NVR. This method will not raise an error if that happens (the log file can’t be downloaded). Instead, it will be simply missing in that directory and also marked in the returned object.
For parameters undocumented here, see
nvr_to_urls()
.Parameters: - nvr (str) – build NVR
- dest (str) – location where to store the log file
Returns: a dictionary of this structure:
{'ok': ['/path/build.log.x86_64', '/path/build.log.i686'], 'error': ['armv7hl'] }
which describes which build logs were correctly downloaded and for which architectures we failed to download the logs.
Return type: dict
Raises: TaskotronRemoteError – when the requested build doesn’t exist or its info can’t be retrieved
-
get_nvr_rpms
(nvr, dest, arches=['all'], arch_exclude=[], debuginfo=False, src=False)¶ Retrieve the RPMs associated with a build NVR into the specified directory.
For parameters undocumented here, see
nvr_to_urls()
.Parameters: - nvr (str) – build NVR
- dest (str) – location where to store the RPMs
Returns: list of local filenames of the grabbed RPMs (might be empty, according to your option choices and the particular NVR)
Return type: list of str
Raises: TaskotronRemoteError – if the files can’t be downloaded
-
get_tagged_rpms
(tag, dest, arches=['all'], arch_exclude=[], debuginfo=False, src=False)¶ Downloads all RPMs of all NVRs tagged by a specific Koji tag.
Note: This works basically the same as
get_nvr_rpms()
, it just downloads a lot of builds instead of a single one. For description of all shared parameters and return values, please see that method.Parameters: tag (str) – Koji tag to be queried for available builds, e.g. f20-updates-pending
-
latest_by_tag
(tags, pkgname)¶ Get the latest Koji build for the given package name in the given tag(s).
Parameters: - tags (list) – list of tags to be searched in
- pkgname (str) – name of the package
Returns: str containing NVR of the latest build, None if no build was found
-
nvr_to_urls
(nvr, arches=['all'], arch_exclude=[], debuginfo=False, src=True)¶ Get list of URLs for RPMs corresponding to a build.
Parameters: - nvr (str) – build NVR
- arches (list of str) –
restrict the arches of builds to provide URLs for. By default, all Taskotron-supported architectures are considered. If you want to consider just some selected arches, provide their names in a list.
Note
If you specify base arches (like
i386
), all concrete binary arches for that base arch will be automatically added (e.g.i[3-6]86
), because Koji query requires concrete binary arches. - arch_exclude (list of str) – exclude some specific arches, even if they are specified in
arches
or they are implicit in the default'all'
value - debuginfo (bool) – whether to provide URLs for debuginfo RPM files or ignore them
- src (bool) – whether to include a URL for the source RPM
Return type: list of str
Raises: TaskotronRemoteError – when the requested build doesn’t exist
-
rpms_to_build
(rpms)¶ Get list of koji build objects for the rpms. Order of koji objects in this list is the same as order of the respective rpm objects.
Parameters: rpms (list of str) – list of filenames as either /my/path/nvr.a.rpm
ornvr.a.rpm
Returns: list of Koji buildinfo dictionaries (as returned e.g. from koji.getBuild()
) in the same respective order as in``rpms``Return type: list Raises: TaskotronRemoteError – if rpm or it’s related build is not found
-
-
libtaskotron.ext.fedora.koji_utils.
getNEVR
(build)¶ Extract RPM version identifier in NEVR format from Koji build object
Parameters: build (dict) – Koji buildinfo dictionary (as returned e.g. from koji.getBuild()
)Returns: NEVR string; epoch is included when non-zero, otherwise omitted Raises: TaskotronValueError – if build
is of incorrect type
-
libtaskotron.ext.fedora.koji_utils.
is_debuginfo
(name)¶ Determine whether RPM is a debuginfo file or not based on its name
Parameters: name (str) – RPM name (meaning a Name header value) Returns: True
orFalse
rpm_utils¶
Utility methods related to RPM
-
libtaskotron.ext.fedora.rpm_utils.
cmpNEVR
(nevr1, nevr2)¶ Compare two RPM version identifiers in NEVR format.
Parameters: - nevr1 (str) – RPM identifier in N(E)VR format
- nevr2 (str) – RPM identifier in N(E)VR format
Returns: -1
/0
/1
ifnevr1 < nevr2
/nevr1 == nevr2
/nevr1 > nevr2
Return type: int
Raises: TaskotronValueError – if name in
nevr1
doesn’t match name innevr2
-
libtaskotron.ext.fedora.rpm_utils.
get_dist_tag
(rpmstr)¶ Parse disttag from an RPM package version string.
Parameters: rpmstr (str) – string to be manipulated in a format of N(E)VR ( foo-1.2-3.fc20
orbar-4:1.2-3.fc20
) or N(E)VRA (foo-1.2-3.fc20.x86_64
orbar-4:1.2-3.fc20.i686
)Returns: string containing dist tag ( fc20
)Raises: TaskotronValueError – if rpmstr
does not contain dist tag
-
libtaskotron.ext.fedora.rpm_utils.
install
(pkgs)¶ Install packages from system repositories using DNF. Either root or sudo access required.
Parameters: pkgs (list of str) – packages to be installed, e.g.
['pidgin']
, or any other argument supported bydnf install
commandRaises: - TaskotronPermissionError – if we don’t have permissions to run DNF as admin
- TaskotronError – if DNF return code is not zero
-
libtaskotron.ext.fedora.rpm_utils.
rpmformat
(rpmstr, fmt='nvr', end_arch=False)¶ Parse and convert an RPM package version string into a different format. String identifiers: N - name, E - epoch, V - version, R - release, A - architecture.
Parameters: - rpmstr (str) – string to be manipulated in a format of N(E)VR
(
foo-1.2-3.fc20
orbar-4:1.2-3.fc20
) or N(E)VRA (foo-1.2-3.fc20.x86_64
orbar-4:1.2-3.fc20.i686
) - fmt (str) – desired format of the string to be returned. Allowed options
are:
nvr
,nevr
,nvra
,nevra
,n
,e
,v
,r
,a
. If arch is not present inrpmstr
but requested infmt
,noarch
is used. Epoch is provided only when specifically requested (e.g.fmt='nevr'
) and being non-zero; otherwise it’s supressed (the only exception isfmt='e'
, where you receive0
for zero epoch). - end_arch (bool) – set this to
True
ifrpmstr
ends with an architecture identifier (foo-1.2-3.fc20.x86_64
). It’s not possible to reliably distinguish that case automatically.
Returns: string based on the specified format, or integer if
fmt='e'
Raises: TaskotronValueError – if
fmt
value is not supported- rpmstr (str) – string to be manipulated in a format of N(E)VR
(
yumrepoinfo¶
A wrapper object for yumrepoinfo.conf to access its information easily
-
class
libtaskotron.ext.fedora.yumrepoinfo.
YumRepoInfo
(arch=None, filelist=None, resolve_baseurl=True, resolve_retry=3)¶ This class is a wrapper for easily accessing repoinfo.conf file.
Parameters: - arch (str) – architecture for which to adjust repo URLs. By default it refers to the architecture of the current machine. It’s always converted to basearch.
- filelist (iterable of str) – list of config files to read information from. The
first available config file is used. If
None
, then the default list of locations is used. - resolve_baseurl (bool) – if baseurl is a known redirect, resolve
it for each section during initialization. If this is
False
, you must call_switch_to_mirror()
manually. - resolve_retry (int) – how many tries to retry resolving the URL for each section in case the network request fails
Raises: - TaskotronConfigError – if no YUM repositories data is found (empty
or non-existent config file). It’s not
raised if you specifically request no data
to load (
filelist=[]
). - TaskotronRemoteError – if url resolving fails
-
arches
(reponame)¶ Get a list of all supported (primary and alternate) architectures for a repo
Parameters: reponame (str) – repository name
-
branched
()¶ Get branched Fedora release (or None if it doesn’t exist).
Return type: str or None
-
get
(reponame, key)¶ Get a specific key value from a repo
Parameters: - reponame (str) – repository name
- key (str) – name of the key you want to retrieve from a repository (section)
Raises: TaskotronConfigError – if the key can’t be retrieved (e.g. wrong key or repo name)
-
release_status
(reponame)¶ Return release status of specified repo. For non-top-parent repos, return release_status of top parent repo.
Parameters: reponame (str) – repository name Returns: release status of specified repo, lowercased. One of: rawhide, branched, stable, obsolete. Return type: str
-
releases
()¶ Get the list of stable (supported) Fedora releases.
Return type: list of str
-
repo
(reponame)¶ Given a repo name, return the yumrepoinfo dict with keys:
arches
,parents
,tag
,url
,path
andname
Parameters: reponame (str) – repository name Return type: dict
-
repo_by_tag
(tag)¶ Given a Koji tag, return the corresponding repo dict.
Parameters: tag (str) – a koji tag, e.g. f20-updates
. Note:rawhide
is not used as a Koji tag, use number identifier instead or userepo('rawhide')['tag']
to discover it first.Returns: repo dict as from repo()
, orNone
if no such Koji tag is found
-
repos
()¶ Get the list of all known repository names.
Return type: list of str
-
top_parent
(reponame)¶ Go through the repo hiearchy and find the top parent for a repo
Parameters: reponame (str) – repository name Returns: the top parent reponame. If reponame
doesn’t have any parent, its name is returned (it’s its own top parent)Return type: str Raises: TaskotronConfigError – if infinite parent loop detected
-
libtaskotron.ext.fedora.yumrepoinfo.
get_yumrepoinfo
(arch=None, filelist=None)¶ Get YumRepoInfo instance. This method is implemented using the singleton pattern - you will always receive the same instance, which will get auto-initialized on the first method call.
Parameters: - arch (str) – architecture to return the YumRepoInfo for. It’s always
converted to basearch. If
None
, then local machine arch is used. - filelist – list of config files to read information from. The
first available config file is used. If
None
, then the default list of locations is used.
Returns: shared
YumRepoInfo
instanceRaises: TaskotronConfigError – if file config parsing and handling failed
- arch (str) – architecture to return the YumRepoInfo for. It’s always
converted to basearch. If