This ticket is a duplicate of https://pagure.io/taskotron/issue/211

This ticket had assigned some Differential requests:
D1135

It's been happening on prod as well :/

! In #905#12966, @mkrizek wrote:
It's been happening on prod as well :/

Yeah, I've been trying to clean off the prod virthosts by hand to keep them running

So, the problem seems to be in this testcloud code (instance.py:_list_system_domains()):
```lang=python
for domain in conn.listAllDomains():
# the libvirt docs seem to indicate that the second int is for state
# details, only used when state is ERROR, so only looking at the first
# int returned for domain.state()

    domains[domain.name()] = DOMAIN_STATUS_ENUM[domain.state()[0]]

There's a race condition. Since many VMs are running and being destroyed at the same time, a domain foo might exist during `listAllDomains()`, but might no longer exist during `domain.state()`. Which raises a libvirt error:

libvirtError: Domain not found: no domain with matching uuid ...

which in turn crashes testcloud, which aborts our teardown method in libtaskotron. Thus the task VM (which is a different one from the one "missing") stays active.
As an example, here's a task 1 that couldn't spawn a VM properly, but testcloud cleaned all the files just fine:
http://taskotron-dev.fedoraproject.org/taskmaster/builders/x86_64/builds/317643/steps/runtask/logs/taskotron-overlord.log

[testcloud.instance:instance.py:468] 2017-02-17 12:21:35 DEBUG removing instance /var/lib/testcloud/instances/taskotron-6e5e695c-efb8-11e6-97cb-525400cb0b45 from disk

Notice the timestamp and the uuid. Now here's a task 2 that crashed because VM for task 1 was removed during listing system domains:
http://taskotron-dev.fedoraproject.org/taskmaster/builders/x86_64/builds/317644/steps/runtask/logs/taskotron-overlord.log

[testcloud.instance:instance.py:446] 2017-02-17 12:21:34 DEBUG removing instance taskotron-6e68700a-efb8-11e6-ba5c-525400cb0b45 from libvirt.
[libtaskotron:minion.py:257] 2017-02-17 12:21:35 ERROR Domain not found: no domain with matching uuid 'be18ca20-1e68-4e67-9c1b-b402674d6f9b' (taskotron-6e5e695c-efb8-11e6-97cb-525400cb0b45)
```
Notice the timestamp (the very same second) and also the uuid (it matches task 1's uuid). As a result, the VM for task 2 was not properly stopped and removed.

[[ https://koji.fedoraproject.org/koji/taskinfo?taskID=17919265 | Testcloud scratch build]] deployed to dev, seems to be working well (it's hard to say whether the race condition was fixed).

testcloud-0.1.11 is in Bodhi, we'll update dev and stg once it hits updates-testing

We still see some zombie VMs around. I found another cause - in [[http://taskotron-dev.fedoraproject.org/taskmaster/builders/x86_64/builds/384547 | this job]] buildbot killed the task for taking more than 40 minutes to run:

command timed out: 2400 seconds without output running ['runtask', '-i', 'java-1.8.0-openjdk-1.8.0.121-5.b14.fc24', '-t', 'koji_build', '-a', 'x86_64', '-j', 'x86_64/384547', '--uuid', 'e702f95e-f7e2-11e6-8068-525400cb0b45', 'runtask.yml'], attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=2810.013207

But that only kills runtask, and the VM is kept running. I have no idea why that happens, because in libtaskotron we have a timer for 15 minutes that should kill the VM if no output is received. But as a workaround, let's add this buildbot step to all jobs:

$ testcloud instance remove --force taskotron-$uuid; true

Also, we should put such packages on abipkgdiff blacklist, here's a list:

mariadb
java-1.8.0-openjdk
libreoffice

New testcloud, new buildbot step and extended abicheck blacklist has been deployed to dev and stg.

Everything has been deployed to production. Things are fixed now, hopefully.