Simple helper: https://pagure.io/koji-tools/pull-request/13

While it would probably be good to have some sort of process to clean these up after the fact. Perhaps either gc or kojira could eventually grow in scope to do other sorts of maintenance work like thjis, I wonder if we could simply prevent this situation from happening in the first place.

Do we have an understanding of the sorts of chain of events that leads to this situation? There aren't really that many places where we cancel tasks in the code:

• manual cli command
• manual web ui action
• cancel_build also cancels the controlling task by default
• the runroot cli plugin cancels the runroot task if interrupted with ^C
• the wait() function in koji.tasks can call cancelTaskChildren when a parent task fails or times out

It is really only the last one I can see being the issue here. Perhaps task cancellation should also cancel a linked build?

I suspect the failBuild method. For cases I've found possible scenario is this:

1. builder A starts build, spawns buildArchs
2. download server quits, so one buildArch fail
3. builder fails parent task correctly
4. hub quits
5. builder wants to completeBuild, but can't reach hub
6. builder quits due to ServerExit, so failBuild is also not called (and wouldn't work as hub is already down)
7. After restart is everything forgotten as tasks are already failed

It is rare race condition, but with amount of builds we've now it is not as rare (about 7 cases last restart time).

Commit a8b57118 relates to this ticket

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/972

Please continue any further discussion there.