PR #4056

I spent some time trying to replicate this under controlled conditions and so far have not been able do, despite that fact that it happens in our instance a few times per day.

Here is what I do know about it though:

ONE. The tracebacks are always in either host.taskWait or host.getHostTasks calls.

TWO. In both types of tracebacks, we're failing in taskWaitCheck()

THREE. The other side of the deadlock is happening in Task.verifyHost. We know this because of the db logs. E.g.

        Process 3815840 waits for ShareLock on transaction 1649461470; blocked by process 3818796.
        Process 3818796: 
        SELECT id, state
          FROM task
         WHERE (awaited IS TRUE)
           AND (parent=60222253)
         FOR UPDATE
        Process 3815840: 
        SELECT host_id, state
          FROM task
         WHERE (id=60222321)
         FOR UPDATE

The latter query can only be coming from Task.verifyHost.

FOUR. Task.verifyHost can be called many different ways. E.g. most host calls hit it via assertHost. However, based on the timing of errors I've looked at, I'm pretty sure the other side of the deadlock is the scheduler assigning a task.

Case in point, I observed a build task failing on this error just after creating its buildArch subtasks. None of them had started yet. So, really the only activity that could have conflicted is assignment by the scheduler. Scheduler logs for the assignments seem to line up with the task failure time.

The scheduler can assign any number of tasks in a given run. It does follow a predictable order, but that likely not the same order that taskWaitCheck is locking.

So it seems like I should be able to replicate this locally, but I so far have not been able to.

I've tried replicating it as well, with no luck so far.

Ok, I was able to replicate this with a bit of tweaking.

This branch has the adjustments:
https://pagure.io/fork/mikem/koji/commits/replicate-wait-deadlock

• add a short sleep on each scheduler assignment
• add some new options to fakehub to facilitate replication

My approach did not use actual kojid or kojihub, but simulated all activity with fakehub against my local test db.

First, make the tasks

devtools/fakehub -u mikem -o LogLevel=WARNING --pdb makeTask sleep '[5]'
devtools/fakehub -x -u builder-01 -o LogLevel=WARNING host.openTask 13924
devtools/fakehub -u mikem -o LogLevel=WARNING --pdb makeTask sleep '[5]' parent=13924
devtools/fakehub -u mikem -o LogLevel=WARNING --pdb makeTask sleep '[5]' parent=13924
...
...make several subtasks to increase the chances (I made 10)...

Mark the subtasks awaited

devtools/fakehub -x -u builder-01 host.taskSetWait 13924 '[13925, 13926, 13927, 13928, 13929, 13930, 13931, 13932, 13933,13934,]'

Make sure the hosts show as ready

devtools/fakehub -x -u builder-01 -o LogLevel=WARNING host.updateHost 0 True
devtools/fakehub -x -u builder-02 -o LogLevel=WARNING host.updateHost 0 True

Trigger the scheduler in a loop without commit

while devtools/fakehub -o RunInterval=0 -o LogLevel=INFO --no-commit -x -u builder-02 host.getLoadData; do echo; done

In parallel, have the builder check for tasks without commit

while ./devtools/fakehub --no-commit -x -u builder-01 host.getHostTasks; do echo ; done

On my machine, this generally hits the deadlock within a few repetitions.

If I merge #4056 on top of this, the error disappears.

This replication setup keeps the scheduler in a constant state of trying to assign a bunch of subtasks. Note that because no actual builder is actually running, the scheduler will eventually start ignoring them due to the ReadyTimeout setting, so if testing goes on a while, you may need to rerun those host.updateHost calls or override ReadyTimeout for the host.getLoadData call.

Is there a way to incorporate the fakehub changes into the regular hub/builder? Even in a temporary side branch would be fine. We do not use the fakehub utility in QE env and I'm not sure if I would be able to create a test case without it.

Commit df86feab relates to this ticket

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/4057

Please continue any further discussion there.