#90 optimize table joins for faster counting performance
Merged by mikem. Opened by rayson.
rayson/koji optimize_joins  into  master

Download 90.patch

It can be very slow to count a large data set. I tried different ways to optimize the counting performance, but a lot of them introduced much complexity.
I found we could significantly improve the counting performance by removing redundant table joins and/or replacing inner joins with left outer joins.

With this approach, the query times reduced from dozens of seconds to milliseconds on a PostgreSQL 9.2 server: https://i.imgur.com/f3v4E4J.png

Versions of PostgreSQL prior to 9.2 don't support for index-only scans. So if we use a PostgreSQL 8.4 server, the performance improvement will not be so obvious: https://i.imgur.com/HNkNypM.png

rebased

This looks reasonable. All the joins you have converted are on non-null fields that reference the primary key of the joined table, so the query result should not change.

It seems to me that the only reason this would be faster is that postgres is optimizing out the join itself (since a left outer join would not affect the count if the joined fields are not part of the query condition).

If I'm right, then there may not be much reason to omit the join in listTasks.

OTOH, if the benefit is happening some other way, then we might well want to selectively omit more joins when we're counting and not querying on the joined fields.

Maybe you are right. Whether omiting the join or not didn't make much difference from my simple tests. But I didn't do further tests to prove the hypothesis.

For postgresql-server-8.4, the difference of the inner join and left outer join doesn't make sense. The planer report on postgresql-server-8.4 is here.

But for postgresql-server-9.2, the difference of the inner join and left outer join does make sense when calling the 'count' aggregate function. When using the left outer join, the postgres only exec a sequential scan on 'build' table, but when using the inner join, the postgres will run 5~6 sequential scans and a index scan. The planer report is here.

And mikem is right. Omitting join in listTasks doesn't make sense when calling the 'count' aggregate function. The planer report is here.

@rayson would please update this PR?

rebased

@mikem @xning
Thanks. I've updated my code.

Commit de4b1414 fixes this pull-request

Pull-Request has been merged by mikem@redhat.com

Metadata