Postgres SELECT DISTINCT Does Not Scale

Automated excerpt

So this query finds all unique non-NULL partition keys among ENQUEUED workflows on a particular queue. Most of our queue workloads were “wide but shallow” with many partitions but few enqueued workflows per partition. Therefore, because each loop iteration does fixed work and the total number of loop iterations is equal to the number of unique partitions, this query provides the O(number of partitions) performance we need. To validate this performance, we benchmark the new query, fixing the number of partitions at 10 but varying the number of rows per partition from 1K to 1M.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.