Half Your Prefill Compute Can Go to 1% of Requests

Automated excerpt

The top 1% of requests, by token count, carries 46.4% of all prefill tokens. I ran 30 simulated days of 100,000 requests each: α = 2.27: plan 457 tokens, worst day +1%, best day -1%. α = 1.2: plan 1,451 tokens, worst day +22%, best day -15%. The top 1% (10,000 requests): 0.71 billion tokens, or 1.58 node-days.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.