AI article
Decode Speed Is Bytes Divided by Bandwidth, and the Batch Ceiling Is 848 Tokens
Community description: At batch 1 decode is a traffic budget. Here is the formula for three models on an H100, the context past which batching stops helping, and why quantizing everything moves the wall without moving the distance to it.
Dev.to | Nov 7, 2026 | Jax Sinclair
Automated excerpt
The KV cache has to be read once per token, per sequence. The KV term is 2 * layers * kv_heads * head_dim * 2 bytes per token of context, for K and V. Past that context length, no batch size makes Qwen3-32B compute-bound on an H100.
Selected automatically from source text; not independently written or fact-checked. Read the original for full context.