Why Qwen2 Was Running at Half Speed

Automated excerpt

In candle-transformers, the quantized Llama backend routes single-token Metal decode steps through Candle's fused Scaled Dot-Product Attention (sdpa) kernel, which computes Grouped-Query Attention (GQA) directly over the original, unexpanded key/value heads. Speeds up isolated Metal decode attention by 4.1×–10.3× per-layer on Qwen2.5-0.5B and 6.3×–15.2× per-layer on Qwen2.5-7B. End-to-End Decode Scaling by Context Length As the KV cache grows, attention accounts for an increasing share of total decode latency.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.