Hybrid-precision attention reduces compute cost with minimal accuracy loss

Automated excerpt

Mixed‑precision quantization can halve the compute cost of LLM attention while keeping accuracy loss below 1 %. By preserving only a narrow set of critical tokens in full precision, HyQuant sidesteps the catastrophic degradation that plagued earlier low‑bit attempts. HyQuant delivers between 1. 32× and 3. 58× decode‑kernel speedup while preserving near‑full‑precision accuracy across multiple long‑context and reasoning benchmarks — "Experimental results show that HyQuant achieves 1. 32 to 3. 58 decode‑kernel speedup and 1. 04 to 1. 17 end‑to‑end decode speedup while maintaining near‑full‑precision accuracy across multiple long‑context and reasoning benchmarks.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.