Breaking the 1.58-bit Barrier for Ternary LLMs

Automated excerpt

Abstract:Ternary Large Language Models (LLM) store every weight as one of three symbols $\{-1,0,+1\}$, so the cost of a ternary model is conventionally referenced to the information-theoretic $\log_2 3 \approx 1. 585$ bits per weight. The prevailing deployment format packs five ternary weights into one byte (five-trit packing), and due to the power-of-two group sizes used in practice this rounds up to $1. 625$ bits per weight. BITCOS stores weights more compactly than the five-trit packing in 26 of the 29 tested models, and reaches $1. 485$ bits per weight on the sparsest of them.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.