From GPT-2 to Llama: four changes and the reason for each

Automated excerpt

The wider feed-forward alone, with full attention heads, gives 72%. GPT-2 has as many key and value heads as query heads. SmolLM2 has 9 query heads and 3 key-value heads; each key and value head serves a group of three queries.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.