Fine-tuning is the wrong tool for knowledge: three training phases and the low-rank shortcut

Automated excerpt

The model completed three of the real ones word for word. The full fine-tune learned all twelve facts at every rate. If you arrived here directly, Part 1 reads the tokenizer, Part 2 the table a token id indexes, Part 3 the shapes, Part 4 attention, Part 5 the block, Part 6 GPT-2 and its two programs, Part 7 what Llama changed, Part 8 where gradients come from, and Part 9 what the optimizer does with them.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.