Slang Is a Dial: Training a 125M Language Model From Scratch on Free GPUs

Automated excerpt

The idea is simple and testable: measure the register of every training document with code, write it as five control tokens in front of the text, and the model learns p(text | register). The register tokens are real vocabulary: 34 special tokens with fixed ids, including end-of-text, padding, register open and close, and every dimension level, so slang level 3 is always token 7. The model wrote 48 tokens from the prefix alone and the annotator measured the output.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.