What training actually does to an LLM

Automated excerpt

Llama 2 (2023) used AdamW with beta1 = 0. 9, beta2 = 0. 95, weight decay 0. 1, gradient clipping at 1. 0, and was pretrained on 2 trillion tokens. Cauchy 1847 first suggested gradient descent (Wikipedia, "Gradient descent"). GPT-3 (Brown et al. , arXiv:2005. 14165): 175B parameters, 96 layers, d_model 12288 (Table 2. 1); batch size 3. 2M tokens, learning rate 6e-5 (Table 2. 1); 300B tokens total for all models (Table 2. 1 caption); Adam beta1 0. 9, beta2 0. 95, epsilon 1e-8, gradient norm clipped at 1.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.