AI article
My Gradient Clipping Was Never Clipping. It Was Secretly Multiplying My Weight Decay
Community description: Yesterday warmup turned out to be free insurance against a hot learning rate, so today I asked whether gradient clipping — the other famous
Dev.to | Nov 4, 2026 | b1ackjack12
Automated excerpt
Gradient clipping has the exact same folklore job description. At the hot rate, where warmup recovered +2.86, clipping deepens the hole by another −3.43. The clipping arms weren't measuring clipping.
Selected automatically from source text; not independently written or fact-checked. Read the original for full context.