AI article
C = 1.25 is a tolerance, not a constant: the load-tail law behind MoE token dropping
Community description: A capacity factor and a target drop rate are the same information in two forms, and the folk constant names only one of them. The exact drop functional, the ceiling balancing cannot reach, which tokens actually get dropped, and one command that reproduces every number.
Dev.to | Oct 16, 2026 | howcani howcani
Automated excerpt
You need a static shape for the kernel, so each expert gets a budget c = C · topk · T / E, and C — the capacity factor — arrives as folklore: 1. 25 from Switch Transformer, 2. 0 from GShard. Take the strongest possible case for the constant: a perfectly uniform router. At training load (µ = 128) the same constant is fine — 8. 29e-05 drops.
Selected automatically from source text; not independently written or fact-checked. Read the original for full context.