Deep Dive into Mixture of Experts: From 1991 to DeepSeek-V3

Automated excerpt

Instead of one giant feed-forward network per layer, an MoE layer holds many smaller so-called "expert" networks, and a learned gate activates only a handful of them per token per layer. Instead of every expert computing on every input, the gate selects only the top-k highest-scoring experts (4/4096 in this scenario), and only those experts actually run. Each MoE layer holds 256 routed experts plus 1 shared expert, and the gate selects the top 8 routed experts per token, so every token is processed by 9 experts total out of 257 available in that layer.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.