Beyond Static Layers: The Architecture of Mixture-of-Depths (MoD)

Automated excerpt

Mixture-of-Depths (MoD): Optimizing compute efficiency by dynamically routing tokens across variable network depths. For years, standard transformer architectures have processed input tokens uniformly. While traditional Mixture-of-Experts (MoE) routes tokens across different specialized parameters at a fixed layer, Mixture-of-Depths routes tokens across different depths of the network.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.