Expert pruning improves decode throughput up to 1.8 with minimal quality loss

Automated excerpt

SlimWise shows that pruning experts only in the decode phase and reusing prefill KV caches can boost throughput by nearly twofold without perceptible quality loss. Without additional training, SlimWise’s KV cache handoff recovers much of the accuracy lost to pruning in many evaluated settings[1]. Because expert pruning preserves the attention architecture and KV cache format, this KV cache handoff requires no KV cache conversion[1].

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.