AI article
Running an MoE Model Larger Than Your VRAM: SGLang Paged Experts, the PCIe Math, and the 4-Request Cliff
Community description: SGLang 0.5.23 adds Paged Experts for MoE models that do not fit in VRAM. Measured speeds, the PCIe math behind them, and where batching breaks it.
Dev.to | Nov 7, 2026 | AI Tech News
Automated excerpt
What the PR measured: on Qwen3-30B-A3B, single-stream decode of 205.6, 87.7 and 32.6 tok/s (int4 on a 16 GB card, FP8 on a 16 GB card, int4 on an 8 GB laptop), against 16.6, 2.9 and 1.2 for --cpu-offload-gb. A token at batch 1 activates 48 x 8 = 384 experts. The FP8 run does 87.7 tok/s, so 11.4 ms per token.
Selected automatically from source text; not independently written or fact-checked. Read the original for full context.