AI article
When Should an LLM Router Stop Being Sticky? llm-d 1.0 Cache-Affinity Routing, Break-Even Math, and a Simulation
Community description: llm-d 1.0 routes by prefix-cache affinity until a token-load limit. Break-even math, a 10-endpoint simulation, and how to calibrate your own limit.
Dev.to | Nov 2, 2026 | AI Tech News
Automated excerpt
Take one request whose prefix of P tokens is cached on endpoint A and absent from endpoint B. Always-sticky ties each prefix to one endpoint. In the published setup, 8,192 tokens take 0.40 s, so one endpoint prefills 20,480 tokens/s.
Selected automatically from source text; not independently written or fact-checked. Read the original for full context.