Multi-Turn Memory, Mixed Precision, and Safer Rollouts — Serving Catches Up to Agentic Workloads

Automated excerpt

DeepSeek-V4. 1-Flash on vLLM: 5x Agentic Throughput Since Day 0 — three kernel tricks, 5. 3x throughput under TPS limits Vesta: Queueing-Aware LLM Scheduling for Real-World Workloads — balances cache locality vs. queueing, cuts TTFT sharply Run fast, private agents on-device with LFM2. 5-24B-A2B — sparse MoE hits 21K tok/s on one GPU MultiAlloc: Budget-Aware Precision Allocation for KV Cache Quantization (2026-10-13): splits a fixed KV quantization budget between layer-wise precision boosts and token-wise preservation, using lightweight offline profiling instead of uniform compression. Relevant for anyone pushing long-context serving memory limits. At 500K context and concurrency 512, reports 1.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.