AI article
Serving Claude Code from DeepInfer on 4x B200: the cache, the tail and the memory ceiling
Community description: DeepInfer ships without a prefix cache. With one, a 4x B200 box read 1.8M tokens/s for 100 agents. What to build, what broke, what to measure.
Dev.to | Nov 5, 2026 | InferenceKey
Automated excerpt
With no cache, our four cards got through about 200K tokens per second, starting each conversation from zero. At 500K tokens per conversation, we had 200 agents ask at once. At two agents per power user, 400 agents is about 200 subscriptions.
Selected automatically from source text; not independently written or fact-checked. Read the original for full context.