Hybrid attention buys cheap reading and expensive writing

Automated excerpt

A hybrid-attention coder model advertises a two-million-token context window, and on my two RTX 3090s it ingested at 1,013 tokens per second while generating at 20. At 53.74 ms per token, twenty tokens costs 1.07 seconds and five hundred costs 26.9. Feed an agent loop ten thousand tokens of context every night and you pay roughly ten seconds of ingestion per 9.6k block while generation stretches into hours.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.