Why Does Your Local Model Crash at 32k Tokens?

Automated excerpt

That second number is the KV cache, and it grows with every token you generate. Weights in fp16 cost roughly two bytes per parameter, so a 7B model comes out to about fourteen gigabytes. Same model, same GPU, same quantization setting — but a completely different memory profile, because one of the two numbers held still and the other one didn't.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.