Attention is a learned weighted average, and the cost is in the square

Automated excerpt

Two tensors in one decoder block carry T twice, [B, H, T, T]: a T × T square per head, H heads, B sequences. Scores are then [B, H, T, T], one square per head. Heads times T squared times two bytes, per layer.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.