A transformer block is six tensors and a bus: read it like a pipeline

Automated excerpt

The shapes in the margin come from running a model with GPT-2 small's shape, d = 768 and 12 heads: qkv = self. c_attn(x) # [B, T, 2304] q, k, v side by side q, k, v = qkv. split(768, dim=2) # three views, [B, T, 768] each q = q. view(B, T, 12, 64). transpose(1, 2) # [B, 12, T, 64] 12 heads of 64; k, v alike att = (q @ k. transpose(-2, -1)). softmax(-1) # [B, 12, T, T] Part 4's square y = (att @ v). transpose(1, 2). reshape(B, T, 768) y = self.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.