From Attention to a Working Language Model

Automated excerpt

To get all the query-key comparisons (every token scored against every other token), you just multiply , the relevance of token j to token i. There's an embedding matrix — one row per vocabulary token. The input is token + position embeddings.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.