You could have invented transformers from memory

Automated excerpt

The $M$ interference terms add up like a random walk, so the interference-to-signal ratio is$$\frac{\norm{\text{interference}}}{\norm{\text{signal}}} \approx \sqrt{\frac{M}{d_k^p}}. $$This means the capacity is about$$M \approx d_k^p$$memories, geometrically more than before! To check this, we run the same experiment as before for five choices of $\sigma$:recall fails: more interference than signal0. 010. 111016642561024409616384number of stored memories M (key dimension dk = 64)interference / signals (linear), M = 16: interference/signal = 0. 46s (linear), M = 32: interference/signal = 0. 68s (linear), M = 64: interference/signal = 0. 97s (linear), M = 128: interference/signal = 1.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.