Recurrent Looped Transformer

Automated excerpt

Causal encoderKnown-token parallelism · global KV memoryRecurrent decoderEncoder cross-attention · local decoder SWA Carry forward: recurrent output + decoder SWA KVThe previous output enters the next merge. Encoder memory is prefix-restricted; decoder attention respects its local window. The temporal path traverses \(48t\) decoder blocks after \(t\) tokens. Decoder updates still proceed in token order, constructing both recurrent outputs and decoder SWA caches. Full BPTT includes paths through recurrent outputs, decoder KV, and encoder memory.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.