Transformers with an Infinite Memory Span

The primary technical contribution of the paper is the implementation of segment-level recurrence. In this framework, the hidden states computed for the previous segment are cached and utilized as an extended context for the current segment. During the forward pass, the attention mechanism for each layer integrates both the local hidden states and the frozen states from the preceding block. This mechanism allows the information to propagate across segment boundaries, effectively creating a temporal memory that spans multiple computational windows. This methodological choice proved that the modeling of long-range relationships is a function of state reuse rather than the absolute size of the training segment.

To facilitate the reuse of hidden states across segments, the researchers introduced a relative positional encoding scheme. Standard positional encodings are tied to absolute indices, causing the model to lose temporal coherence when hidden states from a previous segment are shifted into the current window. The new scheme instead encodes the distance between tokens, ensuring that the attention bias remains consistent regardless of the segment's starting point. This finding demonstrated that a model's perception of sequence structure can be made invariant to absolute coordinates, allowing the attention mechanism to generalize to sequences significantly longer than those encountered during the training phase.

The technical significance of Transformer-XL is evidenced by its performance on large-scale language modeling benchmarks such as WikiText-103 and enwik8. By capturing dependencies spanning thousands of tokens, the architecture achieves a higher degree of narrative coherence compared to models restricted by rigid context windows. Furthermore, the recurrence mechanism eliminates the need for redundant computations during evaluation, as the model does not need to re-process overlapping segments to predict the next token. This application proved that the scalability of language models is determined by the efficiency of their memory management strategy.

The success of this research established that the management of temporal state is a primary constraint on the capacity of attentive systems. The decision to implement recurrence within the Transformer framework revealed that the bottleneck in sequence modeling was the structural isolation of information blocks. This principle remains central to the development of modern context-scaling techniques, including the large-context windows found in foundation models like Gemini and GPT-4. It leaves open the question of how these recurrent mechanisms can be optimized for sub-quadratic complexity or if there exists a fundamental threshold where memory compression becomes a prerequisite for further expansion.