Enformer: Long-Range Attention for Gene Regulation

In human and mammalian genomes, gene expression is rarely determined solely by the promoter adjacent to the transcription start site. Distal enhancers, silencers, and insulator elements loop across 3D nuclear space to regulate genes over distances exceeding 100,000 base pairs. Convolutional networks struggle with these long-range regulatory spans: increasing convolutional depth linearly increases parameter counts and causes training instability, while aggressive pooling erodes single-nucleotide spatial resolution. As a result, earlier models missed distal enhancer-promoter communication, limiting their ability to predict the regulatory consequences of non-coding genetic variants.

Enformer addresses this physical scale disparity through a specialized three-stage pipeline:

1. Convolutional Tower: Processes an input window of 196,608 base pairs through 7 residual convolutional blocks with max pooling. This compresses the raw sequence into 1,536 spatial bins of 128 base pairs each, extracting local motif features while lowering sequence length for attention processing.
2. Transformer Tower: Passes the 1,536 bins through 11 multi-head self-attention blocks. Self-attention provides a bidirectional context span of approximately 100 kb in each direction, allowing any regulatory element in the window to attend directly to distant promoters without relying on intermediate convolutional layers.
3. Relative Positional Encoding: Utilizes distance-dependent basis functions within the attention mechanism, providing an inductive bias that reflects how enhancer influence naturally decays over linear sequence distance.
4. Cropping and Output Heads: To avoid edge artifacts, the outer 320 bins are cropped, leaving the central 896 bins (114,688 base pairs). Linear heads predict 5,313 human and 1,643 mouse functional tracks, covering CAGE transcription start sites, ChIP-seq histone marks, and DNase/ATAC-seq chromatin accessibility.

The network is trained using a Poisson negative log-likelihood loss on binned experimental sequencing read counts.

Enformer delivered a substantial improvement in predicting gene expression for genes controlled by distant regulatory elements. On genes with enhancers located farther than 20 kb from their transcription start sites, Enformer improved the Pearson correlation ($r$) of predicted expression by over 40% compared to convolutional baselines.

Crucially, Enformer demonstrated accurate zero-shot variant effect prediction. Without receiving direct training on genetic variant data, the model accurately prioritized causal non-coding single-nucleotide variants linked to expression quantitative trait loci (eQTLs) and complex disease traits, correctly identifying which non-coding mutations disrupt enhancer-promoter loops and modify gene expression.

Enformer proved that understanding eukaryotic gene regulation requires pairing local pattern recognition with long-range attention mechanisms. By capturing interactions spanning hundreds of thousands of base pairs, Enformer demonstrated that linear sequence models can internalize the functional effects of 3D chromatin folding, establishing the foundation for modern genomic foundation models.