ESM-2: Language Models for Atomic Protein Structure

Traditional structure prediction methods rely on comparing a target protein against millions of related sequences in genetic databases. This homology search step requires minutes to hours of compute time per sequence and fails when analyzing orphan proteins, fast-evolving pathogens, or designed proteins that lack known evolutionary relatives. The Meta AI team explored whether a language model could distill evolutionary patterns directly into its parameter weights, allowing a single neural network pass to infer structural coordinates directly from an individual sequence.

ESM-2 is a Transformer masked language model scaled up to 15 billion parameters across 48 layers. Key technical elements include:

• Masked Token Pre-Training: Trained on 65 million distinct protein sequences from the UniRef50 database. During pre-training, 15% of amino acid tokens are masked or replaced, and the model learns to reconstruct the missing residues using bidirectional sequence context.
• Rotary Position Embeddings (RoPE): The model replaces fixed additive position markers with Rotary Position Embeddings. RoPE applies complex vector rotations that inject relative spatial distance directly into the query-key dot products, helping the model extrapolate to longer sequences.
• Emergence of Contact Maps in Attention: Without receiving any 3D structural supervision during language pre-training, the self-attention heads automatically begin tracking co-evolving residue pairs. Linear combinations of these learned attention patterns accurately reconstruct pairwise physical contact maps.

To translate internal sequence representations into Cartesian space, the authors introduced ESMFold. ESMFold removes the entire MSA representation module used in earlier systems, connecting a folding trunk directly to the single-sequence representations output by ESM-2. It utilizes Invariant Point Attention (IPA) and backbone frame updates to iteratively refine 3D coordinates. ESMFold is supervised with Frame Aligned Point Error (FAPE), distance matrix cross-entropy, and torsion angle losses.

Because it bypasses database searches and MSA compilation, ESMFold predicts atomic coordinates roughly 60 times faster than AlphaFold 2 on median-length proteins, solving full structures in seconds per chain while maintaining competitive accuracy on blind CASP14 and CAMEO evaluations.

To demonstrate its computational throughput, the team deployed ESMFold to fold 617 million metagenomic sequences from the MGnify database in two weeks. Over 225 million predictions were cataloged with reliable confidence scores, unveiling thousands of previously uncharacterized protein folds from dark metagenomic space. ESM-2 established the principle that evolutionary variation and physical constraints are encoded within the statistical distribution of natural sequences, demonstrating that language models provide an efficient alternative to traditional homology pipelines.