scGPT: Foundation Transformer for Single-Cell Biology

Unlike linear DNA or protein sequences, the transcriptome of an individual cell is an unordered collection of thousands of gene expression values. Measuring these expression levels via single-cell sequencing introduces substantial technical noise, severe dropout where expressed transcripts fail to be captured, and batch discrepancies between different sequencing platforms. Traditional bioinformatic pipelines rely on task-specific heuristics, clustering algorithms, and separate integration tools, preventing cross-study generalization. scGPT addressed this fragmentation by formulating single-cell biology as a generative language modeling problem over cellular vocabularies.

scGPT models single-cell transcriptomes through a specialized tokenization and attention scheme:

• Input Representation: Each cell is represented as a sequence of gene tokens. Each input vector sums:
  1. Gene Identity Token: A discrete identifier selected from a vocabulary of over 30,000 human genes.
  2. Expression Value Embedding: A continuous scalar expression value projected into the hidden dimension through learnable dense layers.
  3. Condition Tokens: Metadata tags encoding cell type, tissue source, batch, or experimental protocol.
• Permutation Invariance: Because cellular transcriptomes lack natural grammatical order, genes are ordered by expression magnitude or processed via set-based self-attention. This allows self-attention heads to compute pairwise dependencies between all co-expressed genes.
• Pre-Training Objectives:
  • Masked Gene Expression Prediction: Randomly masks a subset of gene expression values, training the Transformer to predict the masked values from the remaining transcriptomic context using mean squared error and cross-entropy losses.
  • Gene-Gene Relationship Alignment: Aligns learned gene embeddings to preserve biological co-expression networks.

The model was pre-trained on a corpus of more than 33 million individual human cells sourced from the CELLxGENE portal.

Once pre-trained, scGPT transfers across multiple single-cell workflows:

• Cell Type Annotation: Surpassed specialized models such as CellTypist and scVI in classifying rare and unseen cell types across independent datasets.
• Batch Correction & Integration: Successfully integrated disparate datasets across laboratories, harmonizing batch variations while preserving biological cell clusters.
• In Silico Genetic Perturbations: On Perturb-seq CRISPR screens, scGPT predicted post-knockout transcriptomic changes with greater accuracy than specialized graph neural networks, forecasting how gene deletions ripple through regulatory cascades.
• De Novo Pathway Discovery: Attention weight matrices inside the Transformer naturally reconstructed canonical biological pathways and signaling cascades without receiving explicit pathway annotations.

scGPT established that the complex, noisy internal states of single cells can be unified within a generative foundation model. By demonstrating that pre-training on tens of millions of cells yields transferable representations of cellular identity and gene regulation, scGPT bridged molecular genetics and whole-cell phenotypic modeling, providing a platform for computational drug screening and cellular engineering.