![[2602.17532] Systematic Evaluation of Single-Cell Foundation Model Interpretability Reveals Attention Captures Co-Expression Rather Than Unique Regulatory Signal](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.17532] Systematic Evaluation of Single-Cell Foundation Model Interpretability Reveals Attention Captures Co-Expression Rather Than Unique Regulatory Signal
Summary
This article evaluates the interpretability of single-cell foundation models, revealing that attention mechanisms capture co-expression rather than unique regulatory signals, with implications for predictive modeling in genomics.
Why It Matters
Understanding how single-cell models interpret biological data is crucial for advancing genomic research. This study highlights the limitations of attention mechanisms in predictive accuracy, guiding future model development and quality control in genomics.
Key Takeaways
- Attention patterns in models like scGPT and Geneformer encode biological information but lack predictive value.
- Gene-level baselines outperform attention mechanisms in perturbation prediction.
- The relationship between attention and correlation is context-dependent across different cell types.
- Cell-State Stratified Interpretability (CSSI) improves gene regulatory network recovery significantly.
- The study establishes quality-control standards for interpretability in single-cell models.
Quantitative Biology > Genomics arXiv:2602.17532 (q-bio) [Submitted on 19 Feb 2026] Title:Systematic Evaluation of Single-Cell Foundation Model Interpretability Reveals Attention Captures Co-Expression Rather Than Unique Regulatory Signal Authors:Ihor Kendiukhov View a PDF of the paper titled Systematic Evaluation of Single-Cell Foundation Model Interpretability Reveals Attention Captures Co-Expression Rather Than Unique Regulatory Signal, by Ihor Kendiukhov View PDF HTML (experimental) Abstract:We present a systematic evaluation framework - thirty-seven analyses, 153 statistical tests, four cell types, two perturbation modalities - for assessing mechanistic interpretability in single-cell foundation models. Applying this framework to scGPT and Geneformer, we find that attention patterns encode structured biological information with layer-specific organisation - protein-protein interactions in early layers, transcriptional regulation in late layers - but this structure provides no incremental value for perturbation prediction: trivial gene-level baselines outperform both attention and correlation edges (AUROC 0.81-0.88 versus 0.70), pairwise edge scores add zero predictive contribution, and causal ablation of regulatory heads produces no degradation. These findings generalise from K562 to RPE1 cells; the attention-correlation relationship is context-dependent, but gene-level dominance is universal. Cell-State Stratified Interpretability (CSSI) addresses an attention-specif...