![[2602.11618] How Well Do Large-Scale Chemical Language Models Transfer to Downstream Tasks?](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.11618] How Well Do Large-Scale Chemical Language Models Transfer to Downstream Tasks?
Summary
This paper evaluates the effectiveness of large-scale Chemical Language Models (CLMs) in transferring knowledge to downstream molecular property prediction tasks, revealing limited performance improvements despite increased training resources.
Why It Matters
Understanding the transferability of CLMs is crucial for optimizing molecular property predictions in chemistry. The findings challenge the assumption that larger models always yield better performance, highlighting the need for more nuanced evaluation strategies tailored to specific tasks.
Key Takeaways
- Increased training resources reduce pretraining loss but do not guarantee better downstream performance.
- Alternative evaluation metrics based on Hessian or loss landscape are ineffective in predicting downstream success.
- Downstream performance can saturate or degrade even with improved pretraining metrics.
- The study emphasizes the importance of task-specific model evaluation strategies.
- Visualizations of parameter space reveal task-dependent failure modes.
Computer Science > Machine Learning arXiv:2602.11618 (cs) [Submitted on 12 Feb 2026 (v1), last revised 17 Feb 2026 (this version, v2)] Title:How Well Do Large-Scale Chemical Language Models Transfer to Downstream Tasks? Authors:Tatsuya Sagawa, Ryosuke Kojima View a PDF of the paper titled How Well Do Large-Scale Chemical Language Models Transfer to Downstream Tasks?, by Tatsuya Sagawa and Ryosuke Kojima View PDF Abstract:Chemical Language Models (CLMs) pre-trained on large scale molecular data are widely used for molecular property prediction. However, the common belief that increasing training resources such as model size, dataset size, and training compute improves both pretraining loss and downstream task performance has not been systematically validated in the chemical domain. In this work, we evaluate this assumption by pretraining CLMs while scaling training resources and measuring transfer performance across diverse molecular property prediction (MPP) tasks. We find that while pretraining loss consistently decreases with increased training resources, downstream task performance shows limited improvement. Moreover, alternative metrics based on the Hessian or loss landscape also fail to estimate downstream performance in CLMs. We further identify conditions under which downstream performance saturates or degrades despite continued improvements in pretraining metrics, and analyze the underlying task dependent failure modes through parameter space visualizations. These ...
