ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction

Seyone Chithrananda, Gabriel Grand, Bharath Ramsundar
10/19/2020
cs.LGcs.CLphysics.chem-ph
transformer-attentioncontrastive-ssl

Official code: seyonechithrananda/bert-loves-chemistry (503 stars, MIT, last commit 2024-10-27).

Abstract

GNNs and chemical fingerprints are the predominant approaches to representing molecules for property prediction. However, in NLP, transformers have become the de-facto standard for representation learning thanks to their strong downstream task transfer. In parallel, the software ecosystem around transformers is maturing rapidly, with libraries like HuggingFace and BertViz enabling streamlined training and introspection. In this work, we make one of the first attempts to systematically evaluate transformers on molecular property prediction tasks via our ChemBERTa model. ChemBERTa scales well with pretraining dataset size, offering competitive downstream performance on MoleculeNet and useful attention-based visualization modalities. Our results suggest that transformers offer a promising avenue of future work for molecular representation learning and property prediction. To facilitate these efforts, we release a curated dataset of 77M SMILES from PubChem suitable for large-scale self-supervised pretraining.

View on arXivView PDF

Code Implementations(1)

seyonechithrananda/bert-loves-chemistryOfficial100%
MIT

bert-loves-chemistry: a repository of HuggingFace models applied on chemical SMILES data for drug design, chemical modelling, etc.

50375Shell, PythonApr 1, 20201 years agoMIT
bertchemical-modellingchemical-smiles-datahuggingface

Similar Papers

Transformers for molecular property prediction: Lessons learned from the past five years

91% match1 repo42024

Pre-training Transformers for Molecular Property Prediction Using Reaction Prediction

88% match2022

Synergistic Fusion of Graph and Transformer Features for Enhanced Molecular Property Prediction

88% match2023

Transformers for molecular property prediction: Domain adaptation efficiently improves performance

87% match1 repo32025

Transferring a molecular foundation model for polymer property predictions

87% match2023

WEECFP-SuRGE: A Position-Aware Substructure Encoding Method for Molecular Property Prediction

86% match2026

Cite this paper

@article{chithrananda2020chemberta,
  title  = {ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction},
  author = {Seyone Chithrananda and Gabriel Grand and Bharath Ramsundar},
  year   = {2020},
  eprint = {2010.09885},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2010.09885}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2010.09885.svg)](https://codeofpaper.com/paper/2010.09885)

Discussion