Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence

Federico Bianchi, Silvia Terragni, Dirk Hovy
4/8/2020
cs.CL

Official code: MilaNLProc/contextualized-topic-models (1,272 stars, MIT, last commit 2025-07-24).

Abstract

Topic models extract groups of words from documents, whose interpretation as a topic hopefully allows for a better understanding of the data. However, the resulting word groups are often not coherent, making them harder to interpret. Recently, neural topic models have shown improvements in overall coherence. Concurrently, contextual embeddings have advanced the state of the art of neural models in general. In this paper, we combine contextualized representations with neural topic models. We find that our approach produces more meaningful and coherent topics than traditional bag-of-words topic models and recent neural models. Our results indicate that future improvements in language models will translate into better topic models.

View on arXivView PDF

Code Implementations(1)

MilaNLProc/contextualized-topic-modelsOfficial100%
MIT

A python package to run contextualized topic modeling. CTMs combine contextualized embeddings (e.g., BERT) with topic models to get coherent topics. Published at EACL and ACL 2021 (Bianchi et al.).

1,272154Python, MakefileApr 4, 20201 years agoMIT
bertembeddingsmultilingual-modelsmultilingual-topic-modelsneural-topic-models+7 more

Similar Papers

CWTM: Leveraging Contextualized Word Embeddings from BERT for Neural Topic Modeling

91% match1 repo32023

Effective Neural Topic Modeling with Embedding Clustering Regularization

88% match1 repo442023

Improving Neural Topic Modeling with Semantically-Grounded Soft Label Distributions

87% match2026

A Survey on Neural Topic Models: Methods, Applications, and Challenges

87% match2 repos2932024

Revisiting Topic-Guided Language Models

86% match2023

BERTopic: Neural topic modeling with a class-based TF-IDF procedure

85% match2 repos7.9k2022

Cite this paper

@article{bianchi2020pretraining,
  title  = {Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence},
  author = {Federico Bianchi and Silvia Terragni and Dirk Hovy},
  year   = {2020},
  eprint = {2004.03974},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2004.03974}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2004.03974.svg)](https://codeofpaper.com/paper/2004.03974)

Discussion