Contextualized Topic Coherence Metrics

Hamed Rahimi, Jacob Louis Hoover, David Mimno, Hubert Naacke, Camelia Constantin +1 more
5/23/2023
cs.CLcs.IR

No official code found. Best implementation: hamedR96/CTC (8 stars, last commit 2023-10-15).

Abstract

The recent explosion in work on neural topic modeling has been criticized for optimizing automated topic evaluation metrics at the expense of actual meaningful topic identification. But human annotation remains expensive and time-consuming. We propose LLM-based methods inspired by standard human topic evaluations, in a family of metrics called Contextualized Topic Coherence (CTC). We evaluate both a fully automated version as well as a semi-automated CTC that allows human-centered evaluation of coherence while maintaining the efficiency of automated methods. We evaluate CTC relative to five other metrics on six topic models and find that it outperforms automated topic coherence methods, works well on short documents, and is not susceptible to meaningless but high-scoring topics.

View on arXivView PDF

Code Implementations(1)

hamedR96/CTCCommunity100%

Contextualised Topic Coherence Metrics: A new way to evaluate neural topic models.

81Python, Jupyter NotebookApr 25, 20233 years ago
topic-coherencetopic-modeling

Similar Papers

Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysis

89% match1 repo112024

A modified model for topic detection from a corpus and a new metric evaluating the understandability of topics

88% match2023

When Numbers Tell Half the Story: Human-Metric Alignment in Topic Model Evaluation

86% match2026

A Survey on Neural Topic Models: Methods, Applications, and Challenges

84% match2 repos2932024

Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders

83% match2025

Neural Multimodal Topic Modeling: A Comprehensive Evaluation

83% match1 repo52024

Cite this paper

@article{rahimi2023contextualized,
  title  = {Contextualized Topic Coherence Metrics},
  author = {Hamed Rahimi and Jacob Louis Hoover and David Mimno and Hubert Naacke and Camelia Constantin and Bernd Amann},
  year   = {2023},
  eprint = {2305.14587},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.14587v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.14587.svg)](https://codeofpaper.com/paper/2305.14587)

Discussion