BERTScore: Evaluating Text Generation with BERT

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, Yoav Artzi
4/21/2019
cs.CL

Official code: Tiiiger/bert_score (1,921 stars, MIT, last commit 2024-07-30).

Abstract

We propose BERTScore, an automatic evaluation metric for text generation. Analogously to common metrics, BERTScore computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTScore correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BERTScore is more robust to challenging examples when compared to existing metrics.

View on arXivView PDF

Code Implementations(1)

Tiiiger/bert_scoreOfficial100%
MIT

BERT score for text generation

1,921237Shell, PythonApr 20, 20192 years agoMIT
machine-learningnatural-language-processing

Similar Papers

INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedback

88% match1 repo622023

MENLI: Robust Evaluation Metrics from Natural Language Inference

85% match2022

MuLER: Detailed and Scalable Reference-based Evaluation

85% match2023

BARTScore: Evaluating Generated Text as Text Generation

84% match1 repo3702021

MATCHA: Matching Text via Contrastive Semantic Alignment

84% match2026

MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers

84% match2 repos3142021

Cite this paper

@article{zhang2019bertscore,
  title  = {BERTScore: Evaluating Text Generation with BERT},
  author = {Tianyi Zhang and Varsha Kishore and Felix Wu and Kilian Q. Weinberger and Yoav Artzi},
  year   = {2019},
  eprint = {1904.09675},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1904.09675}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1904.09675.svg)](https://codeofpaper.com/paper/1904.09675)

Discussion