Evaluating Paraphrastic Robustness in Textual Entailment Models

Dhruv Verma, Yash Kumar Lal, Shreyashee Sinha, Benjamin Van Durme, Adam Poliak
6/29/2023
cs.CLcs.AI

No confident code implementation found for this paper.

Abstract

We present PaRTE, a collection of 1,126 pairs of Recognizing Textual Entailment (RTE) examples to evaluate whether models are robust to paraphrasing. We posit that if RTE models understand language, their predictions should be consistent across inputs that share the same meaning. We use the evaluation set to determine if RTE models' predictions change when examples are paraphrased. In our experiments, contemporary models change their predictions on 8-16\% of paraphrased examples, indicating that there is still room for improvement.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

How often are errors in natural language reasoning due to paraphrastic variability?

85% match2024

Paraphrase Types Elicit Prompt Engineering Capabilities

83% match1 repo22024

Language Models (Mostly) Know What They Know

83% match2022

Language Modeling and Understanding Through Paraphrase Generation and Detection

82% match2026

Vector-Quantized Prompt Learning for Paraphrase Generation

82% match2023

Are We Evaluating Knowledge or Phrasing? Mitigating MCQA Sensitivity with ParaEval

81% match2026

Cite this paper

@article{verma2023evaluating,
  title  = {Evaluating Paraphrastic Robustness in Textual Entailment Models},
  author = {Dhruv Verma and Yash Kumar Lal and Shreyashee Sinha and Benjamin Van Durme and Adam Poliak},
  year   = {2023},
  eprint = {2306.16722},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.16722v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.16722.svg)](https://codeofpaper.com/paper/2306.16722)

Discussion