Unpacking the Resilience of SNLI Contradiction Examples to Attacks

Chetan Verma, Archit Agarwal
12/15/2024
cs.CL

No official code found. Best implementation: ckvermaAI/SNLI-Attack-Analysis (0 stars, last commit 2025-08-18).

Abstract

Pre-trained models excel on NLI benchmarks like SNLI and MultiNLI, but their true language understanding remains uncertain. Models trained only on hypotheses and labels achieve high accuracy, indicating reliance on dataset biases and spurious correlations. To explore this issue, we applied the Universal Adversarial Attack to examine the model's vulnerabilities. Our analysis revealed substantial drops in accuracy for the entailment and neutral classes, whereas the contradiction class exhibited a smaller decline. Fine-tuning the model on an augmented dataset with adversarial examples restored its performance to near-baseline levels for both the standard and challenge sets. Our findings highlight the value of adversarial triggers in identifying spurious correlations and improving robustness while providing insights into the resilience of the contradiction class to adversarial attacks.

View on arXivView PDF

Code Implementations(1)

ckvermaAI/SNLI-Attack-AnalysisCommunity100%

This repo implements the experiment described in "Unpacking the Resilience of SNLI Contradiction Examples to Attacks"

00Python, Jupyter NotebookOct 30, 20241 years ago

Similar Papers

Enhancing adversarial robustness in Natural Language Inference using explanations

83% match1 repo92024

It Is All About Data: A Survey on the Effects of Data on Adversarial Robustness

83% match2023

Selective Adversarial Attacks on LLM Benchmarks

83% match2025

Adversarial Attacks and Defense for Conversation Entailment Task

82% match2024

Towards Robust and Accurate Stability Estimation of Local Surrogate Models in Text-based Explainable AI

82% match2025

Accidental Vulnerability: Factors in Fine-Tuning that Shift Model Safeguards

82% match1 repo52025

Cite this paper

@article{verma2024unpacking,
  title  = {Unpacking the Resilience of SNLI Contradiction Examples to Attacks},
  author = {Chetan Verma and Archit Agarwal},
  year   = {2024},
  eprint = {2412.11172},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2412.11172v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2412.11172.svg)](https://codeofpaper.com/paper/2412.11172)

Discussion