Semantic Membership Inference Attack against Large Language Models

Hamid Mozaffari, Virendra J. Marathe
6/14/2024
cs.LG

No confident code implementation found for this paper.

Abstract

Membership Inference Attacks (MIAs) determine whether a specific data point was included in the training set of a target model. In this paper, we introduce the Semantic Membership Inference Attack (SMIA), a novel approach that enhances MIA performance by leveraging the semantic content of inputs and their perturbations. SMIA trains a neural network to analyze the target model's behavior on perturbed inputs, effectively capturing variations in output probability distributions between members and non-members. We conduct comprehensive evaluations on the Pythia and GPT-Neo model families using the Wikipedia dataset. Our results show that SMIA significantly outperforms existing MIAs; for instance, SMIA achieves an AUC-ROC of 67.39% on Pythia-12B, compared to 58.90% by the second-best attack.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Do Membership Inference Attacks Work on Large Language Models?

89% match1 repo1922024

Detecting Training Data of Large Language Models via Expectation Maximization

89% match1 repo72024

Towards Label-Only Membership Inference Attack against Pre-trained Large Language Models

88% match2025

Exploring the limits of strong membership inference attacks on large language models

88% match2025

Context-Aware Membership Inference Attacks against Pre-trained Large Language Models

88% match2024

Scaling Up Membership Inference: When and How Attacks Succeed on Large Language Models

88% match1 repo162024

Cite this paper

@article{mozaffari2024semantic,
  title  = {Semantic Membership Inference Attack against Large Language Models},
  author = {Hamid Mozaffari and Virendra J. Marathe},
  year   = {2024},
  eprint = {2406.10218},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.10218v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.10218.svg)](https://codeofpaper.com/paper/2406.10218)

Discussion