Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation

Xinyu Tang, Richard Shin, Huseyin A. Inan, Andre Manoel, FatemehSadat Mireshghallah +4 more
2/12/2026

No official code found. Best implementation: microsoft/dp-few-shot-generation (28 stars, MIT, last commit 2023-11-28).

Abstract

We study the problem of in-context learning (ICL) with large language models (LLMs) on private datasets. This scenario poses privacy risks, as LLMs may leak or regurgitate the private examples demonstrated in the prompt. We propose a novel algorithm that generates synthetic few-shot demonstrations from the private dataset with formal differential privacy (DP) guarantees, and show empirically that it can achieve effective ICL. We conduct extensive experiments on standard benchmarks and compare our algorithm with non-private ICL and zero-shot solutions. Our results demonstrate that our algorithm can achieve competitive performance with strong privacy levels. These results open up new possibilities for ICL with privacy protection for a broad range of applications.

View on arXivView PDF

Code Implementations(1)

microsoft/dp-few-shot-generationCommunity100%
MIT
286Shell, PythonSep 15, 20232 years agoMIT

Similar Papers

Public Data Assisted Differentially Private In-Context Learning

92% match2025

Differentially Private In-context Learning via Sampling Few-shot Mixed with Zero-shot Outputs

92% match2025

Privacy-Preserving In-Context Learning for Large Language Models

92% match2023

Data-adaptive Differentially Private Prompt Synthesis for In-Context Learning

90% match2024

Private PoEtry: Private In-Context Learning via Product of Experts

90% match2026

Privacy Preserving In-Context-Learning Framework for Large Language Models

89% match1 repo42025

Cite this paper

@article{tang2026privacypreserving,
  title  = {Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation},
  author = {Xinyu Tang and Richard Shin and Huseyin A. Inan and Andre Manoel and FatemehSadat Mireshghallah and Zinan Lin and Sivakanth Gopi and Janardhan Kulkarni and Robert Sim},
  year   = {2026},
  eprint = {2309.11765},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/pdf/2309.11765},
  journal = {ICLR 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2309.11765.svg)](https://codeofpaper.com/paper/2309.11765)

Discussion