Stealing Part of a Production Language Model

Nicholas Carlini, Daniel Paleka, K. Dvijotham, Thomas Steinke, Jonathan Hayase +8 more
2/12/2026

No official code found. Best implementation: dpaleka/stealing-part-lm-supplementary (23 stars, MIT, last commit 2024-03-20). 1 more below.

Abstract

We introduce the first model-stealing attack that extracts precise, nontrivial information from black-box production language models like OpenAI's ChatGPT or Google's PaLM-2. Specifically, our attack recovers the embedding projection layer (up to symmetries) of a transformer model, given typical API access. For under \$20 USD, our attack extracts the entire projection matrix of OpenAI's Ada and Babbage language models. We thereby confirm, for the first time, that these black-box models have a hidden dimension of 1024 and 2048, respectively. We also recover the exact hidden dimension size of the gpt-3.5-turbo model, and estimate it would cost under $2,000 in queries to recover the entire projection matrix. We conclude with potential defenses and mitigations, and discuss the implications of possible future work that could extend our attack.

View on arXivView PDF

Code Implementations(2)

dpaleka/stealing-part-lm-supplementaryCommunity69%
MIT

Some code for "Stealing Part of a Production Language Model"

232PythonMar 19, 20242 years agoMIT
sramshetty/stealing-part-of-an-LMCommunity56%
MIT

An unofficial implementation of "Stealing Part of a Production Language Model"

51Python, Jupyter NotebookMar 13, 20242 years agoMIT

Similar Papers

HuRef: HUman-REadable Fingerprint for Large Language Models

82% match1 repo162026

HoneypotNet: Backdoor Attacks Against Model Extraction

81% match2025

MoEcho: Exploiting Side-Channel Attacks to Compromise User Privacy in Mixture-of-Experts LLMs

81% match2025

A Survey on Model Extraction Attacks and Defenses for Large Language Models

81% match2025

A Hard-Label Cryptanalytic Extraction of Non-Fully Connected Deep Neural Networks using Side-Channel Attacks

81% match2024

Label-Only Model Inversion Attacks via Knowledge Transfer

81% match1 repo132026

Cite this paper

@article{carlini2026stealing,
  title  = {Stealing Part of a Production Language Model},
  author = {Nicholas Carlini and Daniel Paleka and K. Dvijotham and Thomas Steinke and Jonathan Hayase and A. F. Cooper and Katherine Lee and Matthew Jagielski and Milad Nasr and Arthur Conmy and Eric Wallace and D. Rolnick and Florian Tramèr},
  year   = {2026},
  eprint = {2403.06634},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2403.06634},
  journal = {ICML 2024 2024}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2403.06634.svg)](https://codeofpaper.com/paper/2403.06634)

Discussion